For the (rare) messages without a Date: header, attempt to guess a date
from the top (most-recent) Received: header. Not perfect, of course, but
better than nothing.
Requires re-indexing to apply to already indexed messages.
Should help for #1083.
Move the various seq_ functions, as well as std::(stable_)sort,
std::accumulate, std::transform, std::find, std::find_if to their C++20
std::ranges counterparts.
Cleanup the query running and handling the results.
- in threading, use stable_sort
- remove dead code
- fix indentation in a few places
- don't need Option in various run_... in mu-query.cc
Support "system" properties, i.e., library versions, whether
language-detection, scm is supported.
Make these more generic (with display strings), so we can handle them
more automatic int mu-cmd-info.
Some improvements:
- Fix or/xor chains (use left-associativity).
- Fix quote handling
- Make parsing O(n) rather than quadratic; limit recursion depth
And update tests.
We already pre-fetched db file paths for the cleanup-case (since
cdb619e4f), now let's generalize this to all indexing.
For now this is mostly performance-neutral (lazy-rescan is slightly
faster); however, this simplifies the code.
Update various places for what we can do with C++20:
- ends_with / starts_with
- std::to_array
- using instead of typedef
- designated initializers
- "[[maybe_unused]]" instead of G_GNUC_UNUSED
Also remove some unnecessary 'static'
Modernize code a bit; and use a temporary dir for the working set.
Typically, this makes the tests run faster (since the temporary dir is
often in-memory), which allows focusing on the part we can most easily
influence.
Do not consider calender-invitations "attachments"; do mark as
"calendar". Do recognize application/ics as calendar messages.
Update mime-object to expose a message part's disposition.
Change the "is-attachment" heuristic to include inline parts if they
have a filename parameter in their content-disposition.
Note that this doesn't change things radically; the delta is +69 and
-202 for ~6500 attachments.
This change adds a new cleanup mode that avoids cleanup having
re-traverse the directories the index pass just looked at.
Additionally, we efficiently query the Xapian database by walking the
term list instead of doing multiple point-wise path lookups.
I'd noticed that most of my time in mu's cleanup pass consisted of
B-tree lookups in Xapian (one 8KB pread64 at a time). The point
lookups forced Xapian to traverse from the root of the B-tree to the
leaf for every single message. Additionally, in order to join on the
message path, we had to do *another* B-tree traversal after locating
each message term. Now we just walk the terms in order, which is much
more efficient, as we touch each B-tree node only once.
On my system, with 1371861 total messages, the total time of mu
index (no lazy check):
--nocleanup: 3.6s
incremental cleanup: 4.2s (0.6s in cleanup)
legacy cleanup: 5.2s (1.6s in cleanup)
With the new mode, we save 1.0s of the 1.6s cleanup, so we're
~63% faster.
But the incremental cleanup works even better with lazy checking.
If I enable --lazy-check, dirty only my INBOX (360778 messages), and
run index, I get:
--nocleanup: 0.9s
incremental cleanup: 1.1s (0.2s in cleanup)
legacy cleanup: 2.5s (1.6s in cleanup)
We save 1.4s out of 1.6s for ~88% speedup.
This change also fixes a timestamp bug: we should be storing
the *start* time of the index pass in metadata, not the end time, so
that on the next index pass, we notice messages that arrived between
the two times.
All tests pass. You can set the environment variable
MU_NO_INCREMENTAL_CLEANUP to use the legacy cleanup path instead.
