Support "system" properties, i.e., library versions, whether
language-detection, scm is supported.
Make these more generic (with display strings), so we can handle them
more automatic int mu-cmd-info.
We already pre-fetched db file paths for the cleanup-case (since
cdb619e4f), now let's generalize this to all indexing.
For now this is mostly performance-neutral (lazy-rescan is slightly
faster); however, this simplifies the code.
This change adds a new cleanup mode that avoids cleanup having
re-traverse the directories the index pass just looked at.
Additionally, we efficiently query the Xapian database by walking the
term list instead of doing multiple point-wise path lookups.
I'd noticed that most of my time in mu's cleanup pass consisted of
B-tree lookups in Xapian (one 8KB pread64 at a time). The point
lookups forced Xapian to traverse from the root of the B-tree to the
leaf for every single message. Additionally, in order to join on the
message path, we had to do *another* B-tree traversal after locating
each message term. Now we just walk the terms in order, which is much
more efficient, as we touch each B-tree node only once.
On my system, with 1371861 total messages, the total time of mu
index (no lazy check):
--nocleanup: 3.6s
incremental cleanup: 4.2s (0.6s in cleanup)
legacy cleanup: 5.2s (1.6s in cleanup)
With the new mode, we save 1.0s of the 1.6s cleanup, so we're
~63% faster.
But the incremental cleanup works even better with lazy checking.
If I enable --lazy-check, dirty only my INBOX (360778 messages), and
run index, I get:
--nocleanup: 0.9s
incremental cleanup: 1.1s (0.2s in cleanup)
legacy cleanup: 2.5s (1.6s in cleanup)
We save 1.4s out of 1.6s for ~88% speedup.
This change also fixes a timestamp bug: we should be storing
the *start* time of the index pass in metadata, not the end time, so
that on the next index pass, we notice messages that arrived between
the two times.
All tests pass. You can set the environment variable
MU_NO_INCREMENTAL_CLEANUP to use the legacy cleanup path instead.
This change makes index cleanup ~4x faster by changing how we
determine whether a file mentioned by the database still exists on
disk. Previously, we'd call access(2) for each file the database
mentioned. Doing so produced a lot of system call overhead. Now, we
read the directory entries of the directories containing the files
whose existence we're checking, build a hash table from what we find,
then do the existence check against this hash table instead of
entering the kernel.
The semantics of the cleanup check do change subtly, however.
Previously, we checked whether the mentioned file was *readable*.
Now we check merely that it exists. Extant but unreadable files in
maildirs should be rare.
BEFORE:
$ time mu index --lazy-check
lazily indexing maildir /home/dancol/Mail -> store /home/dancol/.cache/mu/xapian
/ indexing messages; checked: 0; updated/new: 0; cleaned-up: 0
real 0m19.310s
user 0m1.803s
sys 0m12.999s
AFTER:
$ time mu --debug index --lazy-check
lazily indexing maildir /home/dancol/Mail -> store /home/dancol/.cache/mu/xapian
- indexing messages; checked: 0; updated/new: 0; cleaned-up: 0
real 0m4.584s
user 0m2.433s
sys 0m2.133s
Single-threaded is the build-default, and seems to work well enough for
1.12.7, so remove the option to turn it off.
This is because build-options that influence such low-level/core
behavior are a pain to maintain.
In lazy-mode, we were skipping directories that did not change; however,
this didn't help for the case were users received new messages in big
maildirs.
So, add another check where we compare the ctime of message files with
the time of the last indexing operation. If it's smaller, ignore the
message-file. This is faster than having to consult the Xapian database
for each message.
Note that this requires in mu4e:
(setq mu4e-index-lazy-check t)
or
--lazy-check
as a parameter for 'mu index'.
Try to avoid multi-threaded operations with Xapian.
This remove the thread workers during indexing, and avoids the indexing
background thread. So, mu4e has to wait once again during indexing.
We can improve upon that, but first we need to know if it avoids the
problem of issue #2756.
Add store::consume_message, which is like add message but std::move from
the caller such that the messages longer has copies (with
Xapian::Document) on the caller side; this is to avoid threading issues.
Instead of handling transactions in the store, handle it in xapian-db.
Make the code a bit more natural / cleaner-out
Handle transaction automatically (with a batch-size) and add some RAII
Transaction object, which makes all database interaction transactable
for the duration. So, no more need for explicit parameters to
add_message while indexing.
