#+TITLE: MU INDEX
#+MAN_CLASS_OPTIONS: :section-id "@SECTION_ID@" :date "@MAN_DATE@"
#+include: macros.inc

* NAME

mu-index - index e-mail messages stored in Maildirs

* SYNOPSIS

*mu* [​_COMMON-OPTIONS_​] *index*

* DESCRIPTION

*mu index* is the *mu* command for scanning the messages in Maildir directories and
storing the results in a Xapian database. The data can then be queried using
{{{man-link(mu-find,1)}}}.

Before you can run *mu index*, you must initialize *mu* and its database with
{{{man-link(mu-init,1)}}}.

*mu index* understands Maildirs as defined by Daniel Bernstein for
{{{man-link(qmail,7)}}}. In addition, it understands recursive Maildirs
(Maildirs within Maildirs), Maildir++. It also supports VFAT-based Maildirs
which use "*!"* or "*;"* as the separators, instead of the standard "*:"*.

E-mail messages are only considered for indexing if they reside in a directory
named either ~cur~ or ~new~. The special (as per the Maildir-specification)
directory ~tmp~ is ignored, as are some cache directories for some other
mail-clients. Other directories are scanned recursively.

Symbolic links are followed, and the directories can be spread over multiple
filesystems; however, note that moving files around is much faster when they all
reside on a single file-system. Be careful to avoid self-referential symlinks!

If there is a file called ~.noindex~ in a directory, the contents of that
directory and all of its subdirectories will be ignored. This can be useful to
exclude certain directories from the indexing process, for example directories
with spam-messages.

If there is a file called ~.noupdate~ in a directory, the contents of that
directory and all of its subdirectories will be ignored. This can be useful to
speed up things you have some maildirs that never change.

~.noupdate~ does not affect already-indexed messages: you can still search for
them. ~.noupdate~ is ignored when you start indexing with an empty database (such
as directly after *mu init*).

There also the option *--lazy-check* which can greatly speed up indexing; see
below for details.

A first run of *mu index* may take a few minutes if you have a lot of mail (tens
or hundreds of thousands of messages). Fortunately, such a full scan needs to be
done only rarely; after that, it suffices to index just the changes, which goes
much faster. See *PERFORMANCE (i,ii,iii)* below for more information.

The optional cleanup-phase of the indexing-process is the removal of messages
from the database for which there is no longer a corresponding file in the
Maildir. If you do not want this, you can use *-n*, *--nocleanup*.

When *mu index* catches one of the signals *SIGINT*, *SIGHUP* or *SIGTERM* (e.g., when
you press *Ctrl-C* during the indexing process), it attempts to shutdown
gracefully: save and commit data and close the database etc. If it receives
another signal (e.g., when pressing Ctrl-C once more), *mu index* will terminate
immediately.

* INDEX OPTIONS

** --lazy-check
In lazy-check mode, *mu* does not consider messages for which the time-stamp
(*ctime*) of the directory in which they reside, has not changed since the
previous time this directory was checked.

This is much faster than the non-lazy check, but won't update messages that have
changed (rather than having been added or removed), since merely editing a
message does not update the directory time-stamp. Of course, you can run
*mu-index* occasionally without *--lazy-check*, to pick up such messages.

Furthermore, in lazy-check mode, files which have a *ctime* smaller than the time
the previous indexing operation was completed, are ignored. This helps for the
use-case where new messages can appear in big maildirs.

** --nocleanup
Disable the database cleanup that *mu* does by default after indexing.

** --reindex
Perform a complete reindexing of all the messages in the maildir.

#+include: "muhome.inc" :minlevel 2

#+include: "common-options.inc" :minlevel 1

* ENCRYPTION

*mu index* does _not_ decrypt messages, and only the metadata (such as headers) of
encrypted messages makes it to the database. *mu view* and *mu4e* can decrypt
messages, but those work with the message directly and the information is not
added to the database.

* PERFORMANCE

** indexing in ancient times (2009?)

As a non-scientific benchmark, a simple test on the author's machine (a Thinkpad
X61s laptop using Linux 2.6.35 and an ext3 file system) with no existing
database, and a maildir with 27273 messages:

#+begin_example
$ sudo sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches'
$ time mu index --quiet
66,65s user 6,05s system 27% cpu 4:24,20 total
#+end_example
(about 103 messages per second)

A second run, which is the more typical use case when there is a database
already, goes much faster:

#+begin_example
$ sudo sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches'
$ time mu index --quiet
0,48s user 0,76s system 10% cpu 11,796 total
#+end_example
(more than 56818 messages per second)

Note that each test flushes the caches first; a more common use case might be to
run *mu index* when new mail has arrived; the cache may stay quite `warm' in that
case:

#+begin_example
 $ time mu index --quiet
 0,33s user 0,40s system 80% cpu 0,905 total
#+end_example
which is more than 30000 messages per second.

** indexing in 2012

As per June 2012, we did the same non-scientific benchmark, this time with an
Intel i5-2500 CPU @ 3.30GHz, an ext4 file system and a maildir with 22589
messages. We start without an existing database.

#+begin_example
 $ sudo sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches'
 $ time mu index --quiet
 27,79s user 2,17s system 48% cpu 1:01,47 total
#+end_example
(about 813 messages per second)

A second run, which is the more typical use case when there is a database
already, goes much faster:

#+begin_example
$ sudo sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches'
$ time mu index --quiet
0,13s user 0,30s system 19% cpu 2,162 total
#+end_example
(more than 173000 messages per second)

** indexing in 2016

As per July 2016, we did the same non-scientific benchmark, again with the Intel
i5-2500 CPU @ 3.30GHz, an ext4 file system. This time, the maildir contains
72525 messages.

#+begin_example
$ sudo sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches'
$ time mu index --quiet
40,34s user 2,56s system 64% cpu 1:06,17 total
#+end_example
(about 1099 messages per second).

** indexing in 2022

A few years later and it is June 2022. There's a lot more happening during
indexing, but indexing became multi-threaded and machines are faster; e.g. this
is with an AMD Ryzen Threadripper 1950X (16 cores) @ 3.399GHz.

The instructions are a little different since we have a proper repeatable
benchmark now. After building,

#+begin_example
$ sudo sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches'
% THREAD_NUM=4 build/lib/tests/bench-indexer -m perf
# random seed: R02Sf5c50e4851ec51adaf301e0e054bd52b
1..1
# Start of bench tests
# Start of indexer tests
indexed 5000 messages in 20 maildirs in 3763ms; 752 μs/message; 1328 messages/s (4 thread(s))
ok 1 /bench/indexer/4-cores
# End of indexer tests
# End of bench tests
#+end_example

Things are again a little faster, even though the index does a lot more now
(text-normalization, and pre-generating message-sexps). A faster machine helps,
too!

** recent releases

Indexing the same 93000-message mail corpus with the last few releases:

#+ATTR_MAN: :disable-caption t
|       release | time (sec) | notes                                    |
|---------------+------------+------------------------------------------|
|           1.4 | 160s       |                                          |
|           1.6 | 178s       |                                          |
|           1.8 | 97s        |                                          |
|          1.10 | 120s       | adds html indexing, sexp-caching         |
| 1.11 (master) | 96s        | adds language-guessing, batch-size=50000 |
|               |            |                                          |

Quite some variation!

Over time new features / refactoring can change the timings quite a bit. At
least for now, the latest code is both the fastest and the most feature-rich.

#+include: "exit-code.inc" :minlevel 1

#+include: "prefooter.inc"

* SEE ALSO

{{{man-link(maildir,5)}}},
{{{man-link(mu,1)}}},
{{{man-link(mu-init,1)}}},
{{{man-link(mu-find,1)}}},
{{{man-link(mu-cfind,1)}}}
