Copyright © 2012 Dirk-Jan C. Binnema

     Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license is included in the section entitled ``GNU Free Documentation License.''

mu4e Manual
***********

Welcome to mu-guile!

mu-guile is a binding of the mu email search engine for the guile programming language. That means that you can write simple (and not so simple) programs to data-mine your e-mail database.

Introduction
************

mu is a program for indexing / searching e-mails stored in Maildirs.

guile is the GNU Ubiquitous Intelligent Language for Extensions - a version of the Scheme programming language and the official GNU extension language.

mu-guile connects mu and guile, and allows you to easily write programs for your specific needs.

Getting started
***************

This chapter walks you through the installation and some basic steps to ensure things work correctly.

Installation
============

mu-guile is part of mu - by installing the latter, the former will be installed as well, provided that you have guile version 2.0 installed.

At the time of writing, there are no distribution packages for mu-guile, so we are assuming installation from source packages.

Installation follows the normal sequence of:

     $ tar xvfz mu-<version>.tar.gz  # use the specific version
     $ cd mu-<version>
     $./configure

The output of ./configure should end with a little text describing the detected versions of various libraries mu depends on. In particular, it should mention the guile version, e.g.

     Guile version                        : 2.0.3.82-a2c66

If you don't see any line referring to guile, please install it, and run configure again. Note once more, mu-guile requires guile version 2.0.

After a succesfull ./configure, we can make and install the package:

     $ make && sudo make install

After this, mu and mu-guile should be installed. Note that the above instructions will normally install things under /usr/local; you may need to update guile's %load-path to find it there.

You can check the current load-path with the following:

     guile -c '(display %load-path)(newline)'

If necessary, you can add the %load-path by adding something like the following to your ~/.guile:

     (set! %load-path (cons "/usr/local/share/guile/site/2.0" %load-path))

Or, alternatively, you can set GUILE_LOAD_PATH:

     export GUILE_LOAD_PATH="/usr/local/share/guile/site/2.0"

Note, in both cases the directory should be the directory that contains the installed mu.scm; so if you installed mu under a different prefix, you must change the load-path accordingly.

After this, you should be ready to go.

First steps
===========

Assuming mu-guile has been installed correctly (Installation), and also assuming that you have already indexed your e-mail messages (if necessary, see the mu-index man-page), we are ready to start mu-guile; a session may look something like this:

$ guile
GNU Guile 2.0.3.82-a2c66
Copyright (C) 1995-2011 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)>

Now, we need to load some of the mu-guile modules:

scheme@(guile-user)> (use-modules (mu) (mu message))

This will load the basic modules for dealing with messages. After we have loaded the modules, we need to initialize the mu-guile system:

scheme@(guile-user)> (mu:initialize)

When this is done, we can start querying the database. We discuss various methods and functions later in this manual, but just to give an example, let's get a list of the subjects of all messages that mention hello:

scheme@(guile-user)> (for-each
                        (lambda(msg)
                          (format #t "Subject: ~a\n" (mu:subject msg)))
                        (mu:message-list "hello"))

Note, the multi-lines in the example are only for readability; since it can be a bit uncomfortable to type long sequences at the 'REPL' (the Guile command-line), we recommend using a tool like Geiser (http://www.nongnu.org/geiser/).

Initializing mu-guile
*********************

It is of course possible to write separate programs with mu-guile, but for now we'll do things interactively, i.e., from the Guile-prompt (``REPL'').

We start our mu-guile session by starting guile:

$ guile
GNU Guile 2.0.3.82-a2c66
Copyright (C) 1995-2011 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)>

The first thing we need to do is loading the modules. All the basics are in the (mu) module, with some statistical extras in (mu stats), and some graph plotting functionality in (mu plot).

Let's simply load all of them:

scheme@(guile-user)> (use-modules (mu) (mu stats) (mu plot))

Assuming you have installed everything correctly, the first time you do this, guile will probably respond by showing some message about compiling the modules, and then return to you with another prompt.

Before we can do anything with mu guile, we need to initialize the system. This goes like this:

scheme@(guile-user)> (mu:initialize)

Which will use the default location of ~/.mu. Or, instead, if you keep your mu data in a non-standard place:

scheme@(guile-user)> (mu:initialize "/path/to/my/mu/")

If all worked up until here, we're ready to go with mu-guile - hurray! In the next chapters we'll walk through all the modules.

Messages
********

In this chapter, we discuss how to find messages and how to do various things with them.

Finding messages
================

Now we are ready to retrieve some messages from the system. There are two main functions to do this:

  * (mu:message-list [<search-expression>])

  * (mu:for-each-message <function> [<search-expression>])

The first function, mu:message-list returns a list of all messages matching <search-expression>; if you leave <search-expression> out, it returns all messages.

For example, to get all messages with coffee in the subject line, you could do:

scheme@(guile-user)> (mu:message-list "subject:coffee")
$1 = (#<<mu:message> 9040640> #<<mu:message> 9040630>
      #<<mu:message> 9040570>)

So, since apparently we have three messages matching subject:coffee, we get a list of three <mu:message> objects. Let's just use the mu:subject function ('method') provided by <mu:message> objects to retrieve the subject-field (more about methods in the next section).

For your convenience, guile has saved the result of our last query in a variable called $1, so to get the subject of the first message in the list, we can do:

scheme@(guile-user)> (mu:subject (car $1))
$2 = "Re: best coffee ever!"

The second function we mentioned, mu:for-each-message, executes some function for each message matched by the search expression (or all messages if the search expression is omitted).

scheme@(guile-user)> (mu:for-each-message
                       (lambda(msg)
                         (display (mu:subject msg))
                         (newline))
                      "subject:coffee")
Re: best coffee ever!
best coffee ever!
Coffee beans
scheme@(guile-user)>

Using mu:message-list and/or mu:for-each-message (Implementation node: mu:message-list is implemented in terms of mu:for-each-message, not the other way around. Due to the way mu works, mu:for-each-message is rather more efficient than a combination of for-each and mu:message-list) and a couple of <mu:message> methods, together with what Guile/Scheme provides, should allow for many interesting programs.

Message methods
===============

Now that we've seen how to retrieve lists of message objects (<mu:message>), let's see what we can do with such an object.

<mu:message> defines the following methods that all take a single <mu:message> object as a parameter. We won't go into the exact meanings for all of these functions here - for the details about various flags / properties, please refer to the mu-find man-page.

  * (mu:bcc msg): the Bcc field of the message, or #f if there is none

  * (mu:body-html msg): : the html body of the message, or #f if there is none

  * (mu:body-txt msg): the plain-text body of the message, or #f if there is none

  * (mu:cc msg): the Bcc field of the message, or #f if there is none

  * (mu:date msg): the Date field of the message, or 0 if there is none

  * (mu:flags msg): list of message-flags for this message

  * (mu:from msg): the From field of the message, or #f if there is none

  * (mu:maildir msg): the maildir this message lives in, or #f if there is none

  * (mu:message-id msg): the Message-Id field of the message, or #f if there is none

  * (mu:path msg): the file system path for this message

  * (mu:priority msg): the priority of this message (either mu:prio:low, mu:prio:normal or mu:prio:high

  * (mu:references msg): the list of messages (message-ids) this message refers to in(mu: the References: header

  * (mu:size msg): size of the message in bytes

  * (mu:subject msg): the Subject field of the message, or #f if there is none.

  * (mu:tags msg): list of tags for this message

  * (mu:timestamp msg): the timestamp (mtime) of the message file, or #f if there is none. message file

  * (mu:to msg): the sender of the message, or #f if there is none

With these methods, we can query messages for their properties; for example:

scheme@(guile-user)> (define msg (car (mu:message-list "snow")))
scheme@(guile-user)> (mu:subject msg)
$1 = "Re: Running in the snow is beautiful"
scheme@(guile-user)> (mu:flags msg)
$2 = (mu:flag:replied mu:flag:seen)
scheme@(guile-user)> (strftime "%F" (localtime (mu:date msg)))
$3 = "2011-01-15"

There are a couple more methods:

  * (mu:header msg "<header-name>") returns an arbitrary message header (or #f if not found) -- e.g. (header msg "User-Agent")

  * If you include the mu contact module, the (mu:contacts msg [contact-type]) method (to get a list of contacts) is added. See Contacts.

  * If you include the mu part module, the ((mu:parts msg) and (mu:attachments msg) methods are added. See Attachments and other parts.

Example - the longest subject
=============================

Now, let's write a little example -- let's find out what is the longest subject of any e-mail messages we received in the year 2011.  You can try this if you put the following in a separate file, make it executable, and run it like any program.

     #!/bin/sh
     exec guile -s $0 $
     !#

     (use-modules (mu))
     (use-modules (srfi srfi-1))

     (mu:initialize)

     ;; note: (subject msg) => #f if there is no subject
     (define list-of-subjects
       (map (lambda (msg)
           (or (mu:subject msg) "")) (mu:message-list "date:2011..2011")))
     ;; see the mu-find manpage for the date syntax

     (define longest-subject
       (fold (lambda (subj1 subj2)
           (if (> (string-length subj1) (string-length subj2))
     	subj1 subj2))
         "" list-of-subjects))

     (format #t "Longest subject: ~s\n" longest-subject)

There are many other ways to solve the same problem, for example by using an iterative approach with mu:for-each-message, but it should show how one can easily write little programs to answer specific questions about your e-mail corpus.

Contacts
********

We can retrieve the sender and recipients of an e-mail message using methods like mu:from, mu:to etc.; see Message methods. These functions return the list of recipients as a single string; however, often it is more useful to deal with recipients as separate objects.

Contact functions and objects
=============================

Message objects (see Messages) have a method mu:contacts:

(mu:contacts <message-object> [<contact-type>])

The <contact-type> is a symbol, one of mu:to, mu:from, mu:cc or mu:bcc. This will then get the contact objects for the contacts of the corresponding type. If you leave out the contact-type (or specify #t for it, you will get a list of all contact objects for the message.

A contact object (<mu:contact>) has two methods:

  * mu:name returns the name of the contact, or #f if there is none

  * mu:email returns the e-mail address of the contact, or #f if there is none

Let's get a list of all names and e-mail addresses in the 'To:' field, of messages matching 'book':

     (use-modules (mu))
     (mu:initialize)
     (mu:for-each-message
        (lambda (msg)
           (for-each
             (lambda (contact)
                (format #t "~a => ~a\n"
                  (or (mu:email contact) "") (or (mu:name contact) "no-name")))
             (mu:contacts msg mu:field:to)))
         "book")

This shows what the methods do, but for many uses, it would be more useful to have each of the contacts only show up once - for that, please refer to see All contacts.

All contacts
============

Sometimes you may want to inspect all the different contacts in the mu database. This is useful, for instance, when exporting contacts to some external format that can then be important in an e-mail program.

To enable this, there is the function mu:for-each-contact, defined as

(mu:for-each-contact function [search-expression]).

This will aggregate the unique contacts from all messages matching <search-expression> (when it is left empty, it will match all messages in the database), and execute function for each of them.

The function receives an object of the type <mu:contact-with-stats>, which is a subclass of the <mu:contact> class discussed in see Contact functions and objects. <mu:contact-with-stats> objects expose the following additional methods:

  * (mu:frequency <contact>): returns the number of times this contact occured in one of the address fields

  * (mu:last-seen <contact>): returns the most recent time the contact was seen in one of the address fields, as a time_t value

The method assumes an e-mail address is unique for a certain contact; if a certain e-mail address occurs with different names, it uses the most recent non-empty name.

Utility functions
=================

To make dealing with contacts even easier, there are a number of utility functions that can save you a bit of typing.

For converting contacts to some textual form, there is (mu:contact->string <mu:contact> format), which takes a contact and returns a text string with the given format. Currently supported formats are "org-contact, "mutt-alias", "mutt-ab", "wanderlust" and "plain".

Example - mutt export
=====================

Let's see how we could export the addresses in the mu database to the addressbook format of the venerable mutt (http://www.mutt.org/) e-mail client.

The addressbook format that mutt uses is a sequence of lines that look something like:
alias <nick> [<name>] "<" <email> ">"

Anyway, there is the function (mu:contact->string <mu:contact> format) that we can use to do the conversion.

We may want to focus on people with whom we have frequent correspondence; so we may want to limit ourselves to people we have seen at least 10 times in the last year.

It is a bit hard to guess the nick name for e-mail contacts, but mu:contact->string tries something based on the name. You can always adjust them later by hand, obviously.

     #!/bin/sh
     exec guile -s $0 $
     !#

     (use-modules (mu))
     (mu:initialize)

     ;; Get a list of contacts that were seen at least 20 times since 2010
     (define (selected-contacts)
       (let  ((addrs '())
     	  (start (car (mktime (car (strptime "%F" "2010-01-01")))))
     	  (minfreq 20))
         (mu:for-each-contact
           (lambda (contact)
     	(if (and (mu:email contact)
     	      (>= (mu:frequency contact) minfreq)
     	      (>= (mu:last-seen contact) start))
     	  (set! addrs (cons contact addrs)))))
           addrs))

     (for-each
       (lambda (contact)
         (format #t "~a\n" (mu:contact->string contact "mutt-alias")))
       (selected-contacts))

This simple program could be improved in many ways; this is left as an excercise to the reader.

Attachments and other parts
***************************

To deal with attachments, or, more in general MIME-parts, there is the mu part module.

Parts and their methods
=======================

The module defines the <mu-part> class, and adds two methods to <mu:message> objects:

  * (mu:parts msg) - returns a list <mu-part> objects, one for each MIME-parts in the message.

  * (mu:attachments msg) - like parts, but only list those MIME-parts that look like proper attachments.

A <mu:part> object exposes a few methods to get information about the part:

  * (mu:name <part>) - returns the file name of the mime-part, or #f if there is none.

  * (mu:mime-type <part>) - returns the mime-type of the mime-part, or #f if there is none.

  * (mu:size <part>) - returns the size in bytes of the mime-part

Then, we may want to save the part to a file; this can be done using either:

  * (mu:save part <part>) - save a part to a temporary file, return the file name (the temporary filename is a predictable function of (user-id, msg-path, part-index))

  * (mu:save-as <part> <path>) - save part to file at path

Attachment example
==================

Let's look at some small example. Let's get a list of the biggest attachments in messages about Luxemburg:

     #!/bin/sh
     exec guile -s $0 $
     !#

     (use-modules (mu))
     (mu:initialize)

     (define (all-attachments expr)
       "Return a list of (name . size) for all attachments in messages
     matching EXPR."
       (let ((pairs '()))
         (mu:for-each-message
           (lambda (msg)
     	(for-each
     	  (lambda (att) ;; add (filename . size) to the list
     	    (set! pairs (cons (cons (mu:name att) (or (mu:size att) 0)) pairs)))
     	  (mu:attachments msg)))
           expr)
         pairs))

     (for-each
       (lambda (att)
         (format #t "~a: ~,1fKb\n"
           (car att) (exact->inexact (/ (cdr att) 1024))))
       (sort (all-attachments "Luxemburg")
         (lambda (att1 att2)
           (< (cdr att1) (cdr att2)))))

As an exercise for the reader, you might want to re-rewrite the all-attachments in terms of mu:message-list, which would probably be a bit more elegant.

Statistics
**********

mu-guile offers some convenience functions to determine various statistics about the messages in the database.

(mu:tabulate <function> [<search-expr>]) applies <function> to each message matching <search-expr> (leave empty to match all messages), and returns a associative list (a list of pairs) with each of the different results of <function> and their frequencies.

This can best be demonstrated with a little example. Suppose we want to know how many messages we receive per weekday:

     #!/bin/sh
     exec guile -s $0 $
     !#

     (use-modules (mu) (mu stats) (mu plot))
     (mu:initialize)

     ;; create a list like (("Sun" . 13) ("Mon" . 23) ...)
     (define weekday-table
       (mu:weekday-numbers->names
         (sort
           (mu:tabulate
     	(lambda (msg)
     	  (tm:wday (localtime (mu:date msg)))))
           (lambda (a b) (< (car a) (car b))))))

     (for-each
       (lambda (elm)
         (format #t "~a: ~a\n" (car elm) (cdr elm)))
       weekday-table)

The function weekday-table uses mu:tabulate-message to get the frequencies per hour -- this returns a list of pairs:
((5 . 2339) (0 . 2278) (4 . 2800) (2 . 3184) (6 . 1856) (3 . 2833) (1 . 2993))

We sort these pairs by the day number, and then apply mu:weekday-numbers->names, which takes the list, and returns a list where the day numbers are replace by there abbreviated name (in the current locale). Note, there is also mu:month-numbers->names.

The script then outputs these numbers in the following form:

Sun: 2278
Mon: 2993
Tue: 3184
Wed: 2833
Thu: 2800
Fri: 2339
Sat: 1856

Clearly, Saturday is a slow day for e-mail...

Plotting data
*************

You can plot the results in the format produced by mu:tabulate with the (mu plot) module, an experimental module that requires the gnuplot (http://www.gnuplot.info/) program to be installed on your system.

The mu:plot function takes the following arguments:

(mu:plot <data> <title> <x-label> <y-label> [<want-ascii>])

Here, <data> is a table of data in the format that mu:tabulate produces. <title>, <x-label> and <y-lablel> are, respectively, the title of the graph, and the labels for X- and Y-axis. Finally, if you pass #t for the final <want-ascii> parameter, a plain-text rendering of the graph will be produced; otherwise, a graphical window will be shown.

An example should clarify how this works in practice; let's plot the number of message per hour:

     #!/bin/sh
     exec guile -s $0 $
     !#

     (use-modules (mu) (mu stats) (mu plot))
     (mu:initialize)

     (define (mail-per-hour-table)
       (sort
         (mu:tabulate
           (lambda (msg)
     	(tm:hour (localtime (mu:date msg)))))
         (lambda (x y) (< (car x) (car y)))))

     (mu:plot (mail-per-hour-table) "Mail per hour" "Hour" "Frequency" #t)

                                  Mail per hour
 Frequency
   1200 ++--+--+--+--+-+--+--+--+--+-+--+--+--+-+--+--+--+--+-+--+--+--+--++
        |+  +  +  +  + +  + "/tmp/fileHz7D2u" using 2:xticlabels(1) ********
   1100 ++                                                            *** +*
        ****                                                          * *  *
   1000 *+ *                                                       **** * +*
        *  *                           ******  ****                * ** *  *
    900 *+ *                           * ** ****  *             **** ** * +*
        *  *                           * ** *  *  *********     * ** ** *  *
    800 *+ *                        **** ** *  *  *  * ** *     * ** ** * +*
    700 *+ ***                   ****  * ** *  *  *  * ** ****  * ** ** * +*
        *  * *                ****  *  * ** *  *  *  * ** *  **** ** ** *  *
    600 *+ * ****             *  *  *  * ** *  *  *  * ** *  *  * ** ** * +*
        *  * ** *             *  *  *  * ** *  *  *  * ** *  *  * ** ** *  *
    500 *+ * ** *             *  *  *  * ** *  *  *  * ** *  *  * ** ** * +*
        *  * ** ****        ***  *  *  * ** *  *  *  * ** *  *  * ** ** *  *
    400 *+ * ** ** ****     * *  *  *  * ** *  *  *  * ** *  *  * ** ** * +*
        *+ *+**+**+* +*******+* +* +*+ *+**+* +*+ *+ *+**+* +*+ *+**+**+* +*
    300 ********************************************************************
         0  1  2  3  4 5  6  7  8  910 11 12 1314 15 16 17 1819 20 21 22 23
                                       Hour

GNU Free Documentation License
******************************

     [file fdl.texi not included]