* mm

I haven't written many =emacs-fu= posts recently, but that doesn't mean I
haven't used emacs a lot. In fact, over the last few months I've been working on
a bigger emacs-related project; the working title is =mm=, and it's an
emacs-based e-mail client based on my [[http://www.djcbsoftware.nl/code/mu][mu]] maildir searcher/indexer that I
discussed before. Even though I've been using =mm= myself for about two months,
it's not really ready from prime-time yet - but I'm planning to have something
ready this year still. 

In this post, let me discuss some of the technical background, which may be
interesting for others planning emacs-based front-ends to other tools.

* How to make an emacs-based e-mail client

  Emacs does not (as of today) support threads; but one way to do asynchronous
  processing is to start another process, and let emacs deal with its
  output. Let us see how...
  
** Getting output from =mu=
  
   One way to implement this (for =mu=), is to call the =mu= command-line tool
   with some parameters and then parse its output. In fact, that is how some
   tools do it, and it was my first approach - so I would invoke =mu find= and
   then process the output in emacs (more about that in a minute).

   However, then I realized that I'd need to load the entire e-mail Xapian
   database for each invocation. Wouldn't it be nicer to keep a running =mu=
   instance around?  Indeed, it would - so I implemented the =mu server=
   sub-command. Now, when you run =mu server=, you get a shell, in which you can
   give commands to =mu=, and which will then spit out the results. =mu server=
   is not really meant for humans, but still I can use it manually, which is
   great for debugging.

   The next question was what format mu should use for its output for emacs to
   process. Some other programs use =JSON= here, but I figured that it would be
   easier (and possibly, more efficient) just to use emacs' native
   =s-expressions= (=plists= to be precise). So that is what I did - and I can
   easily evaluate them using =read-from-string=.
  
** Processing the output in emacs
   
   So, now let's look how we process the data from =mu server= in emacs.

   First you create a process with, for example, =start-process=, and then
   register a filter function for it, which will be invoked whenever the process
   has some chunk of output. Something like:

#+BEGIN_SRC emacs-lisp
  (let ((proc (start-process <arguments>)))
    (set-process-filter proc 'my-process-filter)
    (set-process-sentinel proc 'my-process-sentinel))    
#+END_SRC
  
   Note, the process sentinel is invoked when the process is terminated -- so there
   you can clean things up.
   
   The function =my-process-filter= is a user-defined function that takes the
   process and the chunk of output as arguments; in =mm= it looks something like
   (pseudo-lisp):
#+begin_SRC emacs-lisp
(defun my-process-filter (proc str)
  (setq mm/buf (concat mm/buf str)) ;; a global var updated with the new chunk
  (when <we-have-received-a-full-expression>
      <eat-expression-from mm/buf> 
      <evaluate-expression>))
#+end_src

   The =<evaluate-expression>= de-multiplexes the s-expression we got. For example,
   if the s-expression looks like an e-mail message header, it will be processed by
   the header-handling function, which will append it to the header list. If the
   s-expression looks like an error message, it will be reported to the user. And
   so on.

   Finally, let me try to answer some anticipated questions:
   
* Why does the world need yet another e-mail client?
  
  I don't the world needs another client, but I spend a *lot* of time
  (professionally and privately) with my e-mail client, so I'd like it to behave
  exactly like I want it to. An even more important goal for me was to write
  some bigger program in emacs lisp, to better understand the language and its
  idioms.

  Specifically, when it comes to emacs-based clients, I have tried a few of
  them. I never really got into =gnus=; I think it is by far the most popular
  emacs-based mail client, but I found it hard to make behave the way I like it;
  and in particular, I do not like its indirect approach to Maildirs.

  Then, for some years I've been using Wanderlust; a fine, very feature-rich
  client, but it shows its age - and especially with emacs-24, its cache file
  got corrupted very often, requiring me to delete them etc. Still, you will
  recognize some Wanderlust features in =mm/mu=.

* Why not use [[http://notmuchmail.org/][notmuch]]? It seems similar.

  There are certainly similarities with =notmuch= (and to some lesser extent,
  with [[https://github.com/nicferrier/md][md]]) -- the overall architecture is similar: both are scanning maildirs,
  using [[http://spruce.sourceforge.net/gmime/][GMime]] and [[http://xapian.org/][Xapian]]. (=mu= precedes =notmuch= by a year or so; but
  =notmuch= was the first to add an emacs front-end).
  
  There are some differences as well. The main thing is that in =notmuch='s
  philosophy, messages are usually not moved or deleted, but instead uses tags
  in the database. While tags are nice, I like the 'state' to be in the messages
  and the folders they are in, which it easy to synchronize with other email
  clients (or synchronize with IMAP-folders through [[http://offlineimap.org/][OfflineIMAP]]). I'd like to be
  able to move messages around, delete messages and so on. This is in fact one
  of the things I liked in [[http://www.gohome.org/wl/][Wanderlust]], and wouldn't want to live without - so
  =mu=/=mm= make this really easy.
  
  Clearly, the emacs-interface to =notmuch= is more mature, and the development
  team is bigger, so I'd give it a try. On the other hand, if you happen to like
  e-mail the way I like it, =mm= may be something for you.






