Email search in mail clients

I have been researching whether it's necessary to implement a Virtual "All Mail" directory in the Cloudron Mail Server. Cloudron already implements Full Text Search of all emails via Solr. As part of this research, I wanted to answer the following questions:

• How does the mail client perform search?
• What's the best way to search in the mail client?
• Will the mail client benefit from having a "All Mails" folder?
• Should all users be automatically subscribed to "All Mails" folder?
• What is the current status of search in the mail client (bug reports, issues etc)?

Search Index

On the server side, we use Dovecot as our IMAP server. IMAP has a "SEARCH" command. Per the RFC 3501, the SEARCH command is a case-insensitive substring operation. By default, Dovecot does a slow sequential search through all messages.

To speed up search, Dovecot can index messages using the Solr FTS Engine. Solr has a "schema" which specifies the kind of index it maintains. A schema can specify the min/max length of works, min/max ngram length for start/end string matching, stop words, stemming algorithm, word delimiters etc. A number of compromises need to made to keep the size of the index reasonable. On Cloudron, we went with:

• Minimum 3 letters needed for a search
• Can only search for start of a string (EdgeNGramFilter)
• Uses Porter Stem

IMAP Search

While we maintain a search index on the server, we are still reliant on the email client to actually issue the SEARCH command. The SEARCH command only searches a single folder and is not recursive. The mail client has to recurse each folder, issue SEARCH command and collate the results.

The MULTISEARCH command in RFC 7377 allows searching multiple folders in a single shot, but there is no support for this in Dovecot yet. This mail thread suggests creating a virtual folder which has all the mails to search in.

Adding the virtual folder in Dovecot is easy enough, but do mail clients support searching nicely with it? Let's see.

Roundcube

Roundcube is able to search recursively using IMAP SEARCH. For this, one has to select "Entire message" and scope "All folders" before searching.

Roundcube shows only subscribed folders. If the "All Mails" folder was auto-subscribed the search results appear twice (despite the "All Mails" folder having the \ALL flag).

Unfortunately, there is no way keep the search options selected to 'Entire Message' and 'All folders' by default. We have submitted a patch upstream to remedy this.

SOGo

SOGo is able to search recursively using IMAP SEARCH . For this, you have to click on the three dots to the right of the INBOX, select BODY in criteria and search.

A caveat: SOGo shows all folders by default. This means that All Mails folder will appear even if not subscribed automatically. If you search using the three dots to the right of the mailbox name, search results appear twice.

A nice touch in SOGo UI is that it shows the folder name WIP in the search results.

SnappyMail

SnappyMail does not implement recursive SEARCH . However, it's the only client I am aware of that implements MULTISEARCH. Unfortunately, there is no way to test this because Dovecot doesn't implement it.

The suggested solution for SnappyMail is to search by selecting the virtual All Mails folder. Caveat: the free flowing search input box does not search the email body! You can click the options arrow to the right on the search input box and put the text to search in the body field in the resulting popup. Quite cumbersome but functional. We have made a suggestion to search the body by default here .

K-9

In K-9 , searches are local by default. After you search once, One clicks the "cloud icon" in top of the search results bar to get server side results. Unfortunately, the devil is the details and the functionality barely works:

• The server search does not search the message body.
• The server search does not recurse.
• On the plus side, lots of search functionality is planned.

Thunderbird

Thunderbird has a multitude of ways to search:

• Global Search (Ctrl+K)
• Quick Filter bar (Ctrl+Shift+K)
• Classic Search (Ctrl+Shift+F)

The first two searches only search locally (GLODA or Global Database in thunderbird lingo).

The Classic Search has options to search remotely and recursively. The search results also show the folder in which the mail was found. The Open in Folder button will also open the corresponding folder and mail in the main window.

There is a long open bug report on make the QuickSearch automatically use Server side search. It was opened 13 years ago, so I won't hold my breath for that one :-)

Conclusion

Of the analyzed mail clients, only SnappyMail can benefit from the All Mails virtual folder. Rest of them can already do recursive search or in the case of K-9 cannot search even if we had the virtual folder.

We have decided to add the All Mails folder considering that maybe there are other mail clients that can benefit from it. At least, it's not harming the ones we analyzed.