lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Haxby <>
Subject Re: indexing emails
Date Mon, 19 Jun 2006 11:48:09 GMT
Michael J. Prichard wrote:
> I am working on indexing emails and want to have a "to" field.  I am 
> currently putting all the emails on one line seperated w/ 
> spaces...example:
> Then i index that with a StandardAnalyzer as follows:
> doc.add(new Field("to", (String) itemContent.get("to"), 
> Field.Store.YES, Field.Index.UN_TOKENIZED));
> Question this the best way to do it?  I want to be able to 
> search for and pick out just those Documents, etc.
I took a slightly different approach.   Using javamail, given a To: line 
like this:

    To: Fred Smith <>, 
=?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?= <>

I re-constructed the address list to look like this:

    Fred Smith Keld Jørn Simonsen

and fed that to the analyser.   I forget which analyser we eventually 
settled on, but the "" turns into the tokens "fred" 
"example" and "com".   This actually gives rise to a remarkably natural 
way of search for
addresses.   People do things like searching for "" to 
look for mail sent to the lucene lists, they search for me variously as 
"jch", "john haxby" and "haxby"; they even, occasionally, search for 
complete mail addresses.   They all work.

The RFC2047 syntax in the example above gives one hint as to the 
minefield that address parsing can be.   If you look at the javamail 
spec, you'll also see reference to group-syntax -- it's often seen as


but you'll also occasionally see


Javamail knows how to parse these and I threw away the group name and 
just indexed the messages.   It might've been better to keep the group 
name, but groups aren't that widely used so it probably doesn't make 
much difference.

Other heads cause headaches as well.   Things like the subject can be 
RFC2047 encoded so you'll need to decode them.  The various message-id 
headers are also slightly problematic.   If you're using "message-id" 
and "references" and "in-reply-to" you'll need to be careful -- the 
individual message-id's will need their angle brackets removed and they 
really ought not to be tokenized.

It's also worth indexing *all* the message headers.   People do do 
searches on some odd things.   I also index the raw content-type as well 
-- those huge presentations can be found and deleted by searching for 
"content-type:application/".   Or at least I could.  It 
seems to be broken at the moment :-(


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message