lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: Spelt, for better spelling correction
Date Thu, 22 Mar 2007 06:41:47 GMT
Martin,
This sounds like the spellchecker dictionary needs to be built in parallel with the main Lucene
index.  Is it possible to create a dictionary out of an existing (and no longer modified)
Lucene index?

Otis
 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share

----- Original Message ----
From: Martin Haye <m1@snyder-haye.com>
To: Yonik Seeley <yonik@apache.org>
Cc: java-user@lucene.apache.org
Sent: Wednesday, March 21, 2007 2:03:50 PM
Subject: Re: Spelt, for better spelling correction

The dictionary is generated from the corpus, with the result that a larger
corpus gives better results.

Words are queued up during an index run, and at the end are munged to create
an optimized dictionary. It also supports incremental building, though the
overhead would be too much for those applications that are continuously
adding things to an index. Happily, it's not as important to keep the
spelling dictionary absolutely up to date, so it would be fine to queue
words over several index runs, and refresh the dictionary less often.

--Martin

On 3/20/07, Yonik Seeley <yonik@apache.org> wrote:
>
> Sounds interesting Martin!
> Is the dictionary static, or is it generated from the corpus or from
> user queries?
>
> -Yonik
>
> On 3/20/07, Martin Haye <martin.haye@gmail.com> wrote:
> > As part of XTF, an open source publishing engine that uses Lucene, I
> > developed a new spelling correction engine specifically to provide "Did
> you
> > mean..." links for misspelled queries. I and a small group are preparing
> > this for submission as a contrib module to Lucene. And we're inviting
> > interested people to join the discussion about it.
> >
> > The new engine is being called "Spelt" and differs from the one
> currently in
> > Lucene contrib in the following ways:
> >
> > - More accurate: Much better performance on single-word queries (90%
> correct
> > in #1 slot in my tests). On general list including multi-word queries,
> gets
> > 80%+ correct.
> > - Multi-word: Handles and corrects multi-word queries such as
> "harrypotter"
> > -> "harry potter".
> > - Fast: In my tests, builds the dictionary more than 30 times faster.
> > - Small: Dictionary size is roughly a third of that built by the
> existing
> > engine.
> > - Other bells and whistles...
> >
> > There is already a standalone test program that people can try out, and
> > we're interested in feedback. If you're interested in discussing,
> testing,
> > or previewing, consider joining the Google group:
> > http://groups.google.com/group/spelt/
> >
> > --Martin
> >
>




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message