lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nikhil desai <>
Subject Re: Lucene Indexes explanantion
Date Tue, 11 Jun 2013 00:06:36 GMT
Sure. Thanks Jack.
I don't have much experience working with Lucene, however, here is what I
am trying to resolve.

I learned that the Custom attributes cannot be used for indexing or
searching purposes. However I wanted the attributes to be used for indexing
and searching. So I created custom attributes and inserted them as tokens
into the tokenstream by assigning positionIncrement attribute to 0. Now
since my new token stream has attributes(as tokens) and they are used while
indexing, I can now search the document based on the attributes(tokens I
newly inserted). However I still have an issue. And by the way I have a lot
of attributes that I need to assign to an individual token.

Ex: Sentence: "LinkedIn is famous"
After passing through custom analyzer and few filters that I have written
and appending Attributes to the tokens, the new Tokenstream we get is
"LinkedIn Noun SocialSite famous JJ Positive" - (what that means is that
LinkedIn is Noun and is also an Socialsite, famous is an adjective and also
a Positive word, 'is' is removed as it does not make sense to index 'is')

This is now definitely searchable based on Attributes(Here: Noun,
SocialSite, JJ, Positive).

However, since I have put this entire text "LinkedIn is famous" as a Field
while adding a Document, when I search for say "SocialSite", I get a
Document as an output which has "LinkedIn is famous" as one of the fields.

However, is it possible to get only "LinkedIn" as output rather than an
entire text? i.e Only the actual token(the token present in the original
input) as output?
Another example: if I search for say "Positive" I should get "famous" as
output and not the entire "LinkedIn is famous".

I know that if I put it as a Field in the document, I should be able to get
it, but how do I add such a Field? because, only when the Tokens are passed
through the filters we get to know what all Attributes would be attached to
it, so while we do indexwriter.addDocument() we have no idea about the

The typical problem that I see is the indexing is done based on the new
tokenstream which is good, but when it retrieves the Document, it has the
older actual Tokenstream(or actual input) and that is what is given as

Does that make any sense? Or I have a typical use case that does not go
well with Lucene?

Any help comments are appreciated.

On Mon, Jun 10, 2013 at 1:32 PM, Jack Krupansky <>wrote:

> Even though you've posted for Lucene, you might want to consider taking a
> look at Solr because Solr has an Admin UI with an Analysis page which gives
> you a nice display of how index and query text is analyzed into tokens,
> terms, and attributes - all of which Solr inherits from Lucene.
> And check out the unit tests for Lucene (and Solr) for indexing. Then you
> can actually step through code and see it happen.
> Otherwise, google for blogs on various sub-topics of interest with
> specific terms.
> OTOH... don't try diving too deeply until you've written and understood a
> fair amount of Java code using Lucene. Otherwise, you won't have enough
> context to understand or even ask intelligent questions.
> -- Jack Krupansky
> -----Original Message----- From: nikhil desai
> Sent: Monday, June 10, 2013 1:24 PM
> To:
> Subject: Lucene Indexes explanantion
> Hello,
> My first time post in this group.
> I have been using Lucene recently. I have a question.
> Where can I find a good explanation on Indexes. Or rather how indexing (Not
> really the mathematical aspect) happens in Lucene, what all
> attributes(charTerm, Offset etc) come into play? And the way it is
> implemented? I checked the "Lucene In Action" and could not find much on
> actual indexing, what all classes etc are being used.
> Appreciate your help.
> Thanks
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**<>
> For additional commands, e-mail: java-user-help@lucene.apache.**org<>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message