lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Hall <mh...@informatics.jax.org>
Subject Re: Searching index problems with tomcat
Date Fri, 22 May 2009 14:49:56 GMT
For writing indexes?

Well I guess it depends on what you want.. but I personally use this:

(2.3.2 API)

File INDEX_DIR = "/data/searchtool/thisismyindexdirectory"
Analyzer analyzer = new WhateverConcreteAnalyzerYouWant();

writer = new IndexWriter(/INDEX_DIR/, /analyzer/, true);

Your best bet would be to peruse the API docs of whatever lucene version 
you are using.

However, I'm still pretty sure this ISN'T your actual issue here.

Looking at your "full path" example those still seem to be by reference 
to me. Let me be more specific and tell you EXACTLY what I mean by that,

Lets say you are running your program in the following directory:

/home/test/app/

Trying to open an index like you have below will effectively be trying 
to open an index in the following location:

/home/test/app/home/marco/RdfIndexLucene

What I think you MEAN to be doing is:

/home/marco/RdfIndexLucene

That leading slash is VERY VERY important, as its the entire difference 
between an relative path and an absolute one.

Matt

Marco Lazzara wrote:
> I was talking with my teacher.
> Is it correct to use FSDirectory?Could you please look again at the code
> I've posted here??
> Should I choose a different way to Indexing ??
>
> Marco Lazzara
>
>
>
>
> 2009/5/22 Ian Lea <ian.lea@gmail.com>
>
>   
>> OK.  I'd still like to see some evidence, but never mind.
>>
>> Next suggestion is the old standby - cut the code down to the absolute
>> minimum to demonstrate the problem and post it here.  I know you've
>> already posted some code, but maybe not all of it, and definitely not
>> cut down to the absolute minimum.
>>
>>
>> --
>> Ian.
>>
>>
>> On Thu, May 21, 2009 at 10:48 PM, Marco Lazzara <marco.lazzara@gmail.com>
>> wrote:
>>     
>>> _I strongly suggest that you use a full path name and/or provide some
>>> evidence that your readers and writers are using the same directory
>>> and thus lucene index.
>>> _
>>> I try a full path like home/marco/RdfIndexLucene,even
>>> media/disk/users/fratelli/RDFIndexLucene.But nothing is changed.
>>>
>>> MARCOLAZZARA
>>> _
>>>
>>> _
>>>       
>>>> Its been a few days, and we haven't heard back about this issue, can
>>>> we assume that you fixed it via using fully qualified paths then?
>>>>
>>>> Matt
>>>>
>>>> Ian Lea wrote:
>>>>         
>>>>> Marco
>>>>>
>>>>>
>>>>> You haven't answered Matt's question about where you are running it
>>>>> from.  Tomcat's default directory may well not be the same as yours.
>>>>> I strongly suggest that you use a full path name and/or provide some
>>>>> evidence that your readers and writers are using the same directory
>>>>> and thus lucene index.
>>>>>
>>>>>
>>>>> --
>>>>> Ian.
>>>>>
>>>>>
>>>>> On Wed, May 20, 2009 at 9:59 AM, Marco Lazzara
>>>>> <marco.lazzara@gmail.com> wrote:
>>>>>
>>>>>           
>>>>>> I've posted the indexing part,but I don't use this in my app.After
I
>>>>>> create the index,I put that in a folder like
>>>>>>             
>> /home/marco/RDFIndexLucece
>>     
>>>>>> and when I run the query I'm only searching (and not indexing).
>>>>>>
>>>>>> String[] fieldsearch = new String[] {"name", "synonyms", "propIn"};
>>>>>>    //RDFinder rdfind = new RDFinder("RDFIndexLucene/",fieldsearch);
>>>>>> TreeMap<Integer, ArrayList<String>> paths;
>>>>>> try {
>>>>>>                this.paths = this.rdfind.Search(text, "path");
>>>>>>            } catch (ParseException e1) {
>>>>>>                e1.printStackTrace();
>>>>>>            } catch (IOException e1) {
>>>>>>                e1.printStackTrace();
>>>>>>            }
>>>>>>
>>>>>> Marco Lazzara
>>>>>>
>>>>>>             
>>>>>>> Sorry, anyhow looking over this quickly here's a summarization
of
>>>>>>>               
>> what
>>     
>>>>>>> I see:
>>>>>>>
>>>>>>> You have documents in your index that look like the following:
>>>>>>>
>>>>>>> name which is indexed and stored.
>>>>>>> synonyms which are indexed and stored
>>>>>>> path, which is stored but not indexed
>>>>>>> propin, which is stored and indexed
>>>>>>> propinnum, which is stored but not indexed
>>>>>>> and ... vicinity I guess which is stored but not indexed
>>>>>>>
>>>>>>> For an analyzer you are using Standard analyzer (which considering
>>>>>>>               
>> all
>>     
>>>>>>> the Italian? is an interesting choice.)
>>>>>>>
>>>>>>> And you are opening your index using FSDirectory, in what appears
to
>>>>>>> be a by reference fashion (You don't have a fully qualified path
to
>>>>>>> where your index is, you are ASSUMING that its in the same directory
>>>>>>> as this code, unless FSDirectory is not implemented as I think
it
>>>>>>>               
>> is.)
>>     
>>>>>>> Now can I see the consumer code?  Specifically the part where
you are
>>>>>>> opening the index/constructing your queries?
>>>>>>>
>>>>>>> I'm betting what's going on here is you are deploying this as
a war
>>>>>>> file into tomcat, and its just not really finding the index as
a
>>>>>>> result of how the war file is getting deployed, but looking more
>>>>>>> closely at the source code should reveal if my suspicion is correct
>>>>>>> here.
>>>>>>>
>>>>>>> Also runtime wise, when you run your standalone app, where
>>>>>>> specifically in your directory structure are you running it from?
>>>>>>> Cause if you are opening your index reader/searcher in the same
way
>>>>>>>               
>> as
>>     
>>>>>>> you are creating your writer here, I'm pretty darn certain that
will
>>>>>>> cause you problems.
>>>>>>>
>>>>>>> Matt
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Marco Lazzara wrote:
>>>>>>>
>>>>>>>               
>>>>>>>> _Could you further post your Analyzer Setup/Query Building
code from
>>>>>>>> BOTH apps. _
>>>>>>>>
>>>>>>>> there is only one code.It is the same for web and for standalone.
>>>>>>>> And it is exactly the real problem!!the code is the same,libraries
>>>>>>>> are
>>>>>>>> the same,query index etc etc. are the same.
>>>>>>>>
>>>>>>>> This is the class that create index
>>>>>>>>
>>>>>>>>
>>>>>>>> public class AlternativeRDFIndexing {
>>>>>>>>        private Analyzer analyzer;
>>>>>>>>     private Directory directory;
>>>>>>>>     private IndexWriter iwriter;
>>>>>>>>     private WordNetSynonymEngine wns;
>>>>>>>>     private AlternativeResourceAnalysis rs;
>>>>>>>>     public ArrayList<String> commonnodes;
>>>>>>>>        //private RDFinder rdfind = new
>>>>>>>>                 
>> RDFinder("RDFIndexLucene/",new
>>     
>>>>>>>> String[] {"name"});
>>>>>>>>    //    public boolean Exists(String node) throws ParseException,
>>>>>>>> IOException{
>>>>>>>> //           //        return rdfind.Exists(node);
>>>>>>>> //    }
>>>>>>>>        public AlternativeRDFIndexing(String inputfilename)
throws
>>>>>>>> IOException, ParseException{
>>>>>>>>              commonnodes = new ArrayList<String>();
>>>>>>>>                // bisogna istanziare un oggetto per fare
analisi sul
>>>>>>>> documento rdf
>>>>>>>>         rs = new AlternativeResourceAnalysis(inputfilename);
>>>>>>>>
>>>>>>>>                    ArrayList<String> nodelist = rs.getResources();
>>>>>>>>         int nodesize = nodelist.size();
>>>>>>>>         ArrayList<String> sourcelist = rs.getsource();
>>>>>>>>         int sourcesize = sourcelist.size();
>>>>>>>>                //sinonimi
>>>>>>>>         wns = new WordNetSynonymEngine("sinonimi/");
>>>>>>>>                //creazione di un analyzer standard
>>>>>>>>         analyzer = new StandardAnalyzer();
>>>>>>>>
>>>>>>>>         //Memorizza l'indice in RAM:
>>>>>>>>                //Directory directory = new RAMDirector();
>>>>>>>>                //Memorizza l'indice su file
>>>>>>>>                directory =
>>>>>>>> FSDirectory.getDirectory("RDFIndexLucene/");
>>>>>>>>                //Creazione istanza per la scrittura dell'indice
>>>>>>>>         //Tale istanza viene fornita di analyzer, di un boolean
per
>>>>>>>> indicare se ricreare o meno da zero
>>>>>>>>         //la struttura e di una dimensione massima (o infinita
>>>>>>>> IndexWriter.MaxFieldLength.UNLIMITED)
>>>>>>>>         iwriter = new IndexWriter(directory, analyzer, true,
new
>>>>>>>> IndexWriter.MaxFieldLength(25000));
>>>>>>>>                       //costruiamo un indice con solo n documenti:
>>>>>>>>                 
>> un
>>     
>>>>>>>> documento per nodo
>>>>>>>>                for (int i = 0; i < nodesize; i++){
>>>>>>>>                        Document doc = new Document();
>>>>>>>>                        //creazione dei vari campi
>>>>>>>>                        // ogni documento avrˆ
>>>>>>>>             // un campo name: nome del nodo
>>>>>>>>             // indicazione di memorizzazione(Store.YES) e
>>>>>>>> indicizzazione
>>>>>>>> con analyzer(ANALYZED)
>>>>>>>>                        String node = nodelist.get(i);
>>>>>>>>                        //if (sourcelist.contains(node)) break;
>>>>>>>>                        //if (rdfind.Exists(node))
>>>>>>>> commonnodes.add(node);
>>>>>>>>                        Field field = new Field("name", node,
>>>>>>>> Field.Store.YES,Field.Index.ANALYZED);
>>>>>>>>             //Aggiunta campo al documento
>>>>>>>>             doc.add(field);
>>>>>>>>                        //Aggiungo i sinonimi
>>>>>>>>             String[] nodesynonyms = wns.getSynonyms(node);
>>>>>>>>             for (int is = 0; is < nodesynonyms.length;
is++) {
>>>>>>>>                                field = new Field("synonyms",
>>>>>>>> nodesynonyms[is],
>>>>>>>> Field.Store.YES,Field.Index.ANALYZED);
>>>>>>>>                 //Aggiunta campo al documento
>>>>>>>>                 doc.add(field);
>>>>>>>>             }
>>>>>>>>                        // uno o piu campi path_i: path minimali
>>>>>>>>                 
>> dalle
>>     
>>>>>>>> sorgenti al nodo
>>>>>>>>             // non indicizzati
>>>>>>>>                        for (int j = 0; j < sourcesize;
j++) {
>>>>>>>>             String source = sourcelist.get(j);
>>>>>>>> ArrayList<LinkedList<String>> path = new
>>>>>>>> ArrayList<LinkedList<String>>();
>>>>>>>>             try{
>>>>>>>>                            if ((source.equals(node)) ||
>>>>>>>> (sourcelist.contains(node))){
>>>>>>>>                     field = new Field("path", "null",
>>>>>>>> Field.Store.YES,
>>>>>>>> Field.Index.NO);
>>>>>>>>                     doc.add(field);
>>>>>>>>                 }
>>>>>>>>                 else{
>>>>>>>>                     path = rs.getPaths(source, node);
>>>>>>>>                     for (int ii = 0; ii < path.size();
ii++) {
>>>>>>>>                         String pp = rs.getPath(path.get(ii));
>>>>>>>>                         field = new Field("path", pp,
>>>>>>>> Field.Store.YES,
>>>>>>>> Field.Index.NO);
>>>>>>>>                         doc.add(field);                 
   }
>>>>>>>>                 }
>>>>>>>>                                }
>>>>>>>>             catch (IllegalArgumentException e){
>>>>>>>>                 System.out.println("source: "+source+ " node:
>>>>>>>> "+node);
>>>>>>>>                 field = new Field("path", "null", Field.Store.YES,
>>>>>>>> Field.Index.NO);
>>>>>>>>                 doc.add(field);
>>>>>>>>             }
>>>>>>>>                                              }
>>>>>>>>                        // proprietˆ entranti
>>>>>>>>             // indicizzati
>>>>>>>>           //versione con i sinonimi
>>>>>>>>                        ArrayList<String> y = rs.getInProperty(node);
>>>>>>>>                        if (y != null) {
>>>>>>>>
>>>>>>>>             for (int j = 0; j < y.size(); j++) {
>>>>>>>>                                String propin = y.get(j);
>>>>>>>>                 field = new Field("propIn", propin, Field.Store.YES,
>>>>>>>> Field.Index.ANALYZED);
>>>>>>>>                 doc.add(field);
>>>>>>>>             String[] propinsynonyms = wns.getSynonyms(propin);
>>>>>>>>                        for (int is = 0; is < propinsynonyms.length;
>>>>>>>> is++) {
>>>>>>>>                                field = new Field("propIn",
>>>>>>>> propinsynonyms[is],
>>>>>>>> Field.Store.YES,Field.Index.ANALYZED);
>>>>>>>>                 //Aggiunta campo al documento
>>>>>>>>                 doc.add(field);
>>>>>>>>                  }
>>>>>>>>                          }
>>>>>>>>                        // un campo num_propIn: numero di
proprietˆ
>>>>>>>> entranti
>>>>>>>>             // non indicizzato
>>>>>>>>                        String num_propIN = String.valueOf(y.size());
>>>>>>>>                        field = new Field("num_propIn", num_propIN,
>>>>>>>> Field.Store.YES,
>>>>>>>> Field.Index.NO);
>>>>>>>>             doc.add(field);
>>>>>>>>                        }
>>>>>>>>                        else {
>>>>>>>>                                String num_propIN =
>>>>>>>>                 
>> String.valueOf(0);
>>     
>>>>>>>>                                field = new Field("num_propIn",
>>>>>>>> num_propIN,
>>>>>>>> Field.Store.YES, Field.Index.NO);
>>>>>>>>                 doc.add(field);
>>>>>>>>                            }
>>>>>>>>                        // i vicini del nodo
>>>>>>>>                        ArrayList<String> v = rs.getVicini(node);
>>>>>>>>                        if (v != null) {
>>>>>>>>
>>>>>>>>             for (int j = 0; j < v.size(); j++) {
>>>>>>>>                                String vicino = v.get(j);
>>>>>>>>                 field = new Field("vicini", vicino, Field.Store.YES,
>>>>>>>> Field.Index.ANALYZED);
>>>>>>>>                 doc.add(field);                         
       }
>>>>>>>>                        }
>>>>>>>>                                                  //aggiunta
>>>>>>>>                 
>> documento
>>     
>>>>>>>> all'indice
>>>>>>>>             iwriter.addDocument(doc);
>>>>>>>>         }
>>>>>>>>                iwriter.close();
>>>>>>>>         directory.close();
>>>>>>>>            }
>>>>>>>>        public int getNR(){
>>>>>>>>         return rs.NumResource();
>>>>>>>>     }
>>>>>>>>
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> MARCO LAZZARA
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>>>> Things that could help us immensely here.
>>>>>>>>>
>>>>>>>>> Can you post your indexReader/Searcher initialization
code from
>>>>>>>>>                   
>> your
>>     
>>>>>>>>> standalone app, as well as your webapp.
>>>>>>>>>
>>>>>>>>> Could you further post your Analyzer Setup/Query Building
code from
>>>>>>>>> both apps.
>>>>>>>>>
>>>>>>>>> Could you further post the document creation code used
at indexing
>>>>>>>>> time? (Which analyzer, and which fields are indexed/stored)
>>>>>>>>>
>>>>>>>>> Give us this, and I'm pretty darn sure we can nail down
your issue.
>>>>>>>>>
>>>>>>>>> Matt
>>>>>>>>>
>>>>>>>>> Ian Lea wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>>>>> ...
>>>>>>>>>>> There are no exceptions.When I run the query
a new shell is
>>>>>>>>>>> displayed but
>>>>>>>>>>>  with no result.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>>>> New shell?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>>>>>>>>>>> _*Are you sure the index is the same - what do
>>>>>>>>>>> IndexReader.maxDoc(),
>>>>>>>>>>> numDocs() and getVersion() say, standalone
>>>>>>>>>>> and in tomcat?
>>>>>>>>>>>
>>>>>>>>>>> *_What do you mean with this question??
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                       
>>>>>>>>>> IndexReader ir = ...
>>>>>>>>>> System.out.printf("maxDoc=%s, ...", ir.maxDoc(),
...);
>>>>>>>>>>
>>>>>>>>>> and run in tomcat and standalone.  To absolutely
confirm you're
>>>>>>>>>> looking at the same index, and it has documents,
etc.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ian.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>> ---------------------------------------------------------------------
>>     
>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                     
>> ---------------------------------------------------------------------
>>     
>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> __________ Information from ESET NOD32 Antivirus, version
of virus
>>>>>>>>> signature database 4087 (20090519) __________
>>>>>>>>>
>>>>>>>>> The message was checked by ESET NOD32 Antivirus.
>>>>>>>>>
>>>>>>>>> http://www.eset.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> __________ Information from ESET NOD32 Antivirus, version
of virus
>>>>>>>> signature database 4087 (20090519) __________
>>>>>>>>
>>>>>>>> The message was checked by ESET NOD32 Antivirus.
>>>>>>>>
>>>>>>>> http://www.eset.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>> ---------------------------------------------------------------------
>>     
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> __________ Information from ESET NOD32 Antivirus, version of
virus
>>>>>>> signature database 4088 (20090519) __________
>>>>>>>
>>>>>>> The message was checked by ESET NOD32 Antivirus.
>>>>>>>
>>>>>>> http://www.eset.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> __________ Information from ESET NOD32 Antivirus, version of virus
>>>>>> signature database 4088 (20090519) __________
>>>>>>
>>>>>> The message was checked by ESET NOD32 Antivirus.
>>>>>>
>>>>>> http://www.eset.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>           
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>> __________ Information from ESET NOD32 Antivirus, version of virus
>>>> signature database 4093 (20090521) __________
>>>>
>>>> The message was checked by ESET NOD32 Antivirus.
>>>>
>>>> http://www.eset.com
>>>>
>>>>
>>>>         
>>>
>>> __________ Information from ESET NOD32 Antivirus, version of virus
>>>       
>> signature database 4094 (20090521) __________
>>     
>>> The message was checked by ESET NOD32 Antivirus.
>>>
>>> http://www.eset.com
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>       
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>     
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message