lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Can not delete index file after close the IndexSearcher
Date Thu, 13 Jan 2011 10:41:39 GMT
As I said, there is probably a better solution.  At the moment you are
opening searchers at the top and bottom of the loop and on second and
subsequent passes you are not closing the bottom one, that you've only
just opened, before opening a new one using the same instance
variable.  The resources of the bottom one would presumably be
released eventually by GC, but evidently not soon enough,

Replace the top searcher = new IndexSearcher(dir); line with

if (needToOpenNewSearcher()) {
   ...
}

where the logic in needToOpenNewSearcher() is for you to write.


--
Ian.


2011/1/13 张志田 <zhitian.zhang@dianping.com>:
> Ian, thanks for your response. Your suggestion worked for me.
>
> What does oldSearcher.close() do in my code? why I have to close the
> searcher and oldSearcher together? In my opinion, oldSearcher held index1
> while searcher held index2, they are using different resources, the
> resources held by them should be released seperately.
>
> I have another concern for your solution, searcher is a reference created
> here for user searching out of this code snippet, if I closed and reopen it
> here, there may be some service down time because there is no open searcher
> for using.
>
> In my original code, searcher opened all the time, so there is no service
> down time or little, this is the reason I did not close it every time. Do
> you have any suggestion to keep an alive searcher and the program can also
> switch the index smoothly?
>
> Thanks,
> Garry
>
>
>
> 在 2011年1月13日 下午5:47,Ian Lea <ian.lea@gmail.com>写道:
>
>> Try adding
>>
>>      try { searcher.close(); } catch (Exception e) { }
>>
>> before
>>
>>      searcher = new IndexSearcher(dir);
>>
>> at the top of the loop.
>>
>> At the end of a loop searcher is open, and is not closed before being
>> reassigned.  There is probably a better solution along the lines of
>> only opening new searcher if need to.
>>
>>
>> --
>> Ian.
>>
>> 2011/1/13 张志田 <zhitian.zhang@dianping.com>:
>> > Hi Yuhan,
>> >
>> > dir.close() can not solve the problem.
>> >
>> > The reason I have to close the old searcher is my program will replace
>> the
>> > old index, the code posted here is just a scenario to simplify my
>> question.
>> >
>> > Thanks,
>> > Garry
>> >
>> > 在 2011年1月13日 上午10:45,Yuhan Zhang <yzhang@onescreen.com>写道:
>> >
>> >> Hi Garry,
>> >>
>> >> I am guessing the directory needs to be closed before opening a new one.
>> >>
>> >> dir.close();
>> >> dir = FSDirectory.open(new File(getIndexPath()));
>> >>
>> >> why not to open two IndexSearcher objects in an array of two instead of
>> >> swapping them back and forth?
>> >> it would be a lot easier.
>> >>
>> >> yuhan
>> >>
>> >> 2011/1/12 张志田 <zhitian.zhang@dianping.com>
>> >>
>> >> > Hi Mike,
>> >> >
>> >> > Sorry to make you confused. "lock" means the file handle is held by
>> some
>> >> > other progress, the program can not delete it. There is no exception,
>> I
>> >> can
>> >> > see file.delete() method returns false. If I delete the cfs file in
>> the
>> >> OS
>> >> > manually, the warning is "File was using by another person or program"
>> >> >
>> >> > To simplify my question, I made some more code for testing. you can
>> run
>> >> it
>> >> > for reproducing, after two loops, you will see the message e.g. "Can
>> not
>> >> > delete file: D:\index\index2\_0.cfs"
>> >> >
>> >> >
>> >> > Thank you very much
>> >> >
>> >> >
>> >> > public class SearchTest
>> >> > {
>> >> >
>> >> >    private static final int MAX_RESULT = 10000;
>> >> >
>> >> >    private String indexPath1 = "D:\\index\\index1";
>> >> >
>> >> >    private String indexPath2 = "D:\\index\\index2";
>> >> >
>> >> >     private String backupIndexpath = "D:\\index\\index3";
>> >> >
>> >> >    private String indexPath = indexPath1;
>> >> >
>> >> >    private Analyzer analyzer = new
>> StandardAnalyzer(Version.LUCENE_30);
>> >> >
>> >> >     private IndexSearcher searcher;
>> >> >
>> >> >    public void search()
>> >> >    {
>> >> >        while (true)
>> >> >        {
>> >> >            try
>> >> >            {
>> >> >                String keyword = "test";
>> >> >                String fieldName = "searchfield";
>> >> >
>> >> >                 Directory dir = FSDirectory.open(new File(indexPath));
>> >> >
>> >> >                searcher = new IndexSearcher(dir);
>> >> >
>> >> >                QueryParser queryParse = new
>> >> QueryParser(Version.LUCENE_30,
>> >> > fieldName, analyzer);
>> >> >                Query query = queryParse.parse(keyword);
>> >> >
>> >> >                TopDocs hits = searcher.search(query, MAX_RESULT);
>> >> >                int size = 5;
>> >> >                if (hits.scoreDocs.length < size)
>> >> >                {
>> >> >                    size = hits.scoreDocs.length;
>> >> >                }
>> >> >                for (int i = 0; i < size; i++)
>> >> >                {
>> >> >                    Document doc = searcher.doc(hits.scoreDocs[i].doc);
>> >> >                    String text = doc.get(fieldName);
>> >> >                    System.out.println("fieldContent is: " + text);
>> >> >                }
>> >> >
>> >> >                IndexSearcher oldSearcher = searcher;
>> >> >
>> >> >                 File newFile = new File(getIndexPath());
>> >> >                for (File file : newFile.listFiles())
>> >> >                {
>> >> >                    if (!file.delete())
>> >> >                    {
>> >> >                        System.out.println("Can not delete file: " +
>> >> > file.getAbsolutePath());
>> >> >                    }
>> >> >                }
>> >> >
>> >> >                // Copy index File from another folder to this folder
>> >> >                copyDir(new File(backupIndexpath), newFile);
>> >> >
>> >> >                Directory newDir = FSDirectory.open(newFile);
>> >> >                IndexSearcher newSearcher = new IndexSearcher(newDir);
>> >> >                 searcher = newSearcher;
>> >> >
>> >> >                oldSearcher.close();
>> >> >
>> >> >                System.out.println("Closed Searcher: " +
>> >> > oldSearcher.getIndexReader().directory().toString());
>> >> >
>> >> >                System.out.println("input 'Q' to quit testing...");
>> >> >                BufferedReader br = new BufferedReader(new
>> >> > InputStreamReader(System.in));
>> >> >
>> >> >                if (br.readLine().trim().equals("Q"))
>> >> >                {
>> >> >                    break;
>> >> >                }
>> >> >            }
>> >> >            catch (CorruptIndexException e)
>> >> >            {
>> >> >                e.printStackTrace();
>> >> >            }
>> >> >            catch (IOException e)
>> >> >            {
>> >> >                e.printStackTrace();
>> >> >            }
>> >> >            catch (ParseException e)
>> >> >            {
>> >> >                e.printStackTrace();
>> >> >            }
>> >> >        }
>> >> >    }
>> >> >
>> >> >    private String getIndexPath()
>> >> >    {
>> >> >        if (indexPath.equals(indexPath1))
>> >> >        {
>> >> >            indexPath = indexPath2;
>> >> >        }
>> >> >        else
>> >> >        {
>> >> >            indexPath = indexPath1;
>> >> >        }
>> >> >
>> >> >        return indexPath;
>> >> >    }
>> >> >
>> >> >     public static void copyDir(File sourceLocation, File
>> targetLocation)
>> >> > throws IOException
>> >> >    {
>> >> >        String[] children = sourceLocation.list();
>> >> >        for (int i = 0; i < children.length; i++)
>> >> >        {
>> >> >            InputStream in = null;
>> >> >            OutputStream out = null;
>> >> >            try
>> >> >            {
>> >> >                in = new FileInputStream(new File(sourceLocation,
>> >> > children[i]));
>> >> >                out = new FileOutputStream(new File(targetLocation,
>> >> > children[i]));
>> >> >
>> >> >                byte[] buf = new byte[1024];
>> >> >                int len;
>> >> >                while ((len = in.read(buf)) > 0)
>> >> >                {
>> >> >                    out.write(buf, 0, len);
>> >> >                }
>> >> >            }
>> >> >            catch (FileNotFoundException e)
>> >> >            {
>> >> >                e.printStackTrace();
>> >> >            }
>> >> >            catch (IOException ioe)
>> >> >            {
>> >> >                ioe.printStackTrace();
>> >> >            }
>> >> >            finally
>> >> >            {
>> >> >                try
>> >> >                {
>> >> >                    if (in != null)
>> >> >                    {
>> >> >                        in.close();
>> >> >                    }
>> >> >                    if (out != null)
>> >> >                    {
>> >> >                        out.close();
>> >> >                     }
>> >> >                }
>> >> >                catch (IOException e)
>> >> >                {
>> >> >                    e.printStackTrace();
>> >> >                }
>> >> >            }
>> >> >        }
>> >> >    }
>> >> >
>> >> >     public static void main(String[] args)
>> >> >    {
>> >> >        SearchTest searchTest = new SearchTest();
>> >> >        searchTest.search();
>> >> >    }
>> >> >
>> >> > }
>> >> >
>> >> > 在 2011年1月12日 下午11:53,Michael McCandless <lucene@mikemccandless.com
>> >写道:
>> >> >
>> >> > > Hmmm.
>> >> > >
>> >> > > When you say "locked" what actually does that mean?  Can you post
>> the
>> >> > > exception?
>> >> > >
>> >> > > Also, can you whittle down your example even more?   EG if calling
>> >> > > this method twice causes the problem, make a method that calls
it
>> >> > > twice and hits the exception and then start simplifying from
>> there...
>> >> > >
>> >> > > Mike
>> >> > >
>> >> > > 2011/1/12 张志田 <zhitian.zhang@dianping.com>:
>> >> > > > Mike, thanks for your feedback.
>> >> > > >
>> >> > > > I verified this in the debug mode, so I just check the folder
I
>> >> closed
>> >> > in
>> >> > > > the last loop. Actually, both two folders are locked.
>> >> > > >
>> >> > > > tried with new FSDirectory every loop, no help.
>> >> > > >
>> >> > > > Garry
>> >> > > >
>> >> > > > 2011/1/12 Michael McCandless <lucene@mikemccandless.com>
>> >> > > >
>> >> > > >> When you break out of the loop (user enters 'Q') you
don't close
>> the
>> >> > > >> current searcher.  Could that be it?
>> >> > > >>
>> >> > > >> Also you are calling FSDir.open each time but should
only do it
>> once
>> >> > > >> (though this should be "harmless").
>> >> > > >>
>> >> > > >> Mike
>> >> > > >>
>> >> > > >> On Wed, Jan 12, 2011 at 5:39 AM, 张志田 <zhitian.zhang@dianping.com
>> >
>> >> > > wrote:
>> >> > > >> > Dear Luceners,
>> >> > > >> >
>> >> > > >> > I'm using lucene-3.0.2 in our app. There is some
testing code
>> for
>> >> > > >> switching
>> >> > > >> > index, however, when my code run a couple of times,
I found the
>> >> > index
>> >> > > >> file
>> >> > > >> > was locked, I can not delete the old index files.
>> >> > > >> >
>> >> > > >> >
>> >> > > >> > The code looks like:
>> >> > > >> >
>> >> > > >> > public class SearchTest
>> >> > > >> > {
>> >> > > >> >
>> >> > > >> >    private static final int MAX_RESULT = 10000;
>> >> > > >> >
>> >> > > >> >    private String indexPath1 = "D:\\index\\index1";
>> >> > > >> >    private String indexPath2 = "D:\\index\\index2";
>> >> > > >> >
>> >> > > >> >    private String indexPath = indexPath1;
>> >> > > >> >
>> >> > > >> >    private Analyzer analyzer = new
>> >> > > StandardAnalyzer(Version.LUCENE_30);
>> >> > > >> >
>> >> > > >> >    private Directory dir = null;
>> >> > > >> >
>> >> > > >> >    private IndexSearcher searcher;
>> >> > > >> >
>> >> > > >> >    public void search()
>> >> > > >> >    {
>> >> > > >> >        while(true)
>> >> > > >> >        {
>> >> > > >> >            try
>> >> > > >> >            {
>> >> > > >> >                String keyword = "test";
>> >> > > >> >                String fieldName = "searchfield";
>> >> > > >> >
>> >> > > >> >                if(dir == null)
>> >> > > >> >                {
>> >> > > >> >                    dir = FSDirectory.open(new File(indexPath));
>> >> > > >> >                }
>> >> > > >> >                searcher = new IndexSearcher(dir);
>> >> > > >> >
>> >> > > >> >                QueryParser queryParse = new
>> >> > > >> QueryParser(Version.LUCENE_30,
>> >> > > >> > fieldName, analyzer);
>> >> > > >> >                Query query = queryParse.parse(keyword);
>> >> > > >> >
>> >> > > >> >                TopDocs hits = searcher.search(query,
>> MAX_RESULT);
>> >> > > >> >                int size = 5;
>> >> > > >> >                if(hits.scoreDocs.length < size)
>> >> > > >> >                {
>> >> > > >> >                    size = hits.scoreDocs.length;
>> >> > > >> >                }
>> >> > > >> >                for (int i = 0; i < size; i++)
>> >> > > >> >                {
>> >> > > >> >                    Document doc =
>> >> > searcher.doc(hits.scoreDocs[i].doc);
>> >> > > >> >                    String text = doc.get(fieldName);
>> >> > > >> >                    System.out.println("fieldContent
is: " +
>> text);
>> >> > > >> >                }
>> >> > > >> >
>> >> > > >> >                IndexSearcher oldSearcher = searcher;
>> >> > > >> >                dir = FSDirectory.open(new
>> File(getIndexPath()));
>> >> > > >> >                IndexSearcher newSearcher = new
>> IndexSearcher(dir);
>> >> > > >> >                searcher = newSearcher;
>> >> > > >> >
>> >> > > >> >                oldSearcher.close();
>> >> > > >> >                System.out.println("Closed Searcher:
" +
>> >> > > >> > oldSearcher.getIndexReader().directory().toString());
>> >> > > >> >
>> >> > > >> >                System.out.println("input 'Q' to
quit
>> testing...");
>> >> > > >> >                BufferedReader br = new BufferedReader(new
>> >> > > >> > InputStreamReader(System.in));
>> >> > > >> >
>> >> > > >> >                if(br.readLine().trim().equals("Q"))
>> >> > > >> >                {
>> >> > > >> >                    break;
>> >> > > >> >                }
>> >> > > >> >            }
>> >> > > >> >            catch (CorruptIndexException e)
>> >> > > >> >            {
>> >> > > >> >                e.printStackTrace();
>> >> > > >> >            }
>> >> > > >> >            catch (IOException e)
>> >> > > >> >            {
>> >> > > >> >                e.printStackTrace();
>> >> > > >> >            }
>> >> > > >> >            catch (ParseException e)
>> >> > > >> >            {
>> >> > > >> >                e.printStackTrace();
>> >> > > >> >            }
>> >> > > >> >        }
>> >> > > >> >    }
>> >> > > >> >
>> >> > > >> >    private String getIndexPath()
>> >> > > >> >    {
>> >> > > >> >        if(indexPath.equals(indexPath1))
>> >> > > >> >        {
>> >> > > >> >            indexPath = indexPath2;
>> >> > > >> >        }
>> >> > > >> >        else
>> >> > > >> >        {
>> >> > > >> >            indexPath = indexPath1;
>> >> > > >> >        }
>> >> > > >> >
>> >> > > >> >        return indexPath;
>> >> > > >> >    }
>> >> > > >> >
>> >> > > >> >    public static void main(String[] args)
>> >> > > >> >    {
>> >> > > >> >        SearchTest searchTest = new SearchTest();
>> >> > > >> >        searchTest.search();
>> >> > > >> >    }
>> >> > > >> >
>> >> > > >> > }
>> >> > > >> >
>> >> > > >> > Can anybody take a look at the above code snippet?
>> >> > > >> >
>> >> > > >> > I want to search on the different index file every
time so I
>> >> created
>> >> > > two
>> >> > > >> > different folders and switch them time to time.
The index files
>> in
>> >> > the
>> >> > > >> > index1/index2 maybe replaced before the search request
comes.
>> >> > > >> >
>> >> > > >> > The problem I found is after I ran the above code
2 or more
>> loops,
>> >> I
>> >> > > can
>> >> > > >> not
>> >> > > >> > modify/delete the cfs/cfx file in the file system(Windows
>> 2003),
>> >> > > although
>> >> > > >> I
>> >> > > >> > closed the searcher every time in the code. It seems
that the
>> >> index
>> >> > > file
>> >> > > >> is
>> >> > > >> > not released.
>> >> > > >> >
>> >> > > >> > Is the problem caused by the shared reference of
searcher? or
>> some
>> >> > > shared
>> >> > > >> > thread in the lucene?
>> >> > > >> >
>> >> > > >> > Thanks in advance!
>> >> > > >> > Garry
>> >> > > >> >
>> >> > > >>
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > --
>> >> > > > 张志田
>> >> > > >
>> >> > > > 大众点评网 - 技术部
>> >> > > > 电话:52521070 - 1675
>> >> > > >
>> >> > >
>> >> > >
>> ---------------------------------------------------------------------
>> >> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> > > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >> > >
>> >> > >
>> >> >
>> >> >
>> >> > --
>> >> > 张志田
>> >> >
>> >> > 大众点评网 - 技术部
>> >> > 电话:52521070 - 1675
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > 张志田
>> >
>> > 大众点评网 - 技术部
>> > 电话:52521070 - 1675
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message