lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 张志田 <zhitian.zh...@dianping.com>
Subject Re: Can not delete index file after close the IndexSearcher
Date Thu, 13 Jan 2011 11:00:44 GMT
Ian, thank you very much.

I will try to change my switch solution.

Thanks again
Garry

在 2011年1月13日 下午6:41,Ian Lea <ian.lea@gmail.com>写道:

> As I said, there is probably a better solution.  At the moment you are
> opening searchers at the top and bottom of the loop and on second and
> subsequent passes you are not closing the bottom one, that you've only
> just opened, before opening a new one using the same instance
> variable.  The resources of the bottom one would presumably be
> released eventually by GC, but evidently not soon enough,
>
> Replace the top searcher = new IndexSearcher(dir); line with
>
> if (needToOpenNewSearcher()) {
>   ...
> }
>
> where the logic in needToOpenNewSearcher() is for you to write.
>
>
> --
> Ian.
>
>
> 2011/1/13 张志田 <zhitian.zhang@dianping.com>:
> > Ian, thanks for your response. Your suggestion worked for me.
> >
> > What does oldSearcher.close() do in my code? why I have to close the
> > searcher and oldSearcher together? In my opinion, oldSearcher held index1
> > while searcher held index2, they are using different resources, the
> > resources held by them should be released seperately.
> >
> > I have another concern for your solution, searcher is a reference created
> > here for user searching out of this code snippet, if I closed and reopen
> it
> > here, there may be some service down time because there is no open
> searcher
> > for using.
> >
> > In my original code, searcher opened all the time, so there is no service
> > down time or little, this is the reason I did not close it every time. Do
> > you have any suggestion to keep an alive searcher and the program can
> also
> > switch the index smoothly?
> >
> > Thanks,
> > Garry
> >
> >
> >
> > 在 2011年1月13日 下午5:47,Ian Lea <ian.lea@gmail.com>写道:
> >
> >> Try adding
> >>
> >>      try { searcher.close(); } catch (Exception e) { }
> >>
> >> before
> >>
> >>      searcher = new IndexSearcher(dir);
> >>
> >> at the top of the loop.
> >>
> >> At the end of a loop searcher is open, and is not closed before being
> >> reassigned.  There is probably a better solution along the lines of
> >> only opening new searcher if need to.
> >>
> >>
> >> --
> >> Ian.
> >>
> >> 2011/1/13 张志田 <zhitian.zhang@dianping.com>:
> >> > Hi Yuhan,
> >> >
> >> > dir.close() can not solve the problem.
> >> >
> >> > The reason I have to close the old searcher is my program will replace
> >> the
> >> > old index, the code posted here is just a scenario to simplify my
> >> question.
> >> >
> >> > Thanks,
> >> > Garry
> >> >
> >> > 在 2011年1月13日 上午10:45,Yuhan Zhang <yzhang@onescreen.com>写道:
> >> >
> >> >> Hi Garry,
> >> >>
> >> >> I am guessing the directory needs to be closed before opening a new
> one.
> >> >>
> >> >> dir.close();
> >> >> dir = FSDirectory.open(new File(getIndexPath()));
> >> >>
> >> >> why not to open two IndexSearcher objects in an array of two instead
> of
> >> >> swapping them back and forth?
> >> >> it would be a lot easier.
> >> >>
> >> >> yuhan
> >> >>
> >> >> 2011/1/12 张志田 <zhitian.zhang@dianping.com>
> >> >>
> >> >> > Hi Mike,
> >> >> >
> >> >> > Sorry to make you confused. "lock" means the file handle is held
by
> >> some
> >> >> > other progress, the program can not delete it. There is no
> exception,
> >> I
> >> >> can
> >> >> > see file.delete() method returns false. If I delete the cfs file
in
> >> the
> >> >> OS
> >> >> > manually, the warning is "File was using by another person or
> program"
> >> >> >
> >> >> > To simplify my question, I made some more code for testing. you
can
> >> run
> >> >> it
> >> >> > for reproducing, after two loops, you will see the message e.g.
> "Can
> >> not
> >> >> > delete file: D:\index\index2\_0.cfs"
> >> >> >
> >> >> >
> >> >> > Thank you very much
> >> >> >
> >> >> >
> >> >> > public class SearchTest
> >> >> > {
> >> >> >
> >> >> >    private static final int MAX_RESULT = 10000;
> >> >> >
> >> >> >    private String indexPath1 = "D:\\index\\index1";
> >> >> >
> >> >> >    private String indexPath2 = "D:\\index\\index2";
> >> >> >
> >> >> >     private String backupIndexpath = "D:\\index\\index3";
> >> >> >
> >> >> >    private String indexPath = indexPath1;
> >> >> >
> >> >> >    private Analyzer analyzer = new
> >> StandardAnalyzer(Version.LUCENE_30);
> >> >> >
> >> >> >     private IndexSearcher searcher;
> >> >> >
> >> >> >    public void search()
> >> >> >    {
> >> >> >        while (true)
> >> >> >        {
> >> >> >            try
> >> >> >            {
> >> >> >                String keyword = "test";
> >> >> >                String fieldName = "searchfield";
> >> >> >
> >> >> >                 Directory dir = FSDirectory.open(new
> File(indexPath));
> >> >> >
> >> >> >                searcher = new IndexSearcher(dir);
> >> >> >
> >> >> >                QueryParser queryParse = new
> >> >> QueryParser(Version.LUCENE_30,
> >> >> > fieldName, analyzer);
> >> >> >                Query query = queryParse.parse(keyword);
> >> >> >
> >> >> >                TopDocs hits = searcher.search(query, MAX_RESULT);
> >> >> >                int size = 5;
> >> >> >                if (hits.scoreDocs.length < size)
> >> >> >                {
> >> >> >                    size = hits.scoreDocs.length;
> >> >> >                }
> >> >> >                for (int i = 0; i < size; i++)
> >> >> >                {
> >> >> >                    Document doc =
> searcher.doc(hits.scoreDocs[i].doc);
> >> >> >                    String text = doc.get(fieldName);
> >> >> >                    System.out.println("fieldContent is: " + text);
> >> >> >                }
> >> >> >
> >> >> >                IndexSearcher oldSearcher = searcher;
> >> >> >
> >> >> >                 File newFile = new File(getIndexPath());
> >> >> >                for (File file : newFile.listFiles())
> >> >> >                {
> >> >> >                    if (!file.delete())
> >> >> >                    {
> >> >> >                        System.out.println("Can not delete file:
" +
> >> >> > file.getAbsolutePath());
> >> >> >                    }
> >> >> >                }
> >> >> >
> >> >> >                // Copy index File from another folder to this
> folder
> >> >> >                copyDir(new File(backupIndexpath), newFile);
> >> >> >
> >> >> >                Directory newDir = FSDirectory.open(newFile);
> >> >> >                IndexSearcher newSearcher = new
> IndexSearcher(newDir);
> >> >> >                 searcher = newSearcher;
> >> >> >
> >> >> >                oldSearcher.close();
> >> >> >
> >> >> >                System.out.println("Closed Searcher: " +
> >> >> > oldSearcher.getIndexReader().directory().toString());
> >> >> >
> >> >> >                System.out.println("input 'Q' to quit testing...");
> >> >> >                BufferedReader br = new BufferedReader(new
> >> >> > InputStreamReader(System.in));
> >> >> >
> >> >> >                if (br.readLine().trim().equals("Q"))
> >> >> >                {
> >> >> >                    break;
> >> >> >                }
> >> >> >            }
> >> >> >            catch (CorruptIndexException e)
> >> >> >            {
> >> >> >                e.printStackTrace();
> >> >> >            }
> >> >> >            catch (IOException e)
> >> >> >            {
> >> >> >                e.printStackTrace();
> >> >> >            }
> >> >> >            catch (ParseException e)
> >> >> >            {
> >> >> >                e.printStackTrace();
> >> >> >            }
> >> >> >        }
> >> >> >    }
> >> >> >
> >> >> >    private String getIndexPath()
> >> >> >    {
> >> >> >        if (indexPath.equals(indexPath1))
> >> >> >        {
> >> >> >            indexPath = indexPath2;
> >> >> >        }
> >> >> >        else
> >> >> >        {
> >> >> >            indexPath = indexPath1;
> >> >> >        }
> >> >> >
> >> >> >        return indexPath;
> >> >> >    }
> >> >> >
> >> >> >     public static void copyDir(File sourceLocation, File
> >> targetLocation)
> >> >> > throws IOException
> >> >> >    {
> >> >> >        String[] children = sourceLocation.list();
> >> >> >        for (int i = 0; i < children.length; i++)
> >> >> >        {
> >> >> >            InputStream in = null;
> >> >> >            OutputStream out = null;
> >> >> >            try
> >> >> >            {
> >> >> >                in = new FileInputStream(new File(sourceLocation,
> >> >> > children[i]));
> >> >> >                out = new FileOutputStream(new File(targetLocation,
> >> >> > children[i]));
> >> >> >
> >> >> >                byte[] buf = new byte[1024];
> >> >> >                int len;
> >> >> >                while ((len = in.read(buf)) > 0)
> >> >> >                {
> >> >> >                    out.write(buf, 0, len);
> >> >> >                }
> >> >> >            }
> >> >> >            catch (FileNotFoundException e)
> >> >> >            {
> >> >> >                e.printStackTrace();
> >> >> >            }
> >> >> >            catch (IOException ioe)
> >> >> >            {
> >> >> >                ioe.printStackTrace();
> >> >> >            }
> >> >> >            finally
> >> >> >            {
> >> >> >                try
> >> >> >                {
> >> >> >                    if (in != null)
> >> >> >                    {
> >> >> >                        in.close();
> >> >> >                    }
> >> >> >                    if (out != null)
> >> >> >                    {
> >> >> >                        out.close();
> >> >> >                     }
> >> >> >                }
> >> >> >                catch (IOException e)
> >> >> >                {
> >> >> >                    e.printStackTrace();
> >> >> >                }
> >> >> >            }
> >> >> >        }
> >> >> >    }
> >> >> >
> >> >> >     public static void main(String[] args)
> >> >> >    {
> >> >> >        SearchTest searchTest = new SearchTest();
> >> >> >        searchTest.search();
> >> >> >    }
> >> >> >
> >> >> > }
> >> >> >
> >> >> > 在 2011年1月12日 下午11:53,Michael McCandless <lucene@mikemccandless.com
> >> >写道:
> >> >> >
> >> >> > > Hmmm.
> >> >> > >
> >> >> > > When you say "locked" what actually does that mean?  Can
you post
> >> the
> >> >> > > exception?
> >> >> > >
> >> >> > > Also, can you whittle down your example even more?   EG if
> calling
> >> >> > > this method twice causes the problem, make a method that
calls it
> >> >> > > twice and hits the exception and then start simplifying from
> >> there...
> >> >> > >
> >> >> > > Mike
> >> >> > >
> >> >> > > 2011/1/12 张志田 <zhitian.zhang@dianping.com>:
> >> >> > > > Mike, thanks for your feedback.
> >> >> > > >
> >> >> > > > I verified this in the debug mode, so I just check the
folder I
> >> >> closed
> >> >> > in
> >> >> > > > the last loop. Actually, both two folders are locked.
> >> >> > > >
> >> >> > > > tried with new FSDirectory every loop, no help.
> >> >> > > >
> >> >> > > > Garry
> >> >> > > >
> >> >> > > > 2011/1/12 Michael McCandless <lucene@mikemccandless.com>
> >> >> > > >
> >> >> > > >> When you break out of the loop (user enters 'Q')
you don't
> close
> >> the
> >> >> > > >> current searcher.  Could that be it?
> >> >> > > >>
> >> >> > > >> Also you are calling FSDir.open each time but should
only do
> it
> >> once
> >> >> > > >> (though this should be "harmless").
> >> >> > > >>
> >> >> > > >> Mike
> >> >> > > >>
> >> >> > > >> On Wed, Jan 12, 2011 at 5:39 AM, 张志田 <
> zhitian.zhang@dianping.com
> >> >
> >> >> > > wrote:
> >> >> > > >> > Dear Luceners,
> >> >> > > >> >
> >> >> > > >> > I'm using lucene-3.0.2 in our app. There is
some testing
> code
> >> for
> >> >> > > >> switching
> >> >> > > >> > index, however, when my code run a couple of
times, I found
> the
> >> >> > index
> >> >> > > >> file
> >> >> > > >> > was locked, I can not delete the old index
files.
> >> >> > > >> >
> >> >> > > >> >
> >> >> > > >> > The code looks like:
> >> >> > > >> >
> >> >> > > >> > public class SearchTest
> >> >> > > >> > {
> >> >> > > >> >
> >> >> > > >> >    private static final int MAX_RESULT = 10000;
> >> >> > > >> >
> >> >> > > >> >    private String indexPath1 = "D:\\index\\index1";
> >> >> > > >> >    private String indexPath2 = "D:\\index\\index2";
> >> >> > > >> >
> >> >> > > >> >    private String indexPath = indexPath1;
> >> >> > > >> >
> >> >> > > >> >    private Analyzer analyzer = new
> >> >> > > StandardAnalyzer(Version.LUCENE_30);
> >> >> > > >> >
> >> >> > > >> >    private Directory dir = null;
> >> >> > > >> >
> >> >> > > >> >    private IndexSearcher searcher;
> >> >> > > >> >
> >> >> > > >> >    public void search()
> >> >> > > >> >    {
> >> >> > > >> >        while(true)
> >> >> > > >> >        {
> >> >> > > >> >            try
> >> >> > > >> >            {
> >> >> > > >> >                String keyword = "test";
> >> >> > > >> >                String fieldName = "searchfield";
> >> >> > > >> >
> >> >> > > >> >                if(dir == null)
> >> >> > > >> >                {
> >> >> > > >> >                    dir = FSDirectory.open(new
> File(indexPath));
> >> >> > > >> >                }
> >> >> > > >> >                searcher = new IndexSearcher(dir);
> >> >> > > >> >
> >> >> > > >> >                QueryParser queryParse = new
> >> >> > > >> QueryParser(Version.LUCENE_30,
> >> >> > > >> > fieldName, analyzer);
> >> >> > > >> >                Query query = queryParse.parse(keyword);
> >> >> > > >> >
> >> >> > > >> >                TopDocs hits = searcher.search(query,
> >> MAX_RESULT);
> >> >> > > >> >                int size = 5;
> >> >> > > >> >                if(hits.scoreDocs.length <
size)
> >> >> > > >> >                {
> >> >> > > >> >                    size = hits.scoreDocs.length;
> >> >> > > >> >                }
> >> >> > > >> >                for (int i = 0; i < size;
i++)
> >> >> > > >> >                {
> >> >> > > >> >                    Document doc =
> >> >> > searcher.doc(hits.scoreDocs[i].doc);
> >> >> > > >> >                    String text = doc.get(fieldName);
> >> >> > > >> >                    System.out.println("fieldContent
is: " +
> >> text);
> >> >> > > >> >                }
> >> >> > > >> >
> >> >> > > >> >                IndexSearcher oldSearcher =
searcher;
> >> >> > > >> >                dir = FSDirectory.open(new
> >> File(getIndexPath()));
> >> >> > > >> >                IndexSearcher newSearcher =
new
> >> IndexSearcher(dir);
> >> >> > > >> >                searcher = newSearcher;
> >> >> > > >> >
> >> >> > > >> >                oldSearcher.close();
> >> >> > > >> >                System.out.println("Closed Searcher:
" +
> >> >> > > >> > oldSearcher.getIndexReader().directory().toString());
> >> >> > > >> >
> >> >> > > >> >                System.out.println("input 'Q'
to quit
> >> testing...");
> >> >> > > >> >                BufferedReader br = new BufferedReader(new
> >> >> > > >> > InputStreamReader(System.in));
> >> >> > > >> >
> >> >> > > >> >                if(br.readLine().trim().equals("Q"))
> >> >> > > >> >                {
> >> >> > > >> >                    break;
> >> >> > > >> >                }
> >> >> > > >> >            }
> >> >> > > >> >            catch (CorruptIndexException e)
> >> >> > > >> >            {
> >> >> > > >> >                e.printStackTrace();
> >> >> > > >> >            }
> >> >> > > >> >            catch (IOException e)
> >> >> > > >> >            {
> >> >> > > >> >                e.printStackTrace();
> >> >> > > >> >            }
> >> >> > > >> >            catch (ParseException e)
> >> >> > > >> >            {
> >> >> > > >> >                e.printStackTrace();
> >> >> > > >> >            }
> >> >> > > >> >        }
> >> >> > > >> >    }
> >> >> > > >> >
> >> >> > > >> >    private String getIndexPath()
> >> >> > > >> >    {
> >> >> > > >> >        if(indexPath.equals(indexPath1))
> >> >> > > >> >        {
> >> >> > > >> >            indexPath = indexPath2;
> >> >> > > >> >        }
> >> >> > > >> >        else
> >> >> > > >> >        {
> >> >> > > >> >            indexPath = indexPath1;
> >> >> > > >> >        }
> >> >> > > >> >
> >> >> > > >> >        return indexPath;
> >> >> > > >> >    }
> >> >> > > >> >
> >> >> > > >> >    public static void main(String[] args)
> >> >> > > >> >    {
> >> >> > > >> >        SearchTest searchTest = new SearchTest();
> >> >> > > >> >        searchTest.search();
> >> >> > > >> >    }
> >> >> > > >> >
> >> >> > > >> > }
> >> >> > > >> >
> >> >> > > >> > Can anybody take a look at the above code snippet?
> >> >> > > >> >
> >> >> > > >> > I want to search on the different index file
every time so I
> >> >> created
> >> >> > > two
> >> >> > > >> > different folders and switch them time to time.
The index
> files
> >> in
> >> >> > the
> >> >> > > >> > index1/index2 maybe replaced before the search
request
> comes.
> >> >> > > >> >
> >> >> > > >> > The problem I found is after I ran the above
code 2 or more
> >> loops,
> >> >> I
> >> >> > > can
> >> >> > > >> not
> >> >> > > >> > modify/delete the cfs/cfx file in the file
system(Windows
> >> 2003),
> >> >> > > although
> >> >> > > >> I
> >> >> > > >> > closed the searcher every time in the code.
It seems that
> the
> >> >> index
> >> >> > > file
> >> >> > > >> is
> >> >> > > >> > not released.
> >> >> > > >> >
> >> >> > > >> > Is the problem caused by the shared reference
of searcher?
> or
> >> some
> >> >> > > shared
> >> >> > > >> > thread in the lucene?
> >> >> > > >> >
> >> >> > > >> > Thanks in advance!
> >> >> > > >> > Garry
> >> >> > > >> >
> >> >> > > >>
> >> >> > > >
> >> >> > > >
> >> >> > > >
> >> >> > > > --
> >> >> > > > 张志田
> >> >> > > >
> >> >> > > > 大众点评网 - 技术部
> >> >> > > > 电话:52521070 - 1675
> >> >> > > >
> >> >> > >
> >> >> > >
> >> ---------------------------------------------------------------------
> >> >> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> >> > > For additional commands, e-mail:
> java-user-help@lucene.apache.org
> >> >> > >
> >> >> > >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > 张志田
> >> >> >
> >> >> > 大众点评网 - 技术部
> >> >> > 电话:52521070 - 1675
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > 张志田
> >> >
> >> > 大众点评网 - 技术部
> >> > 电话:52521070 - 1675
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message