lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Connolly (JIRA)" <j...@apache.org>
Subject [jira] Updated: (LUCENE-842) ParallelMultiSearcher memory leak
Date Thu, 22 Mar 2007 20:25:32 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Thomas Connolly updated LUCENE-842:
-----------------------------------

    Description: 
When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher
(reading a single index), continuous runs result in a memory leak. 
Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently
used.

And example of the code used is as follows. Note the close routine was added for the individual
searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.

    private void doSearch(Search search)
    {
        IndexSearcher[] indexSearchers = null;
        
        MultiSearcher multiSearcher = null;
        try
        {
            indexSearchers = getIndexSearcher();
            
            // aggregate the searches across multiple indexes
            multiSearcher = new ParallelMultiSearcher(indexSearchers); // causes LEAK BAD
            //multiSearcher = new MultiSearcher(indexSearchers); // NO leak GOOD

            final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
            final Query query = parser.parse(search.getQuery());
            
            final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()),
getSort(search.getSort()));

			// process hits...
        }
        finally
        {
            close(indexSearchers);
            close(multiSearcher);
        }
    }

    /**
     * Close the index searchers.
     * 
     * @param indexSearchers Index Searchers.
     */
    private static void close(IndexSearcher[] indexSearchers)
    {
        if (indexSearchers != null)
        {
            for (IndexSearcher indexSearcher : indexSearchers)
            {
                try
                {
                    indexSearcher.close();
                }
                catch (IOException ioex)
                {
                    LOGGER.warn("Unable to close the index searcher!", ioex);
                }
            }
        }
    }
    
    /**
     * Close the multi-searcher.
     * 
     * @param aMultiSearcher Index Searchers.
     */
    private static void close(MultiSearcher aMultiSearcher)
    {
        try
        {
            aMultiSearcher.close();
        }
        catch (IOException ioex)
        {
            LOGGER.warn("Unable to close the multi searcher!", ioex);
        }
    }


  was:
When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher
(reading a single index), continuous runs result in a memory leak. 
Substituting the MultiSearcher does not result in a memory leak. and is the workaround currently
used.

And example of the code used is as follows. Note the close routine was added for the individual
searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.

    private void doSearch(Search search)
    {
        IndexSearcher[] indexSearchers = null;
        
        MultiSearcher multiSearcher = null;
        try
        {
            indexSearchers = getIndexSearcher();
            
            // aggregate the searches across multiple indexes
            //multiSearcher = new ParallelMultiSearcher(indexSearchers);
            multiSearcher = new MultiSearcher(indexSearchers);

            final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
            final Query query = parser.parse(search.getQuery());
            
            final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()),
getSort(search.getSort()));

			// process hits...
        }
        finally
        {
            close(indexSearchers);
            close(multiSearcher);
        }
    }

    /**
     * Close the index searchers.
     * 
     * @param indexSearchers Index Searchers.
     */
    private static void close(IndexSearcher[] indexSearchers)
    {
        if (indexSearchers != null)
        {
            for (IndexSearcher indexSearcher : indexSearchers)
            {
                try
                {
                    indexSearcher.close();
                }
                catch (IOException ioex)
                {
                    LOGGER.warn("Unable to close the index searcher!", ioex);
                }
            }
        }
    }
    
    /**
     * Close the multi-searcher.
     * 
     * @param aMultiSearcher Index Searchers.
     */
    private static void close(MultiSearcher aMultiSearcher)
    {
        try
        {
            aMultiSearcher.close();
        }
        catch (IOException ioex)
        {
            LOGGER.warn("Unable to close the multi searcher!", ioex);
        }
    }



Updated the code example to leak.

Doron what do you suggest? Do you want me to attached the search index, libraries, and a main
routine? It is pretty easy to execute this code, I'm simply showing the simple logic. 
The first close, on the searchers, should not be necessary as the MultiSearcher should close
all searchers, but I found it is.


> ParallelMultiSearcher memory leak
> ---------------------------------
>
>                 Key: LUCENE-842
>                 URL: https://issues.apache.org/jira/browse/LUCENE-842
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>    Affects Versions: 2.1
>         Environment: Windows XP SP2 and Red Hat EL 4
>            Reporter: Thomas Connolly
>            Priority: Critical
>
> When using the org.apache.lucene.search.ParallelMultiSearcher to search on a single searcher
(reading a single index), continuous runs result in a memory leak. 
> Substituting the MultiSearcher does not result in a memory leak. and is the workaround
currently used.
> And example of the code used is as follows. Note the close routine was added for the
individual searchers and the MultiSearcher otherwise the was a leak in MultiSearcher.
>     private void doSearch(Search search)
>     {
>         IndexSearcher[] indexSearchers = null;
>         
>         MultiSearcher multiSearcher = null;
>         try
>         {
>             indexSearchers = getIndexSearcher();
>             
>             // aggregate the searches across multiple indexes
>             multiSearcher = new ParallelMultiSearcher(indexSearchers); // causes LEAK
BAD
>             //multiSearcher = new MultiSearcher(indexSearchers); // NO leak GOOD
>             final QueryParser parser = new QueryParser("content", new ExtendedStandardAnalyser());
>             final Query query = parser.parse(search.getQuery());
>             
>             final Hits hits = multiSearcher.search(query, getFilter(search.getFilters()),
getSort(search.getSort()));
> 			// process hits...
>         }
>         finally
>         {
>             close(indexSearchers);
>             close(multiSearcher);
>         }
>     }
>     /**
>      * Close the index searchers.
>      * 
>      * @param indexSearchers Index Searchers.
>      */
>     private static void close(IndexSearcher[] indexSearchers)
>     {
>         if (indexSearchers != null)
>         {
>             for (IndexSearcher indexSearcher : indexSearchers)
>             {
>                 try
>                 {
>                     indexSearcher.close();
>                 }
>                 catch (IOException ioex)
>                 {
>                     LOGGER.warn("Unable to close the index searcher!", ioex);
>                 }
>             }
>         }
>     }
>     
>     /**
>      * Close the multi-searcher.
>      * 
>      * @param aMultiSearcher Index Searchers.
>      */
>     private static void close(MultiSearcher aMultiSearcher)
>     {
>         try
>         {
>             aMultiSearcher.close();
>         }
>         catch (IOException ioex)
>         {
>             LOGGER.warn("Unable to close the multi searcher!", ioex);
>         }
>     }

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message