lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zqzuk <ziqizh...@hotmail.co.uk>
Subject 170G index, 1.5 billion documents, out of memory on query
Date Mon, 25 Feb 2013 11:06:50 GMT
Hi

I am really frustrated by this problem.

I have built an index of 1.5 billion data records, with a size of about
170GB. It's been optimised and has 12 separate files in the index directory,
looking like below:

_2.fdt --- 58G
_2.fdx --- 80M
_2.fnm--- 900bytes
_2.si   --- 380bytes
_2.lucene41_0.doc --- 46G
_2_Lucene41_0.pos --- 22G
_2_Lucene41_0.tim --- 37G
_2_Lucene41_0.tip ---  766MB
_2_nrm.cfe  --- 139byte
_2_nrm.cfs  --- 5.7G
segments.gen  --  20byte
segments_1 ---    68byte


It sits on a single server with a memory of 32G allocated to it, using
default solr setting that are provided with the solr example in the
distrubtion.

I started the server ok with 32G memory, but any query other than "q=*:*"
fails, with the following out of memory exception:


java.lang.OutOfMemoryError: OutOfMemoryError likely caused by the Sun VM Bug
described in https://issues.apache.org/jira/browse/LUCENE-1566; try calling
FSDirectory.setReadChunkSize with a value smaller than the current chunk
size (2147483647)</str><str name="trace">java.lang.RuntimeException:
java.lang.OutOfMemoryError: OutOfMemoryError likely caused by the Sun VM Bug
described in https://issues.apache.org/jira/browse/LUCENE-1566; try calling
FSDirectory.setReadChunkSize with a value smaller than the current chunk
size (2147483647)
	at
org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:462)
	at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:290)
	at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
	at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
	at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
	at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
	at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
	at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
	at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
	at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
	at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:365)
	at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
	at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
	at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
	at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
	at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
	at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: OutOfMemoryError likely caused by the
Sun VM Bug described in https://issues.apache.org/jira/browse/LUCENE-1566;
try calling FSDirectory.setReadChunkSize with a value smaller than the
current chunk size (2147483647)
	at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:184)
	at
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:158)
	at
org.apache.lucene.codecs.DocValuesArraySource$ByteValues.<init>(DocValuesArraySource.java:113)
	at
org.apache.lucene.codecs.DocValuesArraySource$ByteValues.newFromInput(DocValuesArraySource.java:135)
	at
org.apache.lucene.codecs.lucene40.values.Ints$IntsReader.loadSource(Ints.java:154)
	at
org.apache.lucene.index.DocValues$SourceCache$DirectSourceCache.load(DocValues.java:791)
	at org.apache.lucene.index.DocValues.getSource(DocValues.java:129)
	at
org.apache.lucene.search.similarities.TFIDFSimilarity$ExactTFIDFDocScorer.<init>(TFIDFSimilarity.java:782)
	at
org.apache.lucene.search.similarities.TFIDFSimilarity.exactSimScorer(TFIDFSimilarity.java:761)
	at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:88)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:603)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
	at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1435)
	at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1310)
	at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:401)
	at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:410)
	at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
	at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
	at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448)
	at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
	... 25 more
Caused by: java.lang.OutOfMemoryError
	at sun.misc.Unsafe.allocateMemory(Native Method)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:101)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
	at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155)
	at sun.nio.ch.IOUtil.read(IOUtil.java:169)
	at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:612)
	at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:176)
	... 45 more




I am quite new to Solr and probably did something obviously wrong with such
a huge index... so my question is what are the possible solutions to fix
this?
-by increasing more memory?
-change solr configuration to optimise for large index? if so what should I
be looking into?
-solrcloud? Again im complete novice... any tips on configuration... # of
shards etc?

Many thanks!




--
View this message in context: http://lucene.472066.n3.nabble.com/170G-index-1-5-billion-documents-out-of-memory-on-query-tp4042696.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message