lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dizh <d...@neusoft.com>
Subject Large Index Query Help!
Date Tue, 29 Jan 2013 05:54:28 GMT
Hi All:

I hava a large Index repo, size is 47G, yes 47G , when I look up for something in it , and
then it is dead, I trace its execution as follows:

        at org.apache.lucene.search.TopFieldCollector.add(TopFieldCollector.java:1178)
        at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.collect(TopFieldCollector.java:87)
        at org.apache.lucene.search.Scorer.score(Scorer.java:62)
        at org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.score(ConstantScoreQuery.java:238)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:588)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:555)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:507)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:484)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)

I think it is because my Index is too large , and all jvm heaps are used, and I want to ask
for some answers:

How I can reduce the memory which lucene used, Linux Top command result is as follow:

top - 13:45:11 up  2:56,  3 users,  load average: 0.59, 0.33, 0.51
Tasks: 193 total,   1 running, 192 sleeping,   0 stopped,   0 zombie
Cpu(s): 13.2%us,  0.2%sy,  0.0%ni, 86.5%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3933684k total,  3793900k used,   139784k free,    21704k buffers
Swap: 20482864k total,   490644k used, 19992220k free,  1408172k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                     
                                          
 5318 root      21   0 47.7g 1.6g 267m S 99.9 41.4   0:50.84 java                        
                                          
 5536 root      15   0  596m  67m 5856 S  3.0  1.8   0:26.22 python    

it is clearly Lucene load all index into memory, and I also know after this query the memory
is returned to OS, but just in this query, it is hung.

Could anyone give some suggestions about how to query large Index on a single machine  not
distributed?

 
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)

is intended only for the use of the intended recipient and may be confidential and/or privileged
of 
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication
is 
not the intended recipient, unauthorized use, forwarding, printing,  storing, disclosure or
copying 
is strictly prohibited, and may be unlawful.If you have received this communication in error,please

immediately notify the sender by return e-mail, and delete the original message and all copies
from 
your system. Thank you. 
---------------------------------------------------------------------------------------------------
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message