From java-user-return-52941-apmail-lucene-java-user-archive=lucene.apache.org@lucene.apache.org Mon Jun 4 14:59:28 2012 Return-Path: X-Original-To: apmail-lucene-java-user-archive@www.apache.org Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1A18D994F for ; Mon, 4 Jun 2012 14:59:28 +0000 (UTC) Received: (qmail 53997 invoked by uid 500); 4 Jun 2012 14:59:26 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 53956 invoked by uid 500); 4 Jun 2012 14:59:25 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 53948 invoked by uid 99); 4 Jun 2012 14:59:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jun 2012 14:59:25 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FSL_RCVD_USER,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jason.rutherglen@gmail.com designates 209.85.220.176 as permitted sender) Received: from [209.85.220.176] (HELO mail-vc0-f176.google.com) (209.85.220.176) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jun 2012 14:59:20 +0000 Received: by vcbfo14 with SMTP id fo14so3255852vcb.35 for ; Mon, 04 Jun 2012 07:58:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=3Gq/Ik+06c/B1GQLc4+eykXu8ju3lm6wuyuo2y3cqYM=; b=Ts8vTnGuTHJdLR029/QN5a2lcjMr2pcvyABaG8YglmmahHl7b6cxj2t159zyxik6qm 8ZvDAV5UoxZMNzTNrpt0HFj8HKkBnxQO1okQtunlnE9gXg0O6yLvS4JGbu+4gt13KKZt XieNQNfsuqU31kuu3w72nHzuE58iQu6Ha5py2lQGJJk574pgtPq/1FsvfpHO4Z2QdKT+ NXlIO1TtbXi0d/Bof01dHhUJ+T6gqnBWyCMR2RIKI9n/Zaa/slfnHy2Ddug1a9bipJax WvKx8lmlsjgZXIcFhErWD3nM5AVMRxTeKAM72pxDspXYdZ454oQdDOTnmoV/IVLXJ7t+ NmaA== MIME-Version: 1.0 Received: by 10.52.91.195 with SMTP id cg3mr10836415vdb.96.1338821939190; Mon, 04 Jun 2012 07:58:59 -0700 (PDT) Received: by 10.52.28.19 with HTTP; Mon, 4 Jun 2012 07:58:59 -0700 (PDT) In-Reply-To: References: Date: Mon, 4 Jun 2012 10:58:59 -0400 Message-ID: Subject: Re: RAMDirectory unexpectedly slows From: Jason Rutherglen To: java-user@lucene.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable If you want the index to be stored completely in RAM, there is the ByteBuffer directory [1]. Though I do not see the point in putting an index in RAM, it will be cached in RAM regardless in the OS system IO cache. 1. https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java= /org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java On Mon, Jun 4, 2012 at 10:55 AM, Cheng wrote: > My indexes are 500MB+. So it seems like that RAMDirectory is not good for > that big a size. > > My challenge, on the other side, is that I need to update the indexes ver= y > frequently. So, do you think =A0MMapDirectory is the solution? > > Thanks. > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky = wrote: > >> From the javadoc for RAMDirectory: >> >> "Warning: This class is not intended to work with huge indexes. Everythi= ng >> beyond several hundred megabytes will waste resources (GC cycles), becau= se >> it uses an internal buffer size of 1024 bytes, producing millions of >> byte[1024] arrays. This class is optimized for small memory-resident >> indexes. It also has bad concurrency on multithreaded environments. >> >> It is recommended to materialize large indexes on disk and use >> MMapDirectory, which is a high-performance directory implementation work= ing >> directly on the file system cache of the operating system, so copying da= ta >> to Java heap space is not useful." >> >> -- Jack Krupansky >> >> -----Original Message----- From: Cheng >> Sent: Monday, June 04, 2012 10:08 AM >> To: java-user@lucene.apache.org >> Subject: RAMDirectory unexpectedly slows >> >> >> Hi, >> >> My apps need to read from and write to some big indexes frequently. So I >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory >> size. >> >> I notice that the speed of reading and writing unexpectedly slows as the >> size of the indexes increases. Since the usage of RAM is less than 20%, = I >> think by default the RAMDirectory doesn't take advantage of the memory I >> assigned to JVM. >> >> What are the steps to improve the reading and writing speed of >> RAMDirectory? >> >> Thanks! >> Jeff >> >> ------------------------------**------------------------------**--------= - >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org >> For additional commands, e-mail: java-user-help@lucene.apache.**org >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org