Return-Path: Delivered-To: apmail-lucene-java-dev-archive@www.apache.org Received: (qmail 55624 invoked from network); 1 Oct 2009 00:48:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 1 Oct 2009 00:48:20 -0000 Received: (qmail 85130 invoked by uid 500); 1 Oct 2009 00:48:19 -0000 Delivered-To: apmail-lucene-java-dev-archive@lucene.apache.org Received: (qmail 85047 invoked by uid 500); 1 Oct 2009 00:48:19 -0000 Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-dev@lucene.apache.org Delivered-To: mailing list java-dev@lucene.apache.org Received: (qmail 85039 invoked by uid 99); 1 Oct 2009 00:48:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 00:48:19 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jason.rutherglen@gmail.com designates 209.85.221.195 as permitted sender) Received: from [209.85.221.195] (HELO mail-qy0-f195.google.com) (209.85.221.195) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 00:48:09 +0000 Received: by qyk33 with SMTP id 33so6579108qyk.29 for ; Wed, 30 Sep 2009 17:47:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=JTiV/kDt5VVV1bptS4LL/B/rAgwlLGKxQX3wW+g1HQA=; b=NdHcZTDKGRNZwxO4pRn/yHEdUC1QlT17CGkFleA6f/SI7QTvCYypjyGjQKOklpmwyU EmrckVGFpDuwdELO0LjS1jkxAWS5PLESiRDQFuRXNoh8NA3X8SwPxmlZLzQJvzX7RFyY I+60Zn/4A50zLR9xRezaCuAAQMXonHbyA8DBs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=G+B5HEOEkP9hget/Fx+grOc2u0BPNNiW+W/7BptsIuQmm6JgMnhEbLi8L4hDXpT2WL 3ZqEg1R6SNwq+5DoCAVRazeAUTFguMqir9NSXsCaAOnvAJGII7UhF8JLtPXMKe0PHY9Z R02Algs2mh1gfCSmshzgDKHgUwkDBeQPJds0g= MIME-Version: 1.0 Received: by 10.229.26.149 with SMTP id e21mr431358qcc.66.1254358068743; Wed, 30 Sep 2009 17:47:48 -0700 (PDT) Date: Wed, 30 Sep 2009 17:47:48 -0700 Message-ID: <85d3c3b60909301747t70f2f0fnf39c77d0d10b6e72@mail.gmail.com> Subject: Create large preallocated file blocks when performing merges From: Jason Rutherglen To: java-dev@lucene.apache.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org I wanted to post this before I forgot. Based on an informal discussion at the Katta meeting regarding the high write throughput of Zookeeper (see http://wiki.apache.org/hadoop/ZooKeeper/Performance ) which uses the database technique of preallocating large empty files before filling them up with real data, it came up that perhaps this technique could help with the speed of Lucene segment merging? Lucene would preallocate new target merge files with zeroes of lets say one megabyte in size, then proceed to fill it in with the merge data, truncating the file to the actual size when completed. This would probably only need to be switched on when merging large segments. --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org For additional commands, e-mail: java-dev-help@lucene.apache.org