Return-Path: Delivered-To: apmail-hadoop-core-commits-archive@www.apache.org Received: (qmail 25750 invoked from network); 6 Feb 2009 23:01:46 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 6 Feb 2009 23:01:46 -0000 Received: (qmail 24021 invoked by uid 500); 6 Feb 2009 23:01:46 -0000 Delivered-To: apmail-hadoop-core-commits-archive@hadoop.apache.org Received: (qmail 23840 invoked by uid 500); 6 Feb 2009 23:01:46 -0000 Mailing-List: contact core-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-commits@hadoop.apache.org Received: (qmail 23831 invoked by uid 99); 6 Feb 2009 23:01:46 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Feb 2009 15:01:46 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.130] (HELO eos.apache.org) (140.211.11.130) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Feb 2009 23:01:35 +0000 Received: from eos.apache.org (localhost [127.0.0.1]) by eos.apache.org (Postfix) with ESMTP id 9306E1112F for ; Fri, 6 Feb 2009 23:01:14 +0000 (GMT) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki To: core-commits@hadoop.apache.org Date: Fri, 06 Feb 2009 23:01:14 -0000 Message-ID: <20090206230114.15833.28116@eos.apache.org> Subject: [Hadoop Wiki] Trivial Update of "Hbase/NewFileFormat/Performance" by stack X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification. The following page has been changed by stack: http://wiki.apache.org/hadoop/Hbase/NewFileFormat/Performance The comment on the change is: Adding in rfile w/ 64k buffers ------------------------------------------------------------------------------ Numbers comparing MapFile and RFile (TFile+mods has dropped from the running for the moment anyways). The code used running tests is available over in [http://github.com/ryanobjc/hbase-rfile/tree/rfile github]. I did following on local filesystem and on 4node hdfs: - {{{$ ./bin/hadoop org.apache.hadoop.hbase.MapFilePerformanceEvaluation ; ./bin/hadoop org.apache.hadoop.hbase.RFilePerformanceEvaluation}}} + {{{$ ./bin/hadoop org.apache.hadoop.hbase.MapFilePerformanceEvaluation + $ ./bin/hadoop org.apache.hadoop.hbase.RFilePerformanceEvaluation}}} + + For more context, see [wiki:Hbase/NewFileFormat New File Format]. == Local Filesystem == Macosx, 10 byte cells and keys. @@ -151, +154 @@ 09/02/06 22:10:48 INFO hbase.MapFilePerformanceEvaluation: Running SequentialReadBenchmark for 100000 rows. 09/02/06 22:10:50 INFO hbase.MapFilePerformanceEvaluation: Running SequentialReadBenchmark for 100000 rows took 2625ms.}}} + == HDFS 1k cells == + + === MapFile === + {{{09/02/06 22:28:58 INFO hbase.MapFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100000 rows. + 09/02/06 22:28:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable + 09/02/06 22:28:58 INFO compress.CodecPool: Got brand-new compressor + 09/02/06 22:28:58 INFO compress.CodecPool: Got brand-new compressor + 09/02/06 22:29:13 INFO hbase.MapFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100000 rows took 14915ms. + 09/02/06 22:29:13 INFO hbase.MapFilePerformanceEvaluation: Running UniformRandomSmallScan for 100000 rows. + 09/02/06 22:29:46 INFO hbase.MapFilePerformanceEvaluation: Running UniformRandomSmallScan for 100000 rows took 32558ms. + 09/02/06 22:29:46 INFO hbase.MapFilePerformanceEvaluation: Running UniformRandomReadBenchmark for 100000 rows. + 09/02/06 22:33:55 INFO hbase.MapFilePerformanceEvaluation: Running UniformRandomReadBenchmark for 100000 rows took 249211ms. + 09/02/06 22:33:55 INFO hbase.MapFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100000 rows. + 09/02/06 22:37:49 INFO hbase.MapFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100000 rows took 234521ms. + 09/02/06 22:37:49 INFO hbase.MapFilePerformanceEvaluation: Running SequentialReadBenchmark for 100000 rows. + 09/02/06 22:37:52 INFO hbase.MapFilePerformanceEvaluation: Running SequentialReadBenchmark for 100000 rows took 2827ms.}}} + + === RFile 64k buffers === + {{{09/02/06 22:37:53 INFO hbase.RFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100000 rows. + 09/02/06 22:37:56 INFO hbase.RFilePerformanceEvaluation: Running SequentialWriteBenchmark for 100000 rows took 3083ms. + 09/02/06 22:37:56 INFO hbase.RFilePerformanceEvaluation: Running UniformRandomSmallScan for 100000 rows. + 09/02/06 22:38:24 INFO hbase.RFilePerformanceEvaluation: Running UniformRandomSmallScan for 100000 rows took 27405ms. + 09/02/06 22:38:24 INFO hbase.RFilePerformanceEvaluation: Running UniformRandomReadBenchmark for 100000 rows. + 09/02/06 22:41:24 INFO hbase.RFilePerformanceEvaluation: Running UniformRandomReadBenchmark for 100000 rows took 180332ms. + 09/02/06 22:41:24 INFO hbase.RFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100000 rows. + 09/02/06 22:44:20 INFO hbase.RFilePerformanceEvaluation: Running GaussianRandomReadBenchmark for 100000 rows took 175614ms. + 09/02/06 22:44:20 INFO hbase.RFilePerformanceEvaluation: Running SequentialReadBenchmark for 100000 rows. + 09/02/06 22:44:23 INFO hbase.RFilePerformanceEvaluation: Running SequentialReadBenchmark for 100000 rows took 2840ms.}}} +