hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Marc Spaggiari (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7845) optimize hfile index key
Date Thu, 21 Mar 2013 17:55:15 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609196#comment-13609196
] 

Jean-Marc Spaggiari commented on HBASE-7845:
--------------------------------------------

Here we go!
jmspaggi@hbasetest:~/hbase$ cat before.txt | grep took | grep Gaus
13/03/21 11:39:36 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 120260ms.
13/03/21 11:44:42 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 119153ms.
13/03/21 11:49:49 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 119066ms.
13/03/21 11:54:55 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 118865ms.
13/03/21 12:00:02 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 119450ms.
13/03/21 12:05:11 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 122048ms.
13/03/21 12:10:14 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 119197ms.
13/03/21 12:15:22 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 119977ms.
13/03/21 12:20:32 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 121030ms.
13/03/21 12:25:38 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 119880ms.
jmspaggi@hbasetest:~/hbase$ cat after.txt | grep took | grep Gaus
13/03/21 12:56:07 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 119765ms.
13/03/21 13:01:14 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 119730ms.
13/03/21 13:06:20 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 119945ms.
13/03/21 13:11:24 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 119283ms.
13/03/21 13:16:29 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 118535ms.
13/03/21 13:21:30 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 117409ms.
13/03/21 13:26:37 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 119458ms.
13/03/21 13:31:43 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 119178ms.
13/03/21 13:36:48 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 118427ms.
13/03/21 13:41:55 INFO hbase.HFilePerformanceEvaluation: Running GaussianRandomReadBenchmark
for 1000000 rows took 119580ms.


jmspaggi@hbasetest:~/hbase$ cat before.txt | grep took | grep Write
13/03/21 11:35:42 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1307ms.
13/03/21 11:40:51 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1333ms.
13/03/21 11:45:58 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1328ms.
13/03/21 11:51:04 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1313ms.
13/03/21 11:56:09 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1302ms.
13/03/21 12:01:15 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1322ms.
13/03/21 12:06:23 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1304ms.
13/03/21 12:11:29 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1299ms.
13/03/21 12:16:37 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1329ms.
13/03/21 12:21:45 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1354ms.
jmspaggi@hbasetest:~/hbase$ cat after.txt | grep took | grep Write
13/03/21 12:52:14 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1261ms.
13/03/21 12:57:21 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1286ms.
13/03/21 13:02:27 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1338ms.
13/03/21 13:07:33 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1309ms.
13/03/21 13:12:39 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1317ms.
13/03/21 13:17:41 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1303ms.
13/03/21 13:22:45 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1328ms.
13/03/21 13:27:51 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1302ms.
13/03/21 13:32:57 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1301ms.
13/03/21 13:38:03 INFO hbase.HFilePerformanceEvaluation: Running SequentialWriteBenchmark
for 1000000 rows took 1292ms.


jmspaggi@hbasetest:~/hbase$ cat before.txt | grep took | grep UniformRandomSmallScan
13/03/21 11:35:46 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3989ms.
13/03/21 11:40:55 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3840ms.
13/03/21 11:46:01 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3898ms.
13/03/21 11:51:07 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3857ms.
13/03/21 11:56:13 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3912ms.
13/03/21 12:01:19 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3991ms.
13/03/21 12:06:27 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3858ms.
13/03/21 12:11:33 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3902ms.
13/03/21 12:16:41 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3898ms.
13/03/21 12:21:49 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3844ms.
jmspaggi@hbasetest:~/hbase$ cat after.txt | grep took | grep UniformRandomSmallScan
13/03/21 12:52:18 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3853ms.
13/03/21 12:57:25 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3915ms.
13/03/21 13:02:31 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3909ms.
13/03/21 13:07:37 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3955ms.
13/03/21 13:12:43 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3852ms.
13/03/21 13:17:45 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3853ms.
13/03/21 13:22:49 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3890ms.
13/03/21 13:27:55 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3848ms.
13/03/21 13:33:01 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3881ms.
13/03/21 13:38:07 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomSmallScan for
1000000 rows took 3893ms.


jmspaggi@hbasetest:~/hbase$ cat before.txt | grep took | grep UniformRandomReadBenchmark
13/03/21 11:37:36 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 110016ms.
13/03/21 11:42:43 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 107885ms.
13/03/21 11:47:50 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 108216ms.
13/03/21 11:52:56 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 108158ms.
13/03/21 11:58:03 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 109513ms.
13/03/21 12:03:09 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 109805ms.
13/03/21 12:08:15 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 108181ms.
13/03/21 12:13:22 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 108953ms.
13/03/21 12:18:31 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 110007ms.
13/03/21 12:23:38 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 108967ms.
jmspaggi@hbasetest:~/hbase$ cat after.txt | grep took | grep UniformRandomReadBenchmark
13/03/21 12:54:08 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 109396ms.
13/03/21 12:59:14 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 109176ms.
13/03/21 13:04:20 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 108506ms.
13/03/21 13:09:25 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 107536ms.
13/03/21 13:14:30 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 107418ms.
13/03/21 13:19:32 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 107035ms.
13/03/21 13:24:37 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 107874ms.
13/03/21 13:29:44 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 108782ms.
13/03/21 13:34:50 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 109058ms.
13/03/21 13:39:55 INFO hbase.HFilePerformanceEvaluation: Running UniformRandomReadBenchmark
for 1000000 rows took 108650ms.


jmspaggi@hbasetest:~/hbase$ cat before.txt | grep took | grep SequentialReadBenchmark
13/03/21 11:39:37 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 414ms.
13/03/21 11:44:42 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 447ms.
13/03/21 11:49:49 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 567ms.
13/03/21 11:54:55 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 418ms.
13/03/21 12:00:03 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 421ms.
13/03/21 12:05:11 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 414ms.
13/03/21 12:10:15 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 433ms.
13/03/21 12:15:22 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 434ms.
13/03/21 12:20:32 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 418ms.
13/03/21 12:25:38 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 432ms.
jmspaggi@hbasetest:~/hbase$ cat after.txt | grep took | grep SequentialReadBenchmark
13/03/21 12:56:08 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 410ms.
13/03/21 13:01:14 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 418ms.
13/03/21 13:06:20 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 436ms.
13/03/21 13:11:25 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 403ms.
13/03/21 13:16:29 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 424ms.
13/03/21 13:21:30 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 423ms.
13/03/21 13:26:37 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 411ms.
13/03/21 13:31:44 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 422ms.
13/03/21 13:36:49 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 425ms.
13/03/21 13:41:55 INFO hbase.HFilePerformanceEvaluation: Running SequentialReadBenchmark for
1000000 rows took 428ms.


Now, number wise...

Here is the average for before your patch and after your patch. The % is the time it took
for AFTER to complete compared to BEFORE. So overall, it has always been faster. From a small
0.6% for the GaussianRandomReadBenchmark to a big 4.5%  for SequentialReadBenchmark.


GaussianRandomReadBenchmark	Before	119892.6	
GaussianRandomReadBenchmark	After	119131	99.36%
			
SequentialWriteBenchmark	Before	1319.1	
SequentialWriteBenchmark	After	1303.7	98.83%
			
UniformRandomallScan	Before	3898.9	
UniformRandomallScan	After	3884.9	99.64%
			
UniformRandomReadBenchmark	Before	108970.1	
UniformRandomReadBenchmark	After	108343.1	99.42%
			
SequentialReadBenchmark	Before	439.8	
SequentialReadBenchmark	After	420	95.50%

                
> optimize hfile index key
> ------------------------
>
>                 Key: HBASE-7845
>                 URL: https://issues.apache.org/jira/browse/HBASE-7845
>             Project: HBase
>          Issue Type: Improvement
>          Components: HFile
>    Affects Versions: 0.96.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>             Fix For: 0.98.0
>
>         Attachments: HBASE-7845.txt, HBASE-7845-v2.txt, HBASE-7845-v3.txt, HBASE-7845-v4.txt,
HBASE-7845-v5.txt, HBASE-7845-v6.txt, HBASE-7845-v7.txt, HBASE-7845-v8.txt
>
>
> Leveldb uses ByteWiseComparatorImpl::FindShortestSeparator() & FindShortSuccessor()
to reduce index key size, it would be helpful under special conditions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message