hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "binlijin (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-16594) ROW_INDEX_V2 DBE
Date Mon, 12 Sep 2016 02:15:21 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15476741#comment-15476741
] 

binlijin edited comment on HBASE-16594 at 9/12/16 2:15 AM:
-----------------------------------------------------------

I do test with one of our very important table and the most requests on this table is random
get. This table have 5 column family(why have so many families, this is for history reason.)
I get one region's data and do the random get performance on a regionserver.

This region's detail information is:
{code}
number of row : 3463153
5 family: a,b,c,d,f

family a : avgKeyLen=54,avgValueLen=12  entries=234100060  length=4369389736(4.07GB)
family b : avgKeyLen=53,avgValueLen=10  entries=51913519   length=981625160(936MB)
family c : avgKeyLen=50,avgValueLen=6   entries=14864860   length=273820502(261MB)
family d : avgKeyLen=50,avgValueLen=6   entries=141422679  length=3216604161(3GB)
family f : avgKeyLen=38,avgValueLen=13  entries=73084074   length=1174375801(1.09GB)

avg cells per row
family a :  67.6
family b :  15
family c :  4.3
family d :  40.8
family f :  21.1

BlockSize=8k  COMPRESSION=LZO RegionSize=9.33GB  DATA_BLOCK_ENCODING => 'NONE'
BlockSize=16k COMPRESSION=LZO RegionSize=8.52GB  DATA_BLOCK_ENCODING => 'NONE'
BlockSize=32k COMPRESSION=LZO RegionSize=7.81GB  DATA_BLOCK_ENCODING => 'NONE'
BlockSize=64k COMPRESSION=LZO RegionSize=7.74GB  DATA_BLOCK_ENCODING => 'NONE'
BlockSize=32k COMPRESSION=LZO RegionSize=7.84GB  DATA_BLOCK_ENCODING => 'ROW_INDEX_V1'
BlockSize=32k COMPRESSION=LZO RegionSize=6.24GB  DATA_BLOCK_ENCODING => 'ROW_INDEX_V2'
{code}


was (Author: aoxiang):
I do test with one of our very important table. This table have 5 column family(why have so
many families, this is for history reason.)
I get one region's data and do the random get performance on a regionserver.

This region's detail information is:
{code}
number of row : 3463153
5 family: a,b,c,d,f

family a : avgKeyLen=54,avgValueLen=12  entries=234100060  length=4369389736(4.07GB)
family b : avgKeyLen=53,avgValueLen=10  entries=51913519   length=981625160(936MB)
family c : avgKeyLen=50,avgValueLen=6   entries=14864860   length=273820502(261MB)
family d : avgKeyLen=50,avgValueLen=6   entries=141422679  length=3216604161(3GB)
family f : avgKeyLen=38,avgValueLen=13  entries=73084074   length=1174375801(1.09GB)

avg cells per row
family a :  67.6
family b :  15
family c :  4.3
family d :  40.8
family f :  21.1

BlockSize=8k  COMPRESSION=LZO RegionSize=9.33GB  DATA_BLOCK_ENCODING => 'NONE'
BlockSize=16k COMPRESSION=LZO RegionSize=8.52GB  DATA_BLOCK_ENCODING => 'NONE'
BlockSize=32k COMPRESSION=LZO RegionSize=7.81GB  DATA_BLOCK_ENCODING => 'NONE'
BlockSize=64k COMPRESSION=LZO RegionSize=7.74GB  DATA_BLOCK_ENCODING => 'NONE'
BlockSize=32k COMPRESSION=LZO RegionSize=7.84GB  DATA_BLOCK_ENCODING => 'ROW_INDEX_V1'
BlockSize=32k COMPRESSION=LZO RegionSize=6.24GB  DATA_BLOCK_ENCODING => 'ROW_INDEX_V2'
{code}

> ROW_INDEX_V2 DBE
> ----------------
>
>                 Key: HBASE-16594
>                 URL: https://issues.apache.org/jira/browse/HBASE-16594
>             Project: HBase
>          Issue Type: Sub-task
>          Components: Performance
>            Reporter: binlijin
>             Fix For: 2.0.0, 1.4.0
>
>         Attachments: HBASE-16594-master_v1.patch, HBASE-16594-master_v2.patch
>
>
> See HBASE-16213, ROW_INDEX_V1 DataBlockEncoding.
> ROW_INDEX_V1 is the first version which have no storage optimization, 
> ROW_INDEX_V2 do storage optimization: store every row only once, store column family
only once in a HFileBlock.
> ROW_INDEX_V1 is : 
> /** 
>  * Store cells following every row's start offset, so we can binary search to a row's
cells. 
>  * 
>  * Format: 
>  * flat cells 
>  * integer: number of rows 
>  * integer: row0's offset 
>  * integer: row1's offset 
>  * .... 
>  * integer: dataSize 
>  * 
> */
> ROW_INDEX_V2 is :
>  * row1 qualifier timestamp type value tag
>  *          qualifier timestamp type value tag
>  *          qualifier timestamp type value tag
>  * row2 qualifier timestamp type value tag
>  * row3 qualifier timestamp type value tag
>  *          qualifier timestamp type value tag
>  * .... 
>  * integer: number of rows 
>  * integer: row0's offset 
>  * integer: row1's offset 
>  * .... 
>  * column family
>  * integer: dataSize 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message