hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-7008) Set scanner caching to a better default
Date Tue, 23 Oct 2012 18:09:12 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481996#comment-13481996
] 

Lars Hofhansl edited comment on HBASE-7008 at 10/23/12 6:08 PM:
----------------------------------------------------------------

Did some quick unscientific testing (with Nagles disabled!):

rows with 1 column (100 bytes):
||Caching||ms/row||
|1|0.17|
|10|0.03|
|100|0.006|
|1000|0.003|
|2000|0.003|
|5000|0.002|
|10000|0.002|

rows with 10 columns (100 bytes each) in a single column family:
||Caching||ms/row||
|1|0.3|
|10|0.2 **|
|100|0.05|
|1000|0.03|
|2000|0.02|
|5000|0.02|
|10000|0.02|

(**) Must have crossed a packet size boundary (localhost packet size is 65k)

rows with 10 columns (1000 bytes each) in a single column family:
||Caching||ms/row||
|1|1|
|10|0.5|
|100|0.2|
|1000|0.09|
|2000|0.08|
|5000|0.07|
|10000|0.1|

rows with 100 columns (1000 bytes each) in a single column family:
||Caching||ms/row||
|1|3|
|10|1.6|
|100|0.8|
|1000|0.7|
|2000|**|

(**) Crashed the RegionServer in the default setting (OOM?)

rows with 100 columns (10000 bytes each - so 1mb rows) in a single column family:
||Caching||ms/row||
|1|18|
|10|8|
|100|8|
|1000|**|

(**) Anything over 150 crashed the RegionServer in the default setting (OOM?)

This was all done with client and server on the same machine, so worst case for a larger cache
setting. If the client is further away these effects will be much more pronounced.

So it seems a 100 would be a good default value to start with.

Edit: Remeasured some of the numbers.
                
      was (Author: lhofhansl):
    Did some quick unscientific testing (with Nagles disabled!):

rows with 1 column (100 bytes):
||Caching||ms/row||
|1|0.17|
|10|0.03|
|100|0.006|
|1000|0.003|
|2000|0.003|
|5000|0.002|
|10000|0.002|

rows with 10 columns (100 bytes each) in a single column family:
||Caching||ms/row||
|1|0.3|
|10|0.2 **|
|100|0.05|
|1000|0.03|
|2000|0.02|
|5000|0.02|
|10000|0.02|

(**) Must have crossed a packet size boundary (localhost packet size is 65k)

rows with 10 columns (1000 bytes each) in a single column family:
||Caching||ms/row||
|1|1|
|10|0.5|
|100|0.2|
|1000|0.09|
|2000|0.08|
|5000|0.07|
|10000|0.1|

rows with 100 columns (1000 bytes each) in a single column family:
||Caching||ms/row||
|1|16|
|10|1.6|
|100|0.8|
|1000|0.7|
|2000|**|

(**) Crashed the RegionServer in the default setting (OOM?)

rows with 100 columns (10000 bytes each - so 1mb rows) in a single column family:
||Caching||ms/row||
|1|28|
|10|8|
|100|8|
|1000|**|

(**) Anything over 150 crashed the RegionServer in the default setting (OOM?)

This was all done with client and server on the same machine, so worst case for a larger cache
setting. If the client is further away these effects will be much more pronounced.

So it seems a 100 would be a good default value to start with.
                  
> Set scanner caching to a better default
> ---------------------------------------
>
>                 Key: HBASE-7008
>                 URL: https://issues.apache.org/jira/browse/HBASE-7008
>             Project: HBase
>          Issue Type: Bug
>          Components: Client
>            Reporter: liang xie
>            Assignee: liang xie
>             Fix For: 0.94.3, 0.96.0
>
>         Attachments: 7008-0.94.txt, 7008-0.94-v2.txt, 7008-v3.txt, 7008-v4.txt, HBASE-7008.patch,
HBASE-7008-v2.patch
>
>
> per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253F&subj=Set+scanner+caching+to+a+better+default+
> let's set to 100 by default

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message