hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME
Date Sun, 27 Jul 2014 22:55:38 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075782#comment-14075782
] 

Lars Hofhansl commented on HBASE-11544:
---------------------------------------

The user just wants to call scanner.next(), which returns a row. That's it. Everything else
is leaking performance concerns that HBase should handle automatically into the API.

In the past we mixed the performance concern (too many RPCs for small rows) in with the API
(set number of rows). That was obviously a quick fix (mistake). The optimal bytes/RPC ratio
is a function of the network bandwidth and latency.

The only extra API needed on the client is setBatch() so that a client application has a way
to deal with rows too large for *it* to handle.

bq. what if response data size is smaller than a chunk?
It just sends what it has.

I am really not proposing anything different from networking protocols have been doing for
30 years :)


> [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even
if it means OOME
> ------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-11544
>                 URL: https://issues.apache.org/jira/browse/HBASE-11544
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>              Labels: noob
>
> Running some tests, I set hbase.client.scanner.caching=1000.  Dataset has large cells.
 I kept OOME'ing.
> Serverside, we should measure how much we've accumulated and return to the client whatever
we've gathered once we pass out a certain size threshold rather than keep accumulating till
we OOME.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message