hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME
Date Thu, 19 Feb 2015 07:06:12 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14327060#comment-14327060

Lars Hofhansl commented on HBASE-11544:

Excellent. Minor comments:

bq. When I changed the default value of caching to Integer.MAX_VALUE I was running into OOME
on the server since caching is used to presize the ArrayList that holds results. 

Meh :) Sorry I missed that when I did the simple change in HBASE-12976.
Yeah, simple solution is good. I'd actually like to change that size in all branches now,
even before we finish this. Any objections?

Quick note on small scans: They do not necessarily finish in one RPC. The difference is that
they do not hold any state on the server, which means each RPC needs to re-setup the scanner
stack, which is expensive, they also do p-reads by default. But you're absolutely right, when
need partial results you sure-as-hell do not want to use a small scan.

This: "The setting of Scan#setMaxResultSize will now operate at the cell level rather than
the row level. " does not compute with this: "whereas the server will only ever see partial
Results for very large rows."

Is the limit per cell or per row?

> [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even
if it means OOME
> ------------------------------------------------------------------------------------------------------
>                 Key: HBASE-11544
>                 URL: https://issues.apache.org/jira/browse/HBASE-11544
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Jonathan Lawlor
>            Priority: Critical
>              Labels: beginner
>         Attachments: HBASE-11544-v1.patch
> Running some tests, I set hbase.client.scanner.caching=1000.  Dataset has large cells.
 I kept OOME'ing.
> Serverside, we should measure how much we've accumulated and return to the client whatever
we've gathered once we pass out a certain size threshold rather than keep accumulating till
we OOME.

This message was sent by Atlassian JIRA

View raw message