hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Lawlor (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME
Date Thu, 19 Feb 2015 00:56:12 GMT

     [ https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jonathan Lawlor updated HBASE-11544:
    Attachment: HBASE-11544-v1.patch

Hey folks,

I've been working on this issue and I am attaching a patch of what I have so far. Below I
have included some discussion points that would be great to get some feedback on:

A few issues were encountered while implementing a solution for this problem. The issues,
as well as their current solutions, are outlined below (any feedback on alternative ways to
solve these problems would be appreciated):
	* In some cases, the concept of partial results doesn't seem appropriate. In these cases,
I ensured that partial results would not be created as it would only hurt performance or cause
confusion. The cases where I felt partial results should be avoided were:
		** When the client has defined a filter for their scan that requires the entire row to be
		** When the client has specified that the scan is a Small scan. Small scans are designed
to execute in a single RPC request and so the idea of having to make multiple RPC requests
to form the complete Result seems inappropriate
	* When I changed the default value of caching to Integer.MAX_VALUE I was running into OOME
on the server since caching is used to presize the ArrayList that holds results. A simple
solution to this is to simply not set an initial size on the array list. However, this solution
may still run into memory issues if the ArrayList must expand the underlying array many times
(e.g. if the table being scanned has many small rows leading to a large amount of Results
in the array list). I was wondering what everyone thought of the simple solution. If a more
sophisticated solution is required it may be best to move the caching change into its own
	* When combining the partial results into a single complete result on the client side, an
exception will be thrown from within ResultScanner#next() if it is found that the partial
results belong to different rows. This is a corner case issue that should never show up since
sequence numbers are already used in each RPC request to ensure proper ordering of request/responses
but I figured it is worth mentioning

The fine grained details of implementation can be seen in the patch, but I thought it would
be worth highlighting  how this new partial result workflow can be used to avoid OOME on the
	* The setting of Scan#setMaxResultSize will now operate at the cell level rather than the
row level. This allows a client to retrieve very large rows in fragments/partials that would
previously cause the server to OOME. By default, the entire complete result will only be formed
on the client side, whereas the server will only ever see partial Results for very large rows.
	* A new option (Scan#setAllowPartials) has been added to Scans to allow the client to see
the partial results returned by the server. This setting will be useful in cases where the
client would OOME if they were forced to reconstruct the complete result. 
	* If clients want to utilize this partial result workflow, they should use non-filtered,
non-small scans (see issues above for reasoning).

Areas for future improvement:
	* As [~lhofhansl] has pointed out, RPC is inefficient and could be improved by prefetching
results server side. This issue has been raised in HBASE-12994
	* As called out in the issues above, the initial sizing of the ArrayList on the server side
seems like it could be improved to avoid resizing of the underlying array
	* Streaming is the most ideal workflow for RPC requests but will require a large rework

Any feedback on the patch would be greatly appreciated. I am expecting the QA run to come
back with some test failures which I will address in a subsequent patch. I'm pinging [~lhofhansl]
and [~stack] as we were discussing this solution above, but if anyone else has any feedback
it would be appreciated as well!


> [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even
if it means OOME
> ------------------------------------------------------------------------------------------------------
>                 Key: HBASE-11544
>                 URL: https://issues.apache.org/jira/browse/HBASE-11544
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Jonathan Lawlor
>            Priority: Critical
>              Labels: beginner
>         Attachments: HBASE-11544-v1.patch
> Running some tests, I set hbase.client.scanner.caching=1000.  Dataset has large cells.
 I kept OOME'ing.
> Serverside, we should measure how much we've accumulated and return to the client whatever
we've gathered once we pass out a certain size threshold rather than keep accumulating till
we OOME.

This message was sent by Atlassian JIRA

View raw message