hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4380) large scan caching size causes RS to throw OOME
Date Tue, 13 Sep 2011 09:14:08 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103478#comment-13103478

Ted Yu commented on HBASE-4380:

There is a config parameter: hbase.client.scanner.max.result.size whose default is Long.MAX_VALUE

Take a look at line 2063 in HRegionServer:
      for (int i = 0; i < nbRows
          && currentScanResultSize < maxScannerResultSize; i++) {

> large scan caching size causes RS to throw OOME
> -----------------------------------------------
>                 Key: HBASE-4380
>                 URL: https://issues.apache.org/jira/browse/HBASE-4380
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>            Reporter: Ming Ma
>            Assignee: Ming Ma
> If the hbase application specifies a large caching size via Scan.setCaching(...),  RS
will try to accumulate enough rows before returning to the client. This could blow up RS memory.
In TableInputFormat scenario, we have couple mappers with large caching size, thus RS memory
usage goes up quickly.
> RS perhaps should take memory usage into account, for example, return less results per
HRegionInterface.next(long scannerId, int numberOfRows) call in the case of low memory.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message