hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Enis Soztutar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17408) Introduce per request limit by number of mutations
Date Tue, 03 Jan 2017 21:29:58 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796238#comment-15796238

Enis Soztutar commented on HBASE-17408:

Ted, giving more context definitely helps others to follow along. 

I have seen applications sending very big "batches" in two different settings. In the first
instance, the application was sending multi-get requests, but each RPC would end up accumulating
about 5K get requests (because application would call HTable.get(List) with default settings
with a list of that size). The server would have to buffer up all the responses causing OOM
on the server side. 

On the second instance (which is the same reason Ted logged this issue), the application was
doing multi requests with Increment. In this case, it was doing 15K increments in a single
RPC, which would take longer than RPC timeout. The following RPCs would then get ~15K exceptions
due to nonce collisions (because previous RPC indeed succeeds regardless of the timeout on
the client side). This RPC response is multi-GB in size causing OOMs. 

I have not checked the recent status for the AP, especially with HBASE-16224, but even with
that, we should have both the heap size limit and number of actions-per-batch limit. With
default settings, doing HTable.get(List) with a 10K gets in that list should not cause OOM
on the server side (same with HTable.batch()). If we already have limits for these (sorry
I did not check yet), then we should adjust down the default values at least. 

> Introduce per request limit by number of mutations
> --------------------------------------------------
>                 Key: HBASE-17408
>                 URL: https://issues.apache.org/jira/browse/HBASE-17408
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Ted Yu
> HBASE-16224 introduced hbase.client.max.perrequest.heapsize to limit the amount of data
sent from client.
> We should consider adding per request limit through the number of mutations in a batch.
> In recent troubleshooting sessions, customer had to do this in their application code
to avoid OOME on the server side.

This message was sent by Atlassian JIRA

View raw message