hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Daniel Cryans (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-5190) Limit the IPC queue size based on calls' payload size
Date Thu, 22 Mar 2012 23:28:22 GMT

     [ https://issues.apache.org/jira/browse/HBASE-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jean-Daniel Cryans updated HBASE-5190:

    Attachment: HBASE-5190-v3.patch

New patch. I added some code (that bends over backward) to support the old ipc.server.max.queue.size
which I'm planning to only add to 0.94, trunk will just read from the new config.

bq. Should we use high_scale_lib for the callQueueSize ?

Added that.

bq. Please wrap the log with braces.

Seems superfluous, anyways I changed that too.

bq. Would giving some details about queue capacity help client make better decision ?

I fixed the comment a bit. I don't want to be too chatty in the log, we can add more in the

bq. Should the above comment be added to 0.90.7 ?

So I cleared that out. We could add a warning in 0.90 and 0.92 yeah.
> Limit the IPC queue size based on calls' payload size
> -----------------------------------------------------
>                 Key: HBASE-5190
>                 URL: https://issues.apache.org/jira/browse/HBASE-5190
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 0.90.5
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.94.0
>         Attachments: HBASE-5190-v2.patch, HBASE-5190-v3.patch, HBASE-5190.patch
> Currently we limit the number of calls in the IPC queue only on their count. It used
to be really high and was dropped down recently to num_handlers * 10 (so 100 by default) because
it was easy to OOME yourself when huge calls were being queued. It's still possible to hit
this problem if you use really big values and/or a lot of handlers, so the idea is that we
should take into account the payload size. I can see 3 solutions:
>  - Do the accounting outside of the queue itself for all calls coming in and out and
when a call doesn't fit, throw a retryable exception.
>  - Same accounting but instead block the call when it comes in until space is made available.
>  - Add a new parameter for the maximum size (in bytes) of a Call and then set the size
the IPC queue (in terms of the number of items) so that it could only contain as many items
as some predefined maximum size (in bytes) for the whole queue.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message