hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Bieniosek (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-3) rest server: configure number of threads for jetty
Date Mon, 04 Feb 2008 23:17:09 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12565597#action_12565597
] 

Michael Bieniosek commented on HBASE-3:
---------------------------------------

Bryan,

I am experimenting with different topologies.  The distribution of our random reads is extremely
skewed, so I set up a single proxy cache (varnishd) for all our traffic.  This works much
better than past experiments with smaller separate caches on individual work nodes.  The single
proxy cache has scaled to our needs, despite living in a single process, so the traffic is
already aggregated by the time the cache misses go to the rest server.  So, I'd like to have
a single rest server serving as the backend.  If this isn't possible, I'm willing to expand
the number of rest servers, but in that case it is not the traffic aggregation that is limiting
our scaling.

> rest server: configure number of threads for jetty
> --------------------------------------------------
>
>                 Key: HBASE-3
>                 URL: https://issues.apache.org/jira/browse/HBASE-3
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: rest
>            Reporter: Michael Bieniosek
>         Attachments: hbase-4.patch
>
>
> I am running a mapreduce job (~400 simultaneous map tasks) that makes random reads from
hbase.  I have put a varnishd reverse proxy cache in front of a single rest server.  The single
rest server's jetty appears to be running out of threads:
> 2008-02-01 23:17:16,971 INFO org.mortbay.http.SocketListener: LOW ON THREADS ((256-256+1)<2)
on SocketListener0@0.0.0.0:61234
> 2008-02-01 23:17:17,116 WARN org.mortbay.http.SocketListener: OUT OF THREADS: SocketListener0@0.0.0.0:61234
> 2008-02-01 23:17:19,255 INFO org.mortbay.http.SocketListener: LOW ON THREADS ((256-256+1)<2)
on SocketListener0@0.0.0.0:61234
> The default for jetty is to use a thread pool of 256 threads.  But I'd like to be able
to specify (preferably in hadoop-site.xml) how large the thread pool should be -- in this
case it needs to have as many threads as I have simultaneous map tasks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message