accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-4039) try out a proactor design pattern for tserver services
Date Wed, 28 Oct 2015 14:31:27 GMT


Josh Elser commented on ACCUMULO-4039:

bq. I'm glad other people seem to be interested in this topic, too!

Big +1

Since my brain is still waking up for the day and I've spent some time trying to think about
this for HBase: let me give some story-time.

HBase has this very fancy (read-as: lots and lots of code but lots of control) that builds
the RPC engine from a socket and up using Hadoop RPC. Eventually, they get to a point where
they inspect the application-level RPC message, and try to push that to a given threadpool
for execution (using annotations on the endpoint the message is destined for). The available
threadpools are something like "normal", "admin", and "priority", the next being more important
than the previous. The fact that all of these pools are bounded caused lots of problems where
a user RPC would trigger an "admin" RPC which then blocked some system-driven RPC (think,
assigning the metadata tables). Thus, the user's request would ultimately block waiting for
some system state to happen, but that system state couldn't happen because the user request(s)
had saturated the pool, not letting the system actually get to the desired state.

In improving the design, I'd say it's a requirement that it's done in such a way that doesn't
lend itself to this kind of application-level RPC deadlock. Maybe there are approaches we
could take that would actually alleviate the concerns for this kind of deadlock? I'm not sure.
If not, the other choice would be to introduce some QoS/preference metric into RPCs and build
it up ourselves to avoid deadlock for the important operations.

Overall, I think I just wanted to push in my $0.02 and say: I think any attempts to improve
the RPC resource management (which understanding would be a great improvement for Accumulo)
also will require a well thought out QoS/scheduling system as well.

> try out a proactor design pattern for tserver services
> ------------------------------------------------------
>                 Key: ACCUMULO-4039
>                 URL:
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: tserver
>            Reporter: Adam Fuchs
>            Priority: Minor
> For large instances (i.e. lots of clients for a given tserver) we create oodles of threads
on the tserver. This makes for difficulty in predicting performance, memory usage, etc. Moreover,
we have operations that recurse, like a server querying itself, that we currently solve by
having separate thread pools for regular table operations and metadata table operations, and
we "disallow" things like an iterator writing to another table. One alternative option would
be to switch to a Proactor pattern:
> The core of this would be to switch to using a selection set rather than a thread per
active connection, and then wrap everything in sessions that make progress in something like
a state model, with states that account for asynchronous communications and remote work.

This message was sent by Atlassian JIRA

View raw message