hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16583) Staged Event-Driven Architecture
Date Fri, 09 Sep 2016 20:06:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15478105#comment-15478105
] 

stack commented on HBASE-16583:
-------------------------------

The 'Availability' issue -- how our current server 'architecture' can lock up if a single
slow DN -- is reason enough to revisit core. But I am also stuck at the moment unable to drive
all CPU on a recent generation server. The bottleneck is more than just our architecture [see
interesting quote below] but I'd be interested in any revamp that ups our resource utilization.

Agree with [~Apache9] that rewriting HBase core in seastar-style would be 'difficult' and
require a discipline that might be, 'ahem', in short supply around these parts.

[~Apache9] and [~phil84111] I like the suggestions and the call-out of 'natural' boundaries.
Lets write them out and write out how we can do stuff like: "....thread swtich is not always
needed when we cross these points." Is it too ideal having a high-level goal and then going
there piecemeal? (Though I hate our thread-per-handler model. To change that, it'd be a big
change).

“The typical design of NoSQL data stores...consists of a JVM which runs on top of Linux,
utilizes the page cache, and uses complex memory allocation strategies to “trick” the
JVM garbage collector to avoid stop-the-world pauses. Such a design suffers from sudden latency
hiccups, expensive locking, and low throughput due to low processor utilization.” from Scylladb
Markitecture Page

> Staged Event-Driven Architecture
> --------------------------------
>
>                 Key: HBASE-16583
>                 URL: https://issues.apache.org/jira/browse/HBASE-16583
>             Project: HBase
>          Issue Type: Umbrella
>            Reporter: Phil Yang
>
> Staged Event-Driven Architecture (SEDA) splits request-handling logic into several stages,
each stage is executed in a thread pool and they are connected by queues.
> Currently, in region server we use a thread pool to handle requests from client. The
number of handlers is configurable, reading and writing use different pools. The current architecture
has two limitations:
> Performance:
> Different part of the handling path has different bottleneck. For example, accessing
MemStore and cache mainly consumes CPU but accessing HDFS mainly consumes network/disk IO.
If we use SEDA and split them into two different stages, we can use different numbers for
two pools according to the CPU/disk/network performance case by case.
> Availability:
> HBASE-16388 described a scene that if the client use a thread pool and use blocking methods
to access region servers, only one slow server may exhaust most of threads of the client.
For HBase, we are the client and HDFS datanodes are the servers. A slow datanode may exhaust
most of handlers. The best way to resolve this issue is make HDFS requests non-blocking, which
is exactly what SEDA does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message