hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-12790) Support fairness across parallelized scans
Date Thu, 12 Nov 2015 07:44:11 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001806#comment-15001806

James Taylor commented on HBASE-12790:

bq. and make sure if you have internal notions of "this is for query A" and "that is for query
B" that you interleave calls to scanner.next for A and B. 
[~apurtell] We already do that, but it won't mitigate the case when a number of clients run
only a single large scan. The RS is the ultimate arbitrator.

bq. Your requirement changes every time you comment
[~stack] Not true. I've stated it clearly above. I've just given you more examples of why
the round-robin-on-connection won't solve this issue. Here's another one: Apache Drill + Phoenix.
There will be large scans parallelized across many distributed client nodes. It's useful for
the RS scheduler to know that they're linked so that it can round-robin between them. It's
similar to the MR job, but these wouldn't be batch queries, but interactive queries.

When I said it was simple, I'm talking from the users POV. It really is just having an extra
optional attribute on an operation that links them together as one "virtual operation". I
agree that the implementation is kind of a mess.

[~apurtell] - those idea you floated are all good potential workarounds that we may need to
resort to. Would be good to start with a good test case and a design IMHO. Maybe that's something
we can all agree on.

> Support fairness across parallelized scans
> ------------------------------------------
>                 Key: HBASE-12790
>                 URL: https://issues.apache.org/jira/browse/HBASE-12790
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: James Taylor
>            Assignee: ramkrishna.s.vasudevan
>              Labels: Phoenix
>         Attachments: AbstractRoundRobinQueue.java, HBASE-12790.patch, HBASE-12790_1.patch,
HBASE-12790_5.patch, HBASE-12790_callwrapper.patch, HBASE-12790_trunk_1.patch, PHOENIX_4.5.3-HBase-0.98-2317-SNAPSHOT.zip
> Some HBase clients parallelize the execution of a scan to reduce latency in getting back
results. This can lead to starvation with a loaded cluster and interleaved scans, since the
RPC queue will be ordered and processed on a FIFO basis. For example, if there are two clients,
A & B that submit largish scans at the same time. Say each scan is broken down into 100
scans by the client (broken down into equal depth chunks along the row key), and the 100 scans
of client A are queued first, followed immediately by the 100 scans of client B. In this case,
client B will be starved out of getting any results back until the scans for client A complete.
> One solution to this is to use the attached AbstractRoundRobinQueue instead of the standard
FIFO queue. The queue to be used could be (maybe it already is) configurable based on a new
config parameter. Using this queue would require the client to have the same identifier for
all of the 100 parallel scans that represent a single logical scan from the clients point
of view. With this information, the round robin queue would pick off a task from the queue
in a round robin fashion (instead of a strictly FIFO manner) to prevent starvation over interleaved
parallelized scans.

This message was sent by Atlassian JIRA

View raw message