hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 范范欣欣 <libistha...@gmail.com>
Subject A new Mutilple-Type Queue idea to handle multiple workloads
Date Tue, 11 Apr 2017 05:58:20 GMT
Now, the feature HBASE-11355 seperates the single Call Queue into
MutilQueue(get call queue, write call queue and scan call queue), and each
type queue can specify fixed number of handlers. It's helpful in some
outages , to avoid all read or all write requests ran out of handler
threads.

however, there are still several problems :

1. workloads in the same request type(get/write/scan) may influence each
other as before, consider the following scenario:

(1) both client-1 and client-2 send write requests, the client-1 write the
large objects(100KB record) , and client-2 write the small objects (1KB
record). the client -1 will ran out of all handler threads of the
write-queue, and decrease the client-2 throughput

(2) both client-3 and client-4 send get requests, the client-3 search all
data from lots of hfiles( all search key are equally popular), read latency
is high. the client-4 do not require any I/O resources(say, data is
cached). the client-3 will ran out of all handler threads of the get-queue,
and increase the read latency of client-4

2. administor can't increate/decrease the handler number for the specified
queue easily


we are trying to implement a new Mutilple-Typed Queue, administor can
create a queue with a specified number of handler for specified table and
specified request type(get/write/scan), as:

create_queue 'queue1' ,{'handler' => 100}

grant_queue 'table1','scan','queue1'

grant_queue 'tableN','scan','queue1'


create_queue 'queue2' ,{'handler' => 50}

grant_queue 'table2','write','queue2'


the idea based on the fact that the workload for a specified table and
request type will be unique.

in addition, administor can manager the queue with commands:

//easily increase/decrease handlers

alter_queue 'queue1' ,{'handler' => 50}

//list all queues

list_queues

//drop the specified queue

drop_queue 'queue1'


I am wondering if the developers could look at the idea and let me know if
anything is incorrect or inaccurate, or if I have missed anything.


Thanks a lot for your help!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message