Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
From: =?UTF-8?B?6IyD6IyD5qyj5qyj?= <libisthanks@gmail.com>
Date: Tue, 11 Apr 2017 13:58:20 +0800
Message-ID: <CAEgSVcDxLFqMyb=7o46yLuyYEYg0Agq8YxNrLfDxoXk+2g5N5w@mail.gmail.com>
Subject: A new Mutilple-Type Queue idea to handle multiple workloads
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a1147c5be6de047054cddc772
archived-at: Tue, 11 Apr 2017 05:58:25 -0000

--001a1147c5be6de047054cddc772
Content-Type: text/plain; charset=UTF-8

Now, the feature HBASE-11355 seperates the single Call Queue into
MutilQueue(get call queue, write call queue and scan call queue), and each
type queue can specify fixed number of handlers. It's helpful in some
outages , to avoid all read or all write requests ran out of handler
threads.

however, there are still several problems :

1. workloads in the same request type(get/write/scan) may influence each
other as before, consider the following scenario:

(1) both client-1 and client-2 send write requests, the client-1 write the
large objects(100KB record) , and client-2 write the small objects (1KB
record). the client -1 will ran out of all handler threads of the
write-queue, and decrease the client-2 throughput

(2) both client-3 and client-4 send get requests, the client-3 search all
data from lots of hfiles( all search key are equally popular), read latency
is high. the client-4 do not require any I/O resources(say, data is
cached). the client-3 will ran out of all handler threads of the get-queue,
and increase the read latency of client-4

2. administor can't increate/decrease the handler number for the specified
queue easily


we are trying to implement a new Mutilple-Typed Queue, administor can
create a queue with a specified number of handler for specified table and
specified request type(get/write/scan), as:

create_queue 'queue1' ,{'handler' => 100}

grant_queue 'table1','scan','queue1'

grant_queue 'tableN','scan','queue1'


create_queue 'queue2' ,{'handler' => 50}

grant_queue 'table2','write','queue2'


the idea based on the fact that the workload for a specified table and
request type will be unique.

in addition, administor can manager the queue with commands:

//easily increase/decrease handlers

alter_queue 'queue1' ,{'handler' => 50}

//list all queues

list_queues

//drop the specified queue

drop_queue 'queue1'


I am wondering if the developers could look at the idea and let me know if
anything is incorrect or inaccurate, or if I have missed anything.


Thanks a lot for your help!

--001a1147c5be6de047054cddc772--