hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Li <fancye...@gmail.com>
Subject Re: speed control on the server side
Date Wed, 18 Jun 2014 11:16:30 GMT
thanks, I will try it.
btw, by put threads will buffer 50,000 Gets and 50,000 Puts. it check
whether the data exist by get, then if not exist, insert by put.
    while(true){
          //process data
          cache.add(data);
          if(cache.size==50_000){
               List<Get> gets=...; // generate gets
               Result[] rs=table.get(gets);
               List<Put> puts=new ArrayList<Put>();
               for(Result r:rs){
                     if(r.isEmpty()){
                           Put put=new Put();
                           puts.add(put);
                     }
               }
               table.put(puts);
           }
           cache.clear();
      }

I have used 20 working thread. at the beginning, there is not
operations because they are buffered in my cache. But when all the
threads fill the buffers and flush. the request per second is very
large(more than 30,000) and some region server die.
The reason I buffer  gets and puts is to make it faster.

On Wed, Jun 18, 2014 at 7:04 PM, Nicolas Liochon <nkeywal@gmail.com> wrote:
> puts will block at a point if the servers cannot follow.
> See the settings starting with hbase.client in the hbase book.
> As a first try, I would recommend to set hbase.client.max.perserver.
> tasks to 1 in the client. You may also want to change the buffer size (
> hbase.client.write.buffer)...
>
>
>
>
> On Wed, Jun 18, 2014 at 12:58 PM, Li Li <fancyerii@gmail.com> wrote:
>
>> and also there so many Puts maintained by background hbase threads
>> that consuming too much resources
>>
>> On Wed, Jun 18, 2014 at 6:54 PM, Li Li <fancyerii@gmail.com> wrote:
>> > I mean client slow itself down. e.g.
>> > my client code(one of many threads)
>> >     while(true){
>> >          // process data and generate data need to insert to hbase
>> >          List<Put> puts=...;
>> >          table.put(puts);
>> >     }
>> >
>> > the client (I mean threads created by hbase api, not by codes) is
>> > slowed down by retries. But if table.put(puts) don't block. my codes
>> > will send requests. And the server is too busy and gc stop the world
>> > and zookeeper don't receive heartbeat and regards it as dead
>> >
>> > On Wed, Jun 18, 2014 at 6:17 PM, Nicolas Liochon <nkeywal@gmail.com>
>> wrote:
>> >> What do you mean by down? Does it crash?
>> >>
>> >> The server does not block on 0.96, it immediately sends back an
>> exception.
>> >> (See HBASE-9467)
>> >> The client is implicitly slowed down by the retries, w/o blocking on the
>> >> server. It's managed by the hbase client itself, and it's transparent
>> for
>> >> the client application. You've got some logs (info level), to tell you
>> that
>> >> something is happening, but that's it.
>> >>
>> >> You can configure the client to be less more or less verbose
>> >> (hbase.client.start.log.errors.counter) or more or less pushy on the
>> server
>> >> (for example hbase.client.max.perserver.tasks).
>> >>
>> >> Cheers,
>> >>
>> >> Nicolas
>> >>
>> >>
>> >> On Wed, Jun 18, 2014 at 11:59 AM, Li Li <fancyerii@gmail.com> wrote:
>> >>
>> >>> hi all,
>> >>>     the hbase client send too much requests and the some region server
>> >>> down.
>> >>>     1. region server down because of gc pause
>> >>>         I can see it from log:
>> >>>        [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM
>> >>> or host machine (eg GC): pause of approximately 3056ms
>> >>>          I can adjust zookeeper config
>> >>>     2. as a client, how can I know server is busy?
>> >>>          I am using hbase 0.96 with hadoop1. I can see log such as:
>> >>> org.apache.hadoop.hbase.client.AsyncProcess #178, table=vc2.url_db,
>> >>> attempt=13/35 SUCCEEDED on mphbase1,60020,1403084938641, tracking
>> >>> started Wed Jun 18 17:31:10 CST 2014
>> >>>          I googled and found
>> >>>
>> >>>
>> http://apache-hbase.679495.n3.nabble.com/RegionTooBusyException-Above-memstore-limit-td4056339.html
>> >>>          it says in hbase 0.94, if the server is busy, it will block
>> >>> the client request. That's what I want, if the client send too much
>> >>> request than server can handle, it will be blocked. But in 0.96, it
>> >>> seems not.
>> >>>          I did not find any exception in client. So I can't slow the
>> clien
>> >>> down.
>> >>>
>>

Mime
View raw message