Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6401A11E12 for ; Wed, 18 Jun 2014 11:17:00 +0000 (UTC) Received: (qmail 83108 invoked by uid 500); 18 Jun 2014 11:16:58 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 83044 invoked by uid 500); 18 Jun 2014 11:16:58 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 83029 invoked by uid 99); 18 Jun 2014 11:16:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Jun 2014 11:16:58 +0000 X-ASF-Spam-Status: No, hits=0.6 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of fancyerii@gmail.com designates 209.85.215.44 as permitted sender) Received: from [209.85.215.44] (HELO mail-la0-f44.google.com) (209.85.215.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Jun 2014 11:16:54 +0000 Received: by mail-la0-f44.google.com with SMTP id ty20so415803lab.17 for ; Wed, 18 Jun 2014 04:16:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=MJbMT15BQfd4vCZoLPPhYgSURudk5JWkwbux7b5XQxQ=; b=xNzQ1X+tAlB4sfNjKOA1tSGwBhs9ZgwfzvjCxzGC/Sw6mBEBj7QqIspRXX4V5i/79a EKsi5Z8sFtL0yUlIyExKPCyeDOPy6RpnEQX5VT9kqDi+mmtV3ITTA+e3BYaUPsqMgNgt iFsgOmDh2IT0wmoEmje5s9bQNg7XCkHmkntlbQaoT9ZDNDTLIb089HjAME2bTa/N3+Li dB/gWayTNMfdE1PZXIX+t031xhsayTEIJ9CiwwOlUKuWSxoyhyJdz4tJ6DI0nSR81gz1 pp5yegLoV6sBcy4lPP2bs/43XNFyNLaFGHH2iElEg4JaGVf9pNL9dJenefV0lDWKS9BN Sjyw== MIME-Version: 1.0 X-Received: by 10.112.13.7 with SMTP id d7mr855190lbc.98.1403090190238; Wed, 18 Jun 2014 04:16:30 -0700 (PDT) Received: by 10.112.151.200 with HTTP; Wed, 18 Jun 2014 04:16:30 -0700 (PDT) In-Reply-To: References:

Date: Wed, 18 Jun 2014 19:16:30 +0800 Message-ID: Subject: Re: speed control on the server side From: Li Li To: user@hbase.apache.org Content-Type: text/plain; charset=UTF-8 X-Virus-Checked: Checked by ClamAV on apache.org thanks, I will try it. btw, by put threads will buffer 50,000 Gets and 50,000 Puts. it check whether the data exist by get, then if not exist, insert by put. while(true){ //process data cache.add(data); if(cache.size==50_000){ List gets=...; // generate gets Result[] rs=table.get(gets); List puts=new ArrayList(); for(Result r:rs){ if(r.isEmpty()){ Put put=new Put(); puts.add(put); } } table.put(puts); } cache.clear(); } I have used 20 working thread. at the beginning, there is not operations because they are buffered in my cache. But when all the threads fill the buffers and flush. the request per second is very large(more than 30,000) and some region server die. The reason I buffer gets and puts is to make it faster. On Wed, Jun 18, 2014 at 7:04 PM, Nicolas Liochon wrote: > puts will block at a point if the servers cannot follow. > See the settings starting with hbase.client in the hbase book. > As a first try, I would recommend to set hbase.client.max.perserver. > tasks to 1 in the client. You may also want to change the buffer size ( > hbase.client.write.buffer)... > > > > > On Wed, Jun 18, 2014 at 12:58 PM, Li Li wrote: > >> and also there so many Puts maintained by background hbase threads >> that consuming too much resources >> >> On Wed, Jun 18, 2014 at 6:54 PM, Li Li wrote: >> > I mean client slow itself down. e.g. >> > my client code(one of many threads) >> > while(true){ >> > // process data and generate data need to insert to hbase >> > List puts=...; >> > table.put(puts); >> > } >> > >> > the client (I mean threads created by hbase api, not by codes) is >> > slowed down by retries. But if table.put(puts) don't block. my codes >> > will send requests. And the server is too busy and gc stop the world >> > and zookeeper don't receive heartbeat and regards it as dead >> > >> > On Wed, Jun 18, 2014 at 6:17 PM, Nicolas Liochon >> wrote: >> >> What do you mean by down? Does it crash? >> >> >> >> The server does not block on 0.96, it immediately sends back an >> exception. >> >> (See HBASE-9467) >> >> The client is implicitly slowed down by the retries, w/o blocking on the >> >> server. It's managed by the hbase client itself, and it's transparent >> for >> >> the client application. You've got some logs (info level), to tell you >> that >> >> something is happening, but that's it. >> >> >> >> You can configure the client to be less more or less verbose >> >> (hbase.client.start.log.errors.counter) or more or less pushy on the >> server >> >> (for example hbase.client.max.perserver.tasks). >> >> >> >> Cheers, >> >> >> >> Nicolas >> >> >> >> >> >> On Wed, Jun 18, 2014 at 11:59 AM, Li Li wrote: >> >> >> >>> hi all, >> >>> the hbase client send too much requests and the some region server >> >>> down. >> >>> 1. region server down because of gc pause >> >>> I can see it from log: >> >>> [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM >> >>> or host machine (eg GC): pause of approximately 3056ms >> >>> I can adjust zookeeper config >> >>> 2. as a client, how can I know server is busy? >> >>> I am using hbase 0.96 with hadoop1. I can see log such as: >> >>> org.apache.hadoop.hbase.client.AsyncProcess #178, table=vc2.url_db, >> >>> attempt=13/35 SUCCEEDED on mphbase1,60020,1403084938641, tracking >> >>> started Wed Jun 18 17:31:10 CST 2014 >> >>> I googled and found >> >>> >> >>> >> http://apache-hbase.679495.n3.nabble.com/RegionTooBusyException-Above-memstore-limit-td4056339.html >> >>> it says in hbase 0.94, if the server is busy, it will block >> >>> the client request. That's what I want, if the client send too much >> >>> request than server can handle, it will be blocked. But in 0.96, it >> >>> seems not. >> >>> I did not find any exception in client. So I can't slow the >> clien >> >>> down. >> >>> >>