Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of fgaule@despegar.com
 designates 64.76.243.92 as permitted sender)
DomainKey-Signature: s=despegar; d=despegar.com; c=simple; q=dns;
  h=Received:X-Default-Received-SPF:Received:Return-Path:
   Received:X-Google-DKIM-Signature:MIME-Version:X-Received:
   Received:In-Reply-To:References:Date:Message-ID:Subject:
   From:To:Content-Type;
  b=qr2+4xUxKaZXXlpvcJMvNAeu6aC1tOp+GR0HWB43wLO64LSYvOJdUcAg
   XZ9Phw+FlWeTXn5VaPz9FCoqbsT9lXMZ4Uon660jZDTQZ957Zt2MO5TsY
   0FAQ7t4MkQj+Blk3/4azb2LreAmT4TNN8S8jscoDGZc13cXl+9/YsWZuj
   bqBX1M2KO5DfQNCn4HDVuF1yQSjEyxIS5RwnRv4jlgCZVA5HW8DD7ZFoE
   jR0q7lnHdfDBTmxB08RwB6ycmUPqJQ3OU05pst5X/o82/evKtCXTj34pJ
   IwgM88G9+n3Tsnn4P7flemehYAYgp8k9kbv8NBAjMcH2WWGZbu6RIJDXH
   A==;
MIME-Version: 1.0
In-Reply-To: 
 <CANZ=kVXY4rH1ejL-gV=bK=LkHyDeGWRYPxSndto0mi89f17NdQ@mail.gmail.com>
References: 
 <CANZ=kVVP7xOeAsSBqP7dWzUTvnBxByoa9EMHz=XQuvkL1CTxvQ@mail.gmail.com>
	<CANZ=kVUUEUtu61pS2hS1+iOS0_8176kYY0318y_gTz6oyaP6qA@mail.gmail.com>
	<CAFWiQHYFseyWHhgewvozQNNkccBTtDUPee3=NdyW=RJ0N6pzng@mail.gmail.com>
	<CANZ=kVX5frcvvd-WvxrM8YbfQQMFHo8+_aEsSC4ez0FkQ8VGPg@mail.gmail.com>
	<CAPQV63UfxWwNg8BH_1rEH15wZzrhzPR+A=PyUOyeB_DOaXR1tA@mail.gmail.com>
	<CANZ=kVX1GvbZ7deKHFUwCY9iMz-WEv3y_vnuBHwJv=7bJq4a=g@mail.gmail.com>
	<CAPQV63X1+T+rG7R2ZgTav=9ggN8F7D_2KzAPA9pfTCHggMhmjw@mail.gmail.com>
	<CANZ=kVXY4rH1ejL-gV=bK=LkHyDeGWRYPxSndto0mi89f17NdQ@mail.gmail.com>
Date: Mon, 9 Dec 2013 13:07:53 -0200
Message-ID: 
 <CANZ=kVVvWUSdwCGm5z2Tm46svBgkReUjwTsmeecDTJBw1tTdhw@mail.gmail.com>
Subject: Re: RPC - Queue Time when handlers are all waiting
From: Federico Gaule <fgaule@despegar.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=047d7b5d23d23b477c04ed1b5c6d

--047d7b5d23d23b477c04ed1b5c6d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Here is a thread saying what i think it should be (
http://grokbase.com/t/hbase/user/13bmndq53k/average-rpc-queue-time)

"The RpcQueueTime metrics are a measurement of how long individual calls
stay in this queued state. If your handlers were never 100% occupied, this
value would be 0. An average of 3 hours is concerning, it basically means
that when a call comes into the RegionServer it takes on average 3 hours to
start processing, because handlers are all occupied for that amount of
time."

Is that correct?


2013/12/9 Federico Gaule <fgaule@despegar.com>

> Correct me if i'm wrong, but, Queues should be used only when handlers ar=
e
> all busy, shouldn't it?.
> If that's true, i don't get why there is activity related to queues.
>
> Maybe i'm missing some piece of knowledge about when hbase is using queue=
s
> :)
>
> Thanks
>
>
> 2013/12/9 Jean-Marc Spaggiari <jean-marc@spaggiari.org>
>
>> There might be something I'm missing ;)
>>
>> On cluster B, as you said, never more than 50% of your handlers are used=
.
>> Your Ganglia metrics are showing that there is activities (num ops is
>> increasing), which is correct.
>>
>> Can you please confirm what you think is wrong from your charts?
>>
>> Thanks,
>>
>> JM
>>
>>
>> 2013/12/9 Federico Gaule <fgaule@despegar.com>
>>
>> > Hi JM,
>> > Cluster B is only receiving replication data (writes), but handlers ar=
e
>> > waiting most of the time (never 50% of them are used). As i have read,
>> RPC
>> > queue is only used when handlers are all waiting, does it count for
>> > replication as well?
>> >
>> > Thanks!
>> >
>> >
>> > 2013/12/9 Jean-Marc Spaggiari <jean-marc@spaggiari.org>
>> >
>> > > Hi,
>> > >
>> > > When you say that B doesn't get any read/write operation, does it me=
an
>> > you
>> > > stopped the replication? Or B is still getting the write operations
>> from
>> > A
>> > > because of the replication? If so, that's why you RPC queue is used.=
..
>> > >
>> > > JM
>> > >
>> > >
>> > > 2013/12/9 Federico Gaule <fgaule@despegar.com>
>> > >
>> > > > Not much information in RS logs (DEBUG level set to
>> > > > org.apache.hadoop.hbase). Here is a sample of one regionserver
>> showing
>> > > > increasing rpc.metrics.RpcQueueTime_num_ops and
>> > > > rpc.metrics.RpcQueueTime_avg_time
>> > > > activity:
>> > > >
>> > > > 2013-12-09 08:09:10,699 DEBUG
>> > > > org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=3D23.=
14
>> MB,
>> > > > free=3D2.73 GB, max=3D2.75 GB, blocks=3D0, accesses=3D122442151,
>> > hits=3D122168501,
>> > > > hitRatio=3D99.77%, , cachingAccesses=3D122192927, cachingHits=3D12=
2162378,
>> > > > cachingHitsRatio=3D99.97%, , evictions=3D0, evicted=3D6768,
>> > > > evictedPerRun=3DInfinity
>> > > > 2013-12-09 08:09:11,396 INFO
>> > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
>> Total
>> > > > replicated: 1
>> > > > 2013-12-09 08:09:14,979 INFO
>> > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
>> Total
>> > > > replicated: 2
>> > > > 2013-12-09 08:09:16,016 INFO
>> > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
>> Total
>> > > > replicated: 1
>> > > > ...
>> > > > 2013-12-09 08:14:07,659 INFO
>> > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
>> Total
>> > > > replicated: 1
>> > > > 2013-12-09 08:14:08,713 INFO
>> > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
>> Total
>> > > > replicated: 3
>> > > > 2013-12-09 08:14:10,699 DEBUG
>> > > > org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=3D23.=
14
>> MB,
>> > > > free=3D2.73 GB, max=3D2.75 GB, blocks=3D0, accesses=3D122442151,
>> > hits=3D122168501,
>> > > > hitRatio=3D99.77%, , cachingAccesses=3D122192927, cachingHits=3D12=
2162378,
>> > > > cachingHitsRatio=3D99.97%, , evictions=3D0, evicted=3D6768,
>> > > > evictedPerRun=3DInfinity
>> > > > 2013-12-09 08:14:12,711 INFO
>> > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
>> Total
>> > > > replicated: 1
>> > > > 2013-12-09 08:14:14,778 INFO
>> > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
>> Total
>> > > > replicated: 3
>> > > > ...
>> > > > 2013-12-09 08:15:09,199 INFO
>> > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
>> Total
>> > > > replicated: 3
>> > > > 2013-12-09 08:15:12,243 INFO
>> > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
>> Total
>> > > > replicated: 2
>> > > > 2013-12-09 08:15:22,086 INFO
>> > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink:
>> Total
>> > > > replicated: 2
>> > > >
>> > > > Thanks
>> > > >
>> > > >
>> > > > 2013/12/7 Bharath Vissapragada <bharathv@cloudera.com>
>> > > >
>> > > > > I'd look into the RS logs to see whats happening there. Difficul=
t
>> to
>> > > > guess
>> > > > > from the given information!
>> > > > >
>> > > > >
>> > > > > On Sat, Dec 7, 2013 at 8:52 PM, Federico Gaule <
>> fgaule@despegar.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Any clue?
>> > > > > > El dic 5, 2013 9:49 a.m., "Federico Gaule" <fgaule@despegar.co=
m
>> >
>> > > > > escribi=F3:
>> > > > > >
>> > > > > > > Hi,
>> > > > > > >
>> > > > > > > I have 2 clusters, Master (a) - Slave (b) replication.
>> > > > > > > B doesn't have client write or reads, all handlers (100) are
>> > > waiting
>> > > > > but
>> > > > > > > rpc.metrics.RpcQueueTime_num_ops and
>> > > > rpc.metrics.RpcQueueTime_avg_time
>> > > > > > reports
>> > > > > > > to be rpc calls to be queued.
>> > > > > > > There are some screenshots below to show ganglia metrics. Ho=
w
>> is
>> > > this
>> > > > > > > behaviour explained? I have looked for metrics specification=
s
>> but
>> > > > can't
>> > > > > > > find much information.
>> > > > > > >
>> > > > > > > Handlers
>> > > > > > > http://i42.tinypic.com/242ssoz.png
>> > > > > > >
>> > > > > > > NumOps
>> > > > > > > http://tinypic.com/r/of2c8k/5
>> > > > > > >
>> > > > > > > AvgTime
>> > > > > > > http://tinypic.com/r/2lsvg5w/5
>> > > > > > >
>> > > > > > > Cheers
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Bharath Vissapragada
>> > > > > <http://www.cloudera.com>
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > >
>> > > > [image:
>> http://www.despegar.com/galeria/images/promos/isodespegar1.png
>> > ]
>> > > >
>> > > > *Ing. Federico Gaule*
>> > > > L=EDder T=E9cnico - PAM <hotels-pam-it@despegar.com>
>> > > > Av. Corrientes 746 - Piso 9 - C.A.B.A. (C1043AAU)
>> > > > tel. +54 (11) 4894-3500
>> > > >
>> > > > *[image: Seguinos en Twitter!] <http://twitter.com/#!/despegarar>
>> > > [image:
>> > > > Seguinos en Facebook!] <http://www.facebook.com/despegar> [image:
>> > > Seguinos
>> > > > en YouTube!] <http://www.youtube.com/Despegar>*
>> > > > *Despegar.com, Inc. *
>> > > > El mejor precio para tu viaje.
>> > > >
>> > > > Este mensaje es confidencial y puede contener informaci=C3=B3n amp=
arada
>> por
>> > > el
>> > > > secreto profesional.
>> > > > Si usted ha recibido este e-mail por error, por favor
>> comun=C3=ADquenoslo
>> > > > inmediatamente respondiendo a este e-mail y luego elimin=C3=A1ndol=
o de
>> su
>> > > > sistema.
>> > > > El contenido de este mensaje no deber=C3=A1 ser copiado ni divulga=
do a
>> > > ninguna
>> > > > persona.
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > [image: http://www.despegar.com/galeria/images/promos/isodespegar1.png=
]
>> >
>> > *Ing. Federico Gaule*
>> > L=EDder T=E9cnico - PAM <hotels-pam-it@despegar.com>
>> > Av. Corrientes 746 - Piso 9 - C.A.B.A. (C1043AAU)
>> > tel. +54 (11) 4894-3500
>> >
>> > *[image: Seguinos en Twitter!] <http://twitter.com/#!/despegarar>
>> [image:
>> > Seguinos en Facebook!] <http://www.facebook.com/despegar> [image:
>> Seguinos
>> > en YouTube!] <http://www.youtube.com/Despegar>*
>> > *Despegar.com, Inc. *
>> > El mejor precio para tu viaje.
>> >
>> > Este mensaje es confidencial y puede contener informaci=C3=B3n amparad=
a por
>> el
>> > secreto profesional.
>> > Si usted ha recibido este e-mail por error, por favor comun=C3=ADqueno=
slo
>> > inmediatamente respondiendo a este e-mail y luego elimin=C3=A1ndolo de=
 su
>> > sistema.
>> > El contenido de este mensaje no deber=C3=A1 ser copiado ni divulgado a
>> ninguna
>> > persona.
>> >
>>
>
>
>
> --
>
> [image: http://www.despegar.com/galeria/images/promos/isodespegar1.png]
>
> *Ing. Federico Gaule*
> L=EDder T=E9cnico - PAM <hotels-pam-it@despegar.com>
>
> Av. Corrientes 746 - Piso 9 - C.A.B.A. (C1043AAU)
> tel. +54 (11) 4894-3500
>
>
> *[image: Seguinos en Twitter!] <http://twitter.com/#!/despegarar> [image:
> Seguinos en Facebook!] <http://www.facebook.com/despegar> [image: Seguino=
s
> en YouTube!] <http://www.youtube.com/Despegar>*
> *Despegar.com, Inc. *
>
> El mejor precio para tu viaje.
>
> Este mensaje es confidencial y puede contener informaci=C3=B3n amparada p=
or el
> secreto profesional.
> Si usted ha recibido este e-mail por error, por favor comun=C3=ADquenoslo
> inmediatamente respondiendo a este e-mail y luego elimin=C3=A1ndolo de su
> sistema.
> El contenido de este mensaje no deber=C3=A1 ser copiado ni divulgado a ni=
nguna
> persona.
>


--=20

[image: http://www.despegar.com/galeria/images/promos/isodespegar1.png]

*Ing. Federico Gaule*
L=EDder T=E9cnico - PAM <hotels-pam-it@despegar.com>
Av. Corrientes 746 - Piso 9 - C.A.B.A. (C1043AAU)
tel. +54 (11) 4894-3500

*[image: Seguinos en Twitter!] <http://twitter.com/#!/despegarar> [image:
Seguinos en Facebook!] <http://www.facebook.com/despegar> [image: Seguinos
en YouTube!] <http://www.youtube.com/Despegar>*
*Despegar.com, Inc. *
El mejor precio para tu viaje.

Este mensaje es confidencial y puede contener informaci=C3=B3n amparada por=
 el
secreto profesional.
Si usted ha recibido este e-mail por error, por favor comun=C3=ADquenoslo
inmediatamente respondiendo a este e-mail y luego elimin=C3=A1ndolo de su
sistema.
El contenido de este mensaje no deber=C3=A1 ser copiado ni divulgado a ning=
una
persona.

--047d7b5d23d23b477c04ed1b5c6d--