hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: Slow reads coinciding with higher compaction time avg time
Date Tue, 03 Nov 2015 03:21:52 GMT
How much memory do you have on this server, what is running and how much
did you give to what?

Also, what is your swapiness level?

http://askubuntu.com/questions/103915/how-do-i-configure-swappiness

2015-11-02 19:20 GMT-05:00 Girish Joshi <gjoshi@groupon.com.invalid>:

> Thanks. Do you have any specific suggestions to avoid swapping during hbase
> compactions.
>
> Thanks,
>
> Girish.
>
> On Sun, Nov 1, 2015 at 6:25 PM, Vladimir Rodionov <vladrodionov@gmail.com>
> wrote:
>
> > >>- There is a spike in compaction time avg time metric. At the same time
> > the
> > >>swap bytes in and swap bytes out also have higher value.
> >
> > Swapping is bad. You have to avoid it.
> >
> > -Vlad
> >
> > On Sun, Nov 1, 2015 at 10:24 AM, Girish Joshi <gjoshi@groupon.com.invalid
> >
> > wrote:
> >
> > > Hello
> > >
> > > In my hbase cluster, I observe the following consistently happening
> over
> > > several days:-
> > >
> > > - There is a spike in compaction time avg time metric. At the same time
> > the
> > > swap bytes in and swap bytes out also have higher value.
> > > - Around the same time, I see the FS PRead and FS Read latencies and
> > client
> > > latencies doing random reads increase.
> > >
> > > My hbase cluster consisting of 16 nodes and setup with a replication to
> > > another cluster of 16 nodes has the following workload:-
> > >
> > > - There are around 4 tables which have lot of write activity(around
> 500k
> > > per second writes on m1/m15 moving average). 2 of these tables have
> > atomic
> > > counter columns keeping track of some analytics data and being
> > incremented
> > > with every write.
> > >
> > > - There are 2 tables which receive bulk uploaded data
> periodically(around
> > > once a day)
> > >
> > > - We expect reads at around 100k per second mainly from tables which
> have
> > > bulk upload data and the one which has counter columns. The read
> > > latencies(p99) spike up to around 1000-5000 ms when the above
> compaction
> > > time avg time metric increases. In other times, they are below 100 ms.
> > >
> > > I have set the hbase.hregion.majorcompaction to 0 on region servers; I
> > plan
> > > to set it to 0 on master nodes too so that I can take out the
> possibility
> > > of time triggered major compactions being the problem. But I suspect
> > there
> > > are lot of minor compactions and those leading to major compactions
> > > happening at the time of spikes.
> > >
> > > *Any suggestions on how to avoid this situation of read latency spikes
> > and
> > > have better read performance?*
> > >
> > > Thanks,
> > >
> > > Girish.
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message