hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Girish Joshi <gjo...@groupon.com.INVALID>
Subject Re: Slow reads coinciding with higher compaction time avg time
Date Fri, 13 Nov 2015 21:42:50 GMT
Thanks everyone. I have 192G RAM on hbase machines in the cluster and out
of it around 100-120G is used in user processes and rest of it for caching.
The swap graphs I have show me that the swapping happens at a rate of 100
million kb/second.

I have disabled swappiness on my machines. but I still have the same issue
of latency spikes during swapping.

Thanks,

Girish.

On Mon, Nov 2, 2015 at 8:07 PM, Yu Li <carp84@gmail.com> wrote:

> Second Vlad. Swapping is necessary in many situations, but as JVM does not
> behave well under swapping, HBase may run into trouble if swapped. Search
> for "swapping" in hbase book <http://hbase.apache.org/book.html> and you
> could see the same suggestion.
>
> Best Regards,
> Yu
>
> On 3 November 2015 at 11:59, Vladimir Rodionov <vladrodionov@gmail.com>
> wrote:
>
> > >> Do you have any specific suggestions to avoid swapping during hbase
> > compactions.
> >
> > You can google "disable swap on linux", make sure that you do
> > not overprovision your system's RAM (too many processes running and
> > consuming all physical memory), monitor swap usage with vmstat.
> >
> > -Vlad
> >
> > On Mon, Nov 2, 2015 at 4:20 PM, Girish Joshi <gjoshi@groupon.com.invalid
> >
> > wrote:
> >
> > > Thanks. Do you have any specific suggestions to avoid swapping during
> > hbase
> > > compactions.
> > >
> > > Thanks,
> > >
> > > Girish.
> > >
> > > On Sun, Nov 1, 2015 at 6:25 PM, Vladimir Rodionov <
> > vladrodionov@gmail.com>
> > > wrote:
> > >
> > > > >>- There is a spike in compaction time avg time metric. At the
same
> > time
> > > > the
> > > > >>swap bytes in and swap bytes out also have higher value.
> > > >
> > > > Swapping is bad. You have to avoid it.
> > > >
> > > > -Vlad
> > > >
> > > > On Sun, Nov 1, 2015 at 10:24 AM, Girish Joshi
> > <gjoshi@groupon.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hello
> > > > >
> > > > > In my hbase cluster, I observe the following consistently happening
> > > over
> > > > > several days:-
> > > > >
> > > > > - There is a spike in compaction time avg time metric. At the same
> > time
> > > > the
> > > > > swap bytes in and swap bytes out also have higher value.
> > > > > - Around the same time, I see the FS PRead and FS Read latencies
> and
> > > > client
> > > > > latencies doing random reads increase.
> > > > >
> > > > > My hbase cluster consisting of 16 nodes and setup with a
> replication
> > to
> > > > > another cluster of 16 nodes has the following workload:-
> > > > >
> > > > > - There are around 4 tables which have lot of write activity(around
> > > 500k
> > > > > per second writes on m1/m15 moving average). 2 of these tables have
> > > > atomic
> > > > > counter columns keeping track of some analytics data and being
> > > > incremented
> > > > > with every write.
> > > > >
> > > > > - There are 2 tables which receive bulk uploaded data
> > > periodically(around
> > > > > once a day)
> > > > >
> > > > > - We expect reads at around 100k per second mainly from tables
> which
> > > have
> > > > > bulk upload data and the one which has counter columns. The read
> > > > > latencies(p99) spike up to around 1000-5000 ms when the above
> > > compaction
> > > > > time avg time metric increases. In other times, they are below 100
> > ms.
> > > > >
> > > > > I have set the hbase.hregion.majorcompaction to 0 on region
> servers;
> > I
> > > > plan
> > > > > to set it to 0 on master nodes too so that I can take out the
> > > possibility
> > > > > of time triggered major compactions being the problem. But I
> suspect
> > > > there
> > > > > are lot of minor compactions and those leading to major compactions
> > > > > happening at the time of spikes.
> > > > >
> > > > > *Any suggestions on how to avoid this situation of read latency
> > spikes
> > > > and
> > > > > have better read performance?*
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Girish.
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message