hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 茅旭峰 <m9s...@gmail.com>
Subject Re: How to control the size of WAL logs
Date Thu, 17 Mar 2011 16:24:13 GMT
Thank Cryans, I'll try them!

On Fri, Mar 18, 2011 at 12:20 AM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:

> You can limit the number of WALs and their size on the region server by
> tuning:
>
> hbase.regionserver.maxlogs the default is 32
> hbase.regionserver.hlog.blocksize the default is whatever your HDFS
> blocksize times 0.95
>
> You can limit the number of parallel threads in the master by tuning:
>
> hbase.regionserver.hlog.splitlog.writer.threads the default is 3
> hbase.regionserver.hlog.splitlog.buffersize the default is 1024*1024*!28
>
> J-D
>
> On Wed, Mar 16, 2011 at 11:57 PM, 茅旭峰 <m9suns@gmail.com> wrote:
> > Hi,
> >
> > In our tests, we've accumulated lots of WAL logs, in .logs, which leads
> to
> > quite long time pause or even
> > OOME when restarting either master or region server. We're doing sort of
> > bulk import and have not using
> > bulk import tricks, like turning off WAL feature. We think it's unknown
> how
> > our application really use the
> > hbase, so it is possible that users doing batch import unless we're
> running
> > out of space. I wonder if there
> > is any property to set to control the size of WAL, would setting smaller
> > 'hbase.regionserver.logroll.period'
> > help?
> >
> > On the other hand, since we have lots of regions, the master is easy to
> run
> > into OOME, due to the occupied
> > memory by the instance of Assignment.regions. When we were trying to
> restart
> > the master, it always died
> > with OOME. I think, from the hprof file,  it is because the instance of
> > HLogSplitter$OutputSink holds too many
> > HLogSplitter$WriterAndPaths in logWriters, which even hold the buffer of
> > wal.SequenceFileLogWriter.
> > Is there any trick to avoid such kind of scenario?
> >
> > Thanks and regards,
> >
> > Mao Xu-Feng
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message