hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 茅旭峰 <m9s...@gmail.com>
Subject How to control the size of WAL logs
Date Thu, 17 Mar 2011 06:57:48 GMT
Hi,

In our tests, we've accumulated lots of WAL logs, in .logs, which leads to
quite long time pause or even
OOME when restarting either master or region server. We're doing sort of
bulk import and have not using
bulk import tricks, like turning off WAL feature. We think it's unknown how
our application really use the
hbase, so it is possible that users doing batch import unless we're running
out of space. I wonder if there
is any property to set to control the size of WAL, would setting smaller
'hbase.regionserver.logroll.period'
help?

On the other hand, since we have lots of regions, the master is easy to run
into OOME, due to the occupied
memory by the instance of Assignment.regions. When we were trying to restart
the master, it always died
with OOME. I think, from the hprof file,  it is because the instance of
HLogSplitter$OutputSink holds too many
HLogSplitter$WriterAndPaths in logWriters, which even hold the buffer of
wal.SequenceFileLogWriter.
Is there any trick to avoid such kind of scenario?

Thanks and regards,

Mao Xu-Feng

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message