hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Peterson <kevin...@gmail.com>
Subject hlogs do not get cleared
Date Tue, 15 Dec 2009 18:34:51 GMT
We're running a 13 node HBase cluster. We had some problems a week ago with
it being overloaded and errors related to not being able to find a block on
HDFS, but adding four more nodes and increasing max heap from 3GB to 4.5GB
on all nodes fixed any problems.

Looking at the logs now, though, we see that HLogs are not getting removed:

2009-12-15 01:45:48,426 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260867136036,
entries=210524, calcsize=63757422, filesize=41073798. New hlog
/hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260870348421
2009-12-15 01:45:48,427 INFO org.apache.hadoop.hbase.regionserver.HLog: Too
many hlogs: logs=130, maxlogs=96; forcing flush of region with oldest edits:
articles-article-id,f15489ea-38a4-4127-9179-1b2dc5f3b5d4,1260083783909
2009-12-15 01:57:14,188 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on region
articles,\x00\x00\x01\x25\x8C\x0F\xCB\x18\xB5U\xF7\xC6\x5DoH\xB8\x98\xEBH,E\x7C\x07\x14,1260830133341
2009-12-15 01:57:17,519 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260870348421,
entries=92795, calcsize=63908073, filesize=54042783. New hlog
/hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260871037510
2009-12-15 01:57:17,519 INFO org.apache.hadoop.hbase.regionserver.HLog: Too
many hlogs: logs=131, maxlogs=96; forcing flush of region with oldest edits:
articles-article-id,f1cd1b02-3d1b-453c-b44f-94ec5a1e3a46,1260007536878

>From reading the log message, I interpret this as saying that every time it
rolls an hlog, if there are more than maxlogs logs, it will flush one
region. I'm assuming that a log could have edits for multiple regions, so
this seems to mean that if we have 100 regions and maxlogs set to 96, if it
flushes one region each time it rolls a log, it will create 100 logs before
it flushes all regions and is able to delete the log, so it will reach
steady state at 196 hlogs. Is this correct?

We're concerned because when we had problems last week, we saw lots of log
messages related to "Too many hlogs" and had assumed they were related to
the problems. Is this anything to worry about?

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message