hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Corgan <mcor...@hotpads.com>
Subject Re: Verbose logging with compression
Date Tue, 11 Jan 2011 23:24:11 GMT
Turns out this is what happens if you don't have the native libraries set up
correctly.  The data still gets compressed using the pure java codec, but it
doesn't cache the codec and gives you a warning each time it creates it for
each block.


On Mon, Jan 10, 2011 at 2:41 PM, Stack <stack@duboce.net> wrote:

> Thats a little silly.  That the message is INFO level is probably
> small potatoes when doing a mapreduce job but in our case with lots of
> file openings, it turns into a little log storm.
>
> I suppose you'll need to disable it.  Set log level to WARN on
> org.apache.hadoop.io.compress?
>
> This might help you making the change:
> http://wiki.apache.org/hadoop/Hbase/FAQ#A5
>
> St.Ack
>
> On Mon, Jan 10, 2011 at 9:46 AM, Matt Corgan <mcorgan@hotpads.com> wrote:
> > I'm trying to use GZIP compression but running into a logging problem.
>  It
> > appears that every time a block is compressed it logs the following:
> >
> > 2011-01-10 12:40:48,407 INFO org.apache.hadoop.io.compress.CodecPool: Got
> > brand-new compressor
> > 2011-01-10 12:40:48,414 INFO org.apache.hadoop.io.compress.CodecPool: Got
> > brand-new compressor
> > 2011-01-10 12:40:48,420 INFO org.apache.hadoop.io.compress.CodecPool: Got
> > brand-new compressor
> > 2011-01-10 12:40:48,426 INFO org.apache.hadoop.io.compress.CodecPool: Got
> > brand-new compressor
> > 2011-01-10 12:40:48,431 INFO org.apache.hadoop.io.compress.CodecPool: Got
> > brand-new compressor
> > 2011-01-10 12:40:48,447 INFO org.apache.hadoop.io.compress.CodecPool: Got
> > brand-new compressor
> > 2011-01-10 12:40:48,453 INFO org.apache.hadoop.io.compress.CodecPool: Got
> > brand-new compressor
> >
> > Same for decompression.  It's logging that 150 times per second during a
> > major compaction which pretty much renders the logs useless.  I assume
> other
> > people are not having this problem, so did we accidentally enable that
> > logging somehow?
> >
> > Thanks,
> > Matt
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message