hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Asaf Mesika <asaf.mes...@gmail.com>
Subject Re: Explosion in datasize using HBase as a MR sink
Date Fri, 31 May 2013 20:02:32 GMT
On your data set size, I would go on HFile OutputFormat and then bulk load in into HBase. Why
go through the Put flow anyway (memstore, flush, WAL), especially if you have the input ready
at your disposal for re-try if something fails?
Sounds faster to me anyway.

On May 30, 2013, at 10:52 PM, Rob Verkuylen <rob@verkuylen.net> wrote:

> 
> On May 30, 2013, at 4:51, Stack <stack@duboce.net> wrote:
> 
>> Triggering a major compaction does not alter the overall 217.5GB size?
> 
> A major compaction reduces the size from the original 219GB to the 217,5GB, so barely
a reduction. 
> 80% of the region sizes are 1,4GB before and after. I haven't merged the smaller regions,
> but that still would not bring the size down to the 2,5-5 or so GB I would expect given
T2's size.
> 
>> You have speculative execution turned on in your MR job so its possible you
>> write many versions?
> 
> I've turned off speculative execution (through conf.set) just for the mappers, since
we're not using reducers, should we? 
> I will triple check the actual job settings in the job tracker, since I need to make
the settings on a job level.
> 
>> Does your MR job fail many tasks (and though it fails, until it fails, it
>> will have written some subset of the task hence bloating your versions?).
> 
> We've had problems with failing mappers, because of zookeeper timeouts on large inserts,
> we increased zookeeper timeout and blockingstorefiles to accommodate. Now we don't
> get failures. This job writes to a cleanly made table, versions set to 1, so there shouldn't
be
> extra versions I assume(?).
> 
>> You are putting everything into protobufs?  Could that be bloating your
>> data?  Can you take a smaller subset and dump to the log a string version
>> of the pb.  Use TextFormat
>> https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/TextFormat#shortDebugString(com.google.protobuf.MessageOrBuilder)
> 
> The protobufs reduce the size to roughly 40% of the original XML data in T1. 
> The MR parser is a port of the python parse code we use going from T1 to T2.
> I've done manual comparisons on 20-30 records from T2.1 and T2 and they are identical,

> with only minute differences, because of slightly different parsing. I've done these
in hbase shell,
> I will try log dumping them too.
> 
>> It can be informative looking at hfile content.  It could give you a clue
>> as to the bloat.  See http://hbase.apache.org/book.html#hfile_tool
> 
> I will give this a go and report back. Any other debugging suggestions are more then
welcome :)
> 
> Thnx, Rob
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message