hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neil Yalowitz <neilyalow...@gmail.com>
Subject multi-threaded HTablePool, incrementColumnValue, compaction and large data set
Date Mon, 16 Jan 2012 01:16:23 GMT
I'm seeing something unusual here and I wanted to see if it has occurred
for any other HBase 0.90 users.  I've read several emails here that
recommend NOT using multi-threading in an MR job, so that's certainly under
consideration.  If anyone could add to their experiences with
multi-threading in an MR job it would be very helpful.  We are testing both
implementations (with threading and without), but the threaded solution is
causing the problem.

We are processing log files with PUTs in the Map and a followup
incrementColumnValue() to a separate "counts" table in the Reducer.  The
reduce phase uses multi-threading.  The Reducer initializes an HTablePool
in the setup(), starts threads in the reduce() (to a
Java BlockingQueue/CompletionService) which do the incrementColumnValue()
and depending on the value returned create a PUT in the "counter" table,
and in the cleanup() performs a completionService.take() which is ignored
and flushes the PUTs queued by the threads.

There are no issues for approximately the first 100GB of data inserted.
 After approximately 100GB however, every subsequent job has a freeze
during the Reduce phase.  What I see happening is at some point the Reduce
(where the incrementColumnValue() takes place) tasks are "hung" and
eventually killed with reason: task client has not responded for 600
seconds.  The counters in the reduce job seem to grow briefly but then all
the tasks' counter stop increasing and the task is eventually killed.

Oddly, the problem does not occur if compaction is completely disabled (not
just major, but also setting hbase.hstore.compactionThreshold = 9999999
and hbase.hstore.blockingStoreFiles = 9999999).

Could there be a bug with HTablePool for large datasets and compaction?
 Again, this works as expected for approximately the first 100 jobs (1GB
each) but consistently fails after that.  Also to repeat, the problem does
not occur with ALL compaction disabled.

Difficult problem to describe, but I'm hoping someone may have some
feedback and/or similar experiences.  I can provide code examples if anyone
is curious.

Neil Yalowitz

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message