hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tanyasch <ta...@tickel.net>
Subject Ways to improve job cleanup speed
Date Thu, 23 Feb 2012 12:08:25 GMT

Hi, I'm running a job that completes in about a 90 seconds, but takes about
10-15 minutes to run cleanup.  I'm looking for ways to affect or even
monitor the cleanup time.  I'd like even advice about whether this is more
of a setup issue (like where I'm storing files, with Accumulo and Hadoop
temporary and log files all writing to the same disk because our cluster is
tiny) or a job issue (can I throw more reducers at it ? the brief
description of the OutputCommitter says it uses available reducers for
cleanup)  or a programming issue (in that case I'd post a different

Basically, I want to know if the first way to go at this is by reconfiguring
the cluster or if I should be programming my way out of this?  Thanks.

View this message in context: http://old.nabble.com/Ways-to-improve-job-cleanup-speed-tp33377374p33377374.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

View raw message