hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Soboroff <ian.sobor...@nist.gov>
Subject Re: Task files in _temporary not getting promoted out
Date Thu, 04 Jun 2009 21:14:36 GMT

No, they were completing successfully.

In the end, I got it to work by manually making a local path (via
JobConf), and then moving the output to HDFS in close().


jason hadoop <jason.hadoop@gmail.com> writes:

> Are your tasks failing or completing successfully. Failed tasks have the
> output directory wiped, only successfully completed tasks have the files
> moved up.
> I don't recall if the FileOutputCommitter class appeared in 0.18
> On Wed, Jun 3, 2009 at 6:43 PM, Ian Soboroff <ian.soboroff@nist.gov> wrote:
>> Ok, help.  I am trying to create local task outputs in my reduce job, and
>> they get created, then go poof when the job's done.
>> My first take was to use FileOutputFormat.getWorkOutputPath, and create
>> directories in there for my outputs (which are Lucene indexes).
>>  Exasperated, I then wrote a small OutputFormat/RecordWriter pair to write
>> the indexes.  In each case, I can see directories being created in
>> attempt_foo/_temporary, but when the task is over they're gone.
>> I've stared at TextOutputFormat and I can't figure out why it's files
>> survive and mine don't.  Help!  Again, this is 0.18.3.
>> Thanks,
>> Ian

View raw message