hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jason hadoop <jason.had...@gmail.com>
Subject Re: Task files in _temporary not getting promoted out
Date Thu, 04 Jun 2009 15:38:09 GMT
Are your tasks failing or completing successfully. Failed tasks have the
output directory wiped, only successfully completed tasks have the files
moved up.

I don't recall if the FileOutputCommitter class appeared in 0.18

On Wed, Jun 3, 2009 at 6:43 PM, Ian Soboroff <ian.soboroff@nist.gov> wrote:

> Ok, help.  I am trying to create local task outputs in my reduce job, and
> they get created, then go poof when the job's done.
> My first take was to use FileOutputFormat.getWorkOutputPath, and create
> directories in there for my outputs (which are Lucene indexes).
>  Exasperated, I then wrote a small OutputFormat/RecordWriter pair to write
> the indexes.  In each case, I can see directories being created in
> attempt_foo/_temporary, but when the task is over they're gone.
> I've stared at TextOutputFormat and I can't figure out why it's files
> survive and mine don't.  Help!  Again, this is 0.18.3.
> Thanks,
> Ian

Alpha Chapters of my book on Hadoop are available
www.prohadoopbook.com a community for Hadoop Professionals

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message