Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: Ian Soboroff <ian.soboroff@nist.gov>
To: core-user@hadoop.apache.org
Subject: Re: Task files in _temporary not getting promoted out
References: <314098690906040838p2e5c9bcqae12d648b7a62940@mail.gmail.com>
Date: Thu, 04 Jun 2009 17:14:36 -0400
In-Reply-To: <314098690906040838p2e5c9bcqae12d648b7a62940@mail.gmail.com>
	(jason hadoop's message of "Thu, 4 Jun 2009 08:38:09 -0700")
Message-ID: <9cfljo7ya4j.fsf@rogue.ncsl.nist.gov>
User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/22.3 (darwin)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii


No, they were completing successfully.

In the end, I got it to work by manually making a local path (via
JobConf), and then moving the output to HDFS in close().

Ian

jason hadoop <jason.hadoop@gmail.com> writes:

> Are your tasks failing or completing successfully. Failed tasks have the
> output directory wiped, only successfully completed tasks have the files
> moved up.
>
> I don't recall if the FileOutputCommitter class appeared in 0.18
>
>
> On Wed, Jun 3, 2009 at 6:43 PM, Ian Soboroff <ian.soboroff@nist.gov> wrote:
>
>> Ok, help.  I am trying to create local task outputs in my reduce job, and
>> they get created, then go poof when the job's done.
>>
>> My first take was to use FileOutputFormat.getWorkOutputPath, and create
>> directories in there for my outputs (which are Lucene indexes).
>>  Exasperated, I then wrote a small OutputFormat/RecordWriter pair to write
>> the indexes.  In each case, I can see directories being created in
>> attempt_foo/_temporary, but when the task is over they're gone.
>>
>> I've stared at TextOutputFormat and I can't figure out why it's files
>> survive and mine don't.  Help!  Again, this is 0.18.3.
>>
>> Thanks,
>> Ian
>>
>>