hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashish Thusoo" <athu...@facebook.com>
Subject RE: Speculative execution and output directory
Date Mon, 11 Feb 2008 16:17:54 GMT
Thanks a bunch Arun. I am ignoring these files for now as you suggest.
Thanks for pointing to the JIRA.

Ashish

-----Original Message-----
From: Arun C Murthy [mailto:acm@yahoo-inc.com] 
Sent: Saturday, February 09, 2008 10:00 PM
To: core-user@hadoop.apache.org
Subject: Re: Speculative execution and output directory


On Feb 9, 2008, at 3:52 PM, Ashish Thusoo wrote:

> Hi Hadoop users,
>
>
>
> We have intermittently hit issues with speculative execution and  
> hadoop
> streaming where we see a directory of the form
>
> _task_200...._...._m_..._.
>

It's an unfortunate side-effect of the current implementation of  
speculative execution, we have been discussing ways to fix it here:

http://issues.apache.org/jira/browse/HADOOP-2391

In the meanwhile, it is completely safe to ignore directories  
starting with with underscore ("_")	and use the rest...

Arun

>
>
> formed in the output directory. Has anyone out there hit similar  
> issues
> or knows what might be happening here? We did scan the JIRA but could
> not find any related issues reported there.
>
>
>
> Thanks,
>
> Ashish
>
>
>


Mime
View raw message