hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjay Dahiya <sanj...@yahoo-inc.com>
Subject Re: [jira] Commented: (HADOOP-476) Streaming should check for correctness of the task
Date Thu, 30 Nov 2006 15:34:08 GMT
On second thoughts if we plan to support file patterns for input  
paths both in streaming and general map reduce. Then either we should  
be match and see if some files matching the pattern exist in DFS or  
not check at all.  Otherwise this check will break functionality.

Sanjay


On 30-Nov-06, at 7:23 PM, Sanjay Dahiya wrote:

> It does take care of existence input in DFS,  scripts on local FS  
> and read permission for scripts. Execute permission check on a file  
> is coming in java 6, so for now I have chmod +x on the file after  
> checking other permissions.
> On descriptive error messages there is another JIRA on that  
> HADOOP-676, that patch contains some changes to improve errors and  
> throwing new exceptions.
>
> Sanjay
>
> On 30-Nov-06, at 1:46 AM, Arkady Borkovsky wrote:
>
>> Sanjay,
>>
>> does you implementation that ALL the scripts and executable  
>> available and have right permissions, and that the input exists?   
>> Also: what about descriptive error messages?
>>
>> -- ab
>>
>> On Nov 29, 2006, at 7:43 AM, Sanjay Dahiya (JIRA) wrote:
>>
>>>     [ http://issues.apache.org/jira/browse/HADOOP-476? 
>>> page=comments#action_12454343 ]
>>>
>>> Sanjay Dahiya commented on HADOOP-476:
>>> --------------------------------------
>>>
>>> I have the cli2 implementation in place, there is one issue with  
>>> cli2 which is holding it back. Command line arguments with - 
>>> jobconf and -env get merged in the same properties set in CLI's  
>>> parser and after parsing we dont know what property was passed  
>>> with which trigger.
>>> I have raised on CLI jira CLI-126. I'll submit the patch once i  
>>> resolve this one..
>>>
>>> (http://issues.apache.org/jira/browse/CLI-126) in case auto link  
>>> doesnt work for other projects.
>>>
>>>> Streaming should check for correctness of the task
>>>> --------------------------------------------------
>>>>
>>>>                 Key: HADOOP-476
>>>>                 URL: http://issues.apache.org/jira/browse/ 
>>>> HADOOP-476
>>>>             Project: Hadoop
>>>>          Issue Type: Bug
>>>>          Components: contrib/streaming
>>>>            Reporter: arkady borkovsky
>>>>         Assigned To: Sanjay Dahiya
>>>>
>>>> Currently, if anythin is wrong with streaming job, it dies  
>>>> without any explanation.
>>>> Before creating and running actual MapReduce job, Streaming  
>>>> should check if:
>>>> -- the executables (or scripts) for -mapper and -reducer are  
>>>> available and have right permissions
>>>> -- the input fragments exist
>>>
>>> -- 
>>> This message is automatically generated by JIRA.
>>> -
>>> If you think it was sent incorrectly contact one of the  
>>> administrators: http://issues.apache.org/jira/secure/ 
>>> Administrators.jspa
>>> -
>>> For more information on JIRA, see: http://www.atlassian.com/ 
>>> software/jira
>>>
>>>
>>
>


Mime
View raw message