crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: mr.MRPipeline error running job : java.io.IOException: No such file or directory
Date Mon, 13 May 2013 14:44:34 GMT
It does sound like a permission issue-- you can set the crunch.tmp.dir
property on the commandline (assuming you're implementing the Tool
interface) by setting -Dcrunch.tmp.dir=... to see if that helps.


On Mon, May 13, 2013 at 5:15 AM, Christian Tzolov <tzolov@apache.org> wrote:

> You can try MRPipelien.enableDebug() to lower the log level.
>
>
> On Mon, May 13, 2013 at 12:06 PM, Quentin Ambard <quentin.ambard@gmail.com
> > wrote:
>
>> The problem is that I d'ont see my job on the JobTracker page. It's like
>> the job don't even start !
>> Is there a way to improve log level to get more information on the error ?
>>
>>
>> 2013/5/12 Josh Wills <jwills@cloudera.com>
>>
>>> Something probably failed in the MapReduce job itself, which meant that
>>> there weren't any outputs for Crunch to move around. What do the error logs
>>> for the individual tasks look like on the JobTracker status page(s)?
>>>
>>>
>>> On Sat, May 11, 2013 at 5:02 PM, Quentin Ambard <
>>> quentin.ambard@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I'm running a simple job on hadoop cdh 4.1.2 based on crunch.
>>>> The job is quite simple : it scan a hbase table,  extract some data
>>>> from each entry of hbase, group the result by key and combine them using
an
>>>> aggreator, then write it back to another hbase table.
>>>> It works fine on my computer, however when I try to launch it on my
>>>> hadoop cluster I get the following :
>>>>
>>>> >>hadoop jar uber-crunch-1.0-SNAPSHOT.jar description
>>>> /home/quentin/default.properties
>>>> 13/05/12 01:57:50 INFO support.ClassPathXmlApplicationContext:
>>>> Refreshing
>>>> org.springframework.context.support.ClassPathXmlApplicationContext@1f4384c2:
>>>> startup date [Sun May 12 01:57:50 CEST 2013]; root of context hierarchy
>>>> 13/05/12 01:57:50 INFO xml.XmlBeanDefinitionReader: Loading XML bean
>>>> definitions from class path resource [context/job-description-context.xml]
>>>> 13/05/12 01:57:50 INFO xml.XmlBeanDefinitionReader: Loading XML bean
>>>> definitions from class path resource [context/default-batch-context.xml]
>>>> 13/05/12 01:57:51 INFO annotation.ClassPathBeanDefinitionScanner:
>>>> JSR-330 'javax.inject.Named' annotation found and supported for component
>>>> scanning
>>>> 13/05/12 01:57:51 INFO config.PropertyPlaceholderConfigurer: Loading
>>>> properties file from URL
>>>> [file:/tmp/hadoop-hdfs/hadoop-unjar7637839123250781784/default.properties]
>>>> 13/05/12 01:57:51 INFO config.PropertyPlaceholderConfigurer: Loading
>>>> properties file from URL
>>>> [jar:file:/home/quentin/uber-crunch-1.0-SNAPSHOT.jar!/default.properties]
>>>> 13/05/12 01:57:51 INFO config.PropertyPlaceholderConfigurer: Loading
>>>> properties file from URL [file:/home/quentin/default.properties]
>>>> 13/05/12 01:57:51 INFO annotation.AutowiredAnnotationBeanPostProcessor:
>>>> JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
>>>> 13/05/12 01:57:51 INFO support.DefaultListableBeanFactory:
>>>> Pre-instantiating singletons in
>>>> org.springframework.beans.factory.support.DefaultListableBeanFactory@5b7b0998:
>>>> defining beans
>>>> [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,applicationContextHolder,descriptionLauncher,descriptionExtractor,emailExtractor,rawTextExtractor,keywordsExtractor,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0];
>>>> root of factory hierarchy
>>>> 13/05/12 01:57:52 INFO hbase.HBaseTarget: HBaseTarget ignores checks
>>>> for existing outputs...
>>>> 13/05/12 01:57:53 INFO collect.PGroupedTableImpl: Setting num reduce
>>>> tasks to 2
>>>> 13/05/12 01:57:53 ERROR mr.MRPipeline:
>>>> org.apache.crunch.CrunchRuntimeException: java.io.IOException:
>>>> java.lang.RuntimeException: java.io.IOException: No such file or directory
>>>> 13/05/12 01:57:53 WARN mr.MRPipeline: Not running cleanup while output
>>>> targets remain
>>>>
>>>> Any idea of the origin of the problem ? Maybe it's something with
>>>> permissions or a crunch tmp file, but I can't find out where it come from
>>>>
>>>> Thanks for your help
>>>>
>>>>
>>>> Quentin
>>>>
>>>
>>>
>>>
>>> --
>>> Director of Data Science
>>> Cloudera <http://www.cloudera.com>
>>> Twitter: @josh_wills <http://twitter.com/josh_wills>
>>>
>>
>>
>>
>> --
>> Quentin Ambard
>>
>
>


-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
View raw message