crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Quentin Ambard <quentin.amb...@gmail.com>
Subject Re: mr.MRPipeline error running job : java.io.IOException: No such file or directory
Date Mon, 13 May 2013 10:06:56 GMT
The problem is that I d'ont see my job on the JobTracker page. It's like
the job don't even start !
Is there a way to improve log level to get more information on the error ?


2013/5/12 Josh Wills <jwills@cloudera.com>

> Something probably failed in the MapReduce job itself, which meant that
> there weren't any outputs for Crunch to move around. What do the error logs
> for the individual tasks look like on the JobTracker status page(s)?
>
>
> On Sat, May 11, 2013 at 5:02 PM, Quentin Ambard <quentin.ambard@gmail.com>wrote:
>
>> Hi,
>> I'm running a simple job on hadoop cdh 4.1.2 based on crunch.
>> The job is quite simple : it scan a hbase table,  extract some data from
>> each entry of hbase, group the result by key and combine them using an
>> aggreator, then write it back to another hbase table.
>> It works fine on my computer, however when I try to launch it on my
>> hadoop cluster I get the following :
>>
>> >>hadoop jar uber-crunch-1.0-SNAPSHOT.jar description
>> /home/quentin/default.properties
>> 13/05/12 01:57:50 INFO support.ClassPathXmlApplicationContext: Refreshing
>> org.springframework.context.support.ClassPathXmlApplicationContext@1f4384c2:
>> startup date [Sun May 12 01:57:50 CEST 2013]; root of context hierarchy
>> 13/05/12 01:57:50 INFO xml.XmlBeanDefinitionReader: Loading XML bean
>> definitions from class path resource [context/job-description-context.xml]
>> 13/05/12 01:57:50 INFO xml.XmlBeanDefinitionReader: Loading XML bean
>> definitions from class path resource [context/default-batch-context.xml]
>> 13/05/12 01:57:51 INFO annotation.ClassPathBeanDefinitionScanner: JSR-330
>> 'javax.inject.Named' annotation found and supported for component scanning
>> 13/05/12 01:57:51 INFO config.PropertyPlaceholderConfigurer: Loading
>> properties file from URL
>> [file:/tmp/hadoop-hdfs/hadoop-unjar7637839123250781784/default.properties]
>> 13/05/12 01:57:51 INFO config.PropertyPlaceholderConfigurer: Loading
>> properties file from URL
>> [jar:file:/home/quentin/uber-crunch-1.0-SNAPSHOT.jar!/default.properties]
>> 13/05/12 01:57:51 INFO config.PropertyPlaceholderConfigurer: Loading
>> properties file from URL [file:/home/quentin/default.properties]
>> 13/05/12 01:57:51 INFO annotation.AutowiredAnnotationBeanPostProcessor:
>> JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
>> 13/05/12 01:57:51 INFO support.DefaultListableBeanFactory:
>> Pre-instantiating singletons in
>> org.springframework.beans.factory.support.DefaultListableBeanFactory@5b7b0998:
>> defining beans
>> [org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,applicationContextHolder,descriptionLauncher,descriptionExtractor,emailExtractor,rawTextExtractor,keywordsExtractor,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0];
>> root of factory hierarchy
>> 13/05/12 01:57:52 INFO hbase.HBaseTarget: HBaseTarget ignores checks for
>> existing outputs...
>> 13/05/12 01:57:53 INFO collect.PGroupedTableImpl: Setting num reduce
>> tasks to 2
>> 13/05/12 01:57:53 ERROR mr.MRPipeline:
>> org.apache.crunch.CrunchRuntimeException: java.io.IOException:
>> java.lang.RuntimeException: java.io.IOException: No such file or directory
>> 13/05/12 01:57:53 WARN mr.MRPipeline: Not running cleanup while output
>> targets remain
>>
>> Any idea of the origin of the problem ? Maybe it's something with
>> permissions or a crunch tmp file, but I can't find out where it come from
>>
>> Thanks for your help
>>
>>
>> Quentin
>>
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>



-- 
Quentin Ambard

Mime
View raw message