crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Quentin Ambard <quentin.amb...@gmail.com>
Subject mr.MRPipeline error running job : java.io.IOException: No such file or directory
Date Sun, 12 May 2013 00:02:11 GMT
Hi,
I'm running a simple job on hadoop cdh 4.1.2 based on crunch.
The job is quite simple : it scan a hbase table,  extract some data from
each entry of hbase, group the result by key and combine them using an
aggreator, then write it back to another hbase table.
It works fine on my computer, however when I try to launch it on my hadoop
cluster I get the following :

>>hadoop jar uber-crunch-1.0-SNAPSHOT.jar description
/home/quentin/default.properties
13/05/12 01:57:50 INFO support.ClassPathXmlApplicationContext: Refreshing
org.springframework.context.support.ClassPathXmlApplicationContext@1f4384c2:
startup date [Sun May 12 01:57:50 CEST 2013]; root of context hierarchy
13/05/12 01:57:50 INFO xml.XmlBeanDefinitionReader: Loading XML bean
definitions from class path resource [context/job-description-context.xml]
13/05/12 01:57:50 INFO xml.XmlBeanDefinitionReader: Loading XML bean
definitions from class path resource [context/default-batch-context.xml]
13/05/12 01:57:51 INFO annotation.ClassPathBeanDefinitionScanner: JSR-330
'javax.inject.Named' annotation found and supported for component scanning
13/05/12 01:57:51 INFO config.PropertyPlaceholderConfigurer: Loading
properties file from URL
[file:/tmp/hadoop-hdfs/hadoop-unjar7637839123250781784/default.properties]
13/05/12 01:57:51 INFO config.PropertyPlaceholderConfigurer: Loading
properties file from URL
[jar:file:/home/quentin/uber-crunch-1.0-SNAPSHOT.jar!/default.properties]
13/05/12 01:57:51 INFO config.PropertyPlaceholderConfigurer: Loading
properties file from URL [file:/home/quentin/default.properties]
13/05/12 01:57:51 INFO annotation.AutowiredAnnotationBeanPostProcessor:
JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
13/05/12 01:57:51 INFO support.DefaultListableBeanFactory:
Pre-instantiating singletons in
org.springframework.beans.factory.support.DefaultListableBeanFactory@5b7b0998:
defining beans
[org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,applicationContextHolder,descriptionLauncher,descriptionExtractor,emailExtractor,rawTextExtractor,keywordsExtractor,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,org.springframework.context.annotation.ConfigurationClassPostProcessor$ImportAwareBeanPostProcessor#0];
root of factory hierarchy
13/05/12 01:57:52 INFO hbase.HBaseTarget: HBaseTarget ignores checks for
existing outputs...
13/05/12 01:57:53 INFO collect.PGroupedTableImpl: Setting num reduce tasks
to 2
13/05/12 01:57:53 ERROR mr.MRPipeline:
org.apache.crunch.CrunchRuntimeException: java.io.IOException:
java.lang.RuntimeException: java.io.IOException: No such file or directory
13/05/12 01:57:53 WARN mr.MRPipeline: Not running cleanup while output
targets remain

Any idea of the origin of the problem ? Maybe it's something with
permissions or a crunch tmp file, but I can't find out where it come from

Thanks for your help


Quentin

Mime
View raw message