hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Schneider" <Ch.Schnei...@gmx-topmail.de>
Subject NullPointerException in Spring Data Hadoop with CDH4
Date Tue, 12 Feb 2013 12:51:00 GMT
Hi,
I try to use Spring Data Hadoop with CDH4 to write a Map Reduce Job.

On startup, I get the following exception:

Exception in thread "SimpleAsyncTaskExecutor-1" java.lang.ExceptionInInitializerError
	at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:183)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.NullPointerException
	at org.springframework.util.ReflectionUtils.makeAccessible(ReflectionUtils.java:405)
	at org.springframework.data.hadoop.mapreduce.JobUtils.<clinit>(JobUtils.java:123)
	... 2 more

I guess there is a problem with my Hadoop related dependencies. I couldn't find any reference
showing how to configure Spring Data together with CDH4. But Costin showed, he is able to
configure it: https://build.springsource.org/browse/SPRINGDATAHADOOP-CDH4-JOB1


**Maven Setup**

<properties>
	<spring.hadoop.version>1.0.0.BUILD-SNAPSHOT</spring.hadoop.version>
	<hadoop.version>2.0.0-cdh4.1.3</hadoop.version>
</properties>

<dependencies>
	...
	<dependency>
		<groupId>org.springframework.data</groupId>
		<artifactId>spring-data-hadoop</artifactId>
		<version>${spring.hadoop.version}</version>
	</dependency>

	<dependency>
		<groupId>org.apache.hadoop</groupId>
		<artifactId>hadoop-common</artifactId>
		<version>${hadoop.version}</version>
	</dependency>

	<dependency>
		<groupId>org.apache.hadoop</groupId>
		<artifactId>hadoop-client</artifactId>
		<version>${hadoop.version}</version>
	</dependency>

	<dependency>
		<groupId>org.apache.hadoop</groupId>
		<artifactId>hadoop-streaming</artifactId>
		<version>${hadoop.version}</version>
	</dependency>

	<dependency>
		<groupId>org.apache.hadoop</groupId>
		<artifactId>hadoop-test</artifactId>
		<version>2.0.0-mr1-cdh4.1.3</version>
	</dependency>

	<dependency>
		<groupId>org.apache.hadoop</groupId>
		<artifactId>hadoop-tools</artifactId>
		<version>2.0.0-mr1-cdh4.1.3</version>
	</dependency>
	...
</dependencies>
...
<repositories>   
	<repository>
		<id>cloudera</id>
		<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
		<snapshots>
			<enabled>false</enabled>
		</snapshots>
	</repository>

	<repository>
		<id>spring-snapshot</id>
		<name>Spring Maven SNAPSHOT Repository</name>
		<url>http://repo.springframework.org/snapshot</url>
	</repository>
</repositories>

**Application Context**

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns:hdp="http://www.springframework.org/schema/hadoop" xmlns:context="http://www.springframework.org/schema/context"
	xmlns:hadoop="http://www.springframework.org/schema/hadoop"
	xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd
http://www.springframework.org/schema/context/spring-context.xsd http://www.springframework.org/schema/integration
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.1.xsd">

	<context:property-placeholder location="classpath:hadoop.properties" />

	<hdp:configuration id="hadoopConfiguration">
		fs.defaultFS=${hd.fs}
	</hdp:configuration>

	<hdp:job id="wordCountJob" input-path="${input.path}"
		output-path="${output.path}" mapper="com.example.WordMapper"
		reducer="com.example.WordReducer" />
		
<hdp:job-runner job-ref="wordCountJob" run-at-startup="true" wait-for-completion="true"/>
	

</beans>

**Cluster version**

Hadoop 2.0.0-cdh4.1.3


**Note:**

This small Unittest is running fine with the current configuration:

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "classpath:/applicationContext.xml" })
public class Starter {

	 @Autowired
	 private Configuration configuration;
		
	 @Test
	 public void shellOps() {
	 	 Assert.assertNotNull(this.configuration);
	 	 FsShell fsShell = new FsShell(this.configuration);
	 	 final Collection<FileStatus> coll = fsShell.ls("/user");
	 	 System.out.println(coll);
	 }
}


It would be nice if someone can give me an example configuration.

Best Regards,
Christian.

Mime
View raw message