Return-Path: 
 <user-return-5024-apmail-hadoop-mapreduce-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 823B3E390
	for <apmail-hadoop-mapreduce-user-archive@minotaur.apache.org>;
 Tue, 12 Feb 2013 16:01:15 +0000 (UTC)
Received: (qmail 77663 invoked by uid 500); 12 Feb 2013 16:01:10 -0000
Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org
Received: (qmail 77408 invoked by uid 500); 12 Feb 2013 16:01:09 -0000
Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:user@hadoop.apache.org>
List-Id: <user.hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Delivered-To: mailing list user@hadoop.apache.org
Received: (qmail 77400 invoked by uid 99); 12 Feb 2013 16:01:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Feb 2013 16:01:09 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ch.schneider@gmx-topmail.de
 designates 212.227.15.19 as permitted sender)
Received: from [212.227.15.19] (HELO mout.gmx.net) (212.227.15.19)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Feb 2013 16:01:03 +0000
Received: from mailout-de.gmx.net ([10.1.76.33]) by mrigmx.server.lan
 (mrigmx001) with ESMTP (Nemesis) id 0LcmGP-1UoDD33ag3-00kBlv for
 <user@hadoop.apache.org>; Tue, 12 Feb 2013 17:00:42 +0100
Received: (qmail 21773 invoked by uid 0); 12 Feb 2013 16:00:42 -0000
Received: from 91.89.54.189 by www037.gmx.net with HTTP;
 Tue, 12 Feb 2013 17:00:42 +0100 (CET)
Content-Type: text/plain; charset="utf-8"
Date: Tue, 12 Feb 2013 17:00:42 +0100
From: "Christian Schneider" <Ch.Schneider@gmx-topmail.de>
In-Reply-To: 
 <CAO=enCK+ZxhAyHc3JZq3rvzT1conFBJvu_fS1renpPgz_JpaeQ@mail.gmail.com>
Message-ID: <20130212160042.27240@gmx.net>
MIME-Version: 1.0
References: <20130212125100.68600@gmx.net>	<511A43E8.2080702@gmail.com>
 <CAO=enCK+ZxhAyHc3JZq3rvzT1conFBJvu_fS1renpPgz_JpaeQ@mail.gmail.com>
Subject: Re: NullPointerException in Spring Data Hadoop with CDH4
To: user@hadoop.apache.org
X-Authenticated: #7857841
X-Flags: 0001
X-Mailer: WWW-Mail 6100 (Global Message Exchange)
X-Priority: 3
X-Provags-ID: V01U2FsdGVkX1/zTYkCkfDzwXCE/ac8QRP1PPn5GXFEmwVejCUfVv
 8Kzy+vPsC2FCmTqtkm30vIIEIakdVYbiz6wQ==
Content-Transfer-Encoding: 8bit
X-GMX-UID: fQJoccBreSEqaNCFHHQh8R1+IGRvb0Au
X-Virus-Checked: Checked by ClamAV on apache.org

With the help of Costin I got a running Maven configuration.

Thank you :).

This is a pom.xml for Spring Data Hadoop and CDH4:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.example</groupId>
	<artifactId>com.example.main</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>jar</packaging>

	<properties>
		<java-version>1.7</java-version>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<spring.version>3.2.0.RELEASE</spring.version>
		<spring.hadoop.version>1.0.0.BUILD-SNAPSHOT</spring.hadoop.version>
		<hadoop.version.generic>2.0.0-cdh4.1.3</hadoop.version.generic>
		<hadoop.version.mr1>2.0.0-mr1-cdh4.1.3</hadoop.version.mr1>
	</properties>

	<dependencies>

		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-core</artifactId>
			<version>${spring.version}</version>
			<exclusions>
				<exclusion>
					<groupId>commons-logging</groupId>
					<artifactId>commons-logging</artifactId>
				</exclusion>
			</exclusions>
		</dependency>

		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-context</artifactId>
			<version>${spring.version}</version>
		</dependency>


		<dependency>
			<groupId>org.springframework.data</groupId>
			<artifactId>spring-data-hadoop</artifactId>
			<version>${spring.hadoop.version}</version>

			<exclusions>
				<!-- Excluded the Hadoop dependencies to be sure that they are not mixed with them provided by cloudera. -->
				<exclusion>
					<artifactId>hadoop-streaming</artifactId>
					<groupId>org.apache.hadoop</groupId>
				</exclusion>
				<exclusion>
					<artifactId>hadoop-tools</artifactId>
					<groupId>org.apache.hadoop</groupId>
				</exclusion>
			</exclusions>

		</dependency>

		<!-- Hadoop Cloudera Dependencies -->
		<dependency>
			<groupId>org.apache.hadoop</groupId>
			<artifactId>hadoop-common</artifactId>
			<version>${hadoop.version.generic}</version>
		</dependency>
		
		<dependency>
			<groupId>org.apache.hadoop</groupId>
			<artifactId>hadoop-hdfs</artifactId>
			<version>${hadoop.version.generic}</version>
		</dependency>

		<dependency>
			<groupId>org.apache.hadoop</groupId>
			<artifactId>hadoop-tools</artifactId>
			<version>2.0.0-mr1-cdh4.1.3</version>
		</dependency>

		<dependency>
			<groupId>org.apache.hadoop</groupId>
			<artifactId>hadoop-streaming</artifactId>
			<version>2.0.0-mr1-cdh4.1.3</version>
		</dependency>

	</dependencies>

	<build>
		<plugins>

			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<configuration>
					<source>${java-version}</source>
					<target>${java-version}</target>
				</configuration>
			</plugin>

		</plugins>
	</build>

	<repositories>
		<repository>
			<id>spring-milestones</id>
			<url>http://repo.springsource.org/libs-milestone</url>
			<snapshots>
				<enabled>false</enabled>
			</snapshots>
		</repository>

		<repository>
			<id>cloudera</id>
			<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
			<snapshots>
				<enabled>false</enabled>
			</snapshots>
		</repository>

		<repository>
			<id>spring-snapshot</id>
			<name>Spring Maven SNAPSHOT Repository</name>
			<url>http://repo.springframework.org/snapshot</url>
		</repository>
	</repositories>
</project>


Best Regards,
Christian.

-------- Original-Nachricht --------
> Datum: Tue, 12 Feb 2013 16:56:50 +0100
> Von: Christian Schneider <christian.frank.schneider@gmail.com>
> An: ironhawk@gmx.com
> Betreff: Fwd: NullPointerException in Spring Data Hadoop with CDH4

> ---------- Forwarded message ----------
> From: Costin Leau <costin.leau@gmail.com>
> Date: 2013/2/12
> Subject: Re: NullPointerException in Spring Data Hadoop with CDH4
> To: user@hadoop.apache.org
> 
> 
> Hi,
> 
> For Spring Data Hadoop problems, it's best to use the designated forum
> [1].
> These being said I've tried to reproduce your error but I can't - I've
> upgraded the build to CDH 4.1.3 which runs fine against the VM on the CI
> (4.1.1).
> Maybe you have some other libraries on the client classpath?
> 
> From the stacktrace, it looks like the org.apache.hadoop.mapreduce.**Job
> class has no 'state' or 'info' fields...
> 
> Anyway, let's continue the discussion on the forum.
> 
> Cheers,
> [1]
> http://forum.springsource.org/**forumdisplay.php?87-Hadoop<http://forum.springsource.org/forumdisplay.php?87-Hadoop>
> 
> 
> On 02/12/13 2:51 PM, Christian Schneider wrote:
> 
> > Hi,
> > I try to use Spring Data Hadoop with CDH4 to write a Map Reduce Job.
> >
> > On startup, I get the following exception:
> >
> > Exception in thread "SimpleAsyncTaskExecutor-1" java.lang.**
> > ExceptionInInitializerError
> >         at org.springframework.data.**hadoop.mapreduce.JobExecutor$**
> > 2.run(JobExecutor.java:183)
> >         at java.lang.Thread.run(Thread.**java:722)
> > Caused by: java.lang.NullPointerException
> >         at
> org.springframework.util.**ReflectionUtils.**makeAccessible(**
> > ReflectionUtils.java:405)
> >         at org.springframework.data.**hadoop.mapreduce.JobUtils.<**
> > clinit>(JobUtils.java:123)
> >         ... 2 more
> >
> > I guess there is a problem with my Hadoop related dependencies. I
> couldn't
> > find any reference showing how to configure Spring Data together with
> CDH4.
> > But Costin showed, he is able to configure it:
> https://build.springsource.
> >
> **org/browse/SPRINGDATAHADOOP-**CDH4-JOB1<https://build.springsource.org/browse/SPRINGDATAHADOOP-CDH4-JOB1>
> >
> >
> > **Maven Setup**
> >
> > <properties>
> >         <spring.hadoop.version>1.0.0.**BUILD-SNAPSHOT</spring.hadoop.**
> > version>
> >         <hadoop.version>2.0.0-cdh4.1.**3</hadoop.version>
> > </properties>
> >
> > <dependencies>
> >         ...
> >         <dependency>
> >                 <groupId>org.springframework.**data</groupId>
> >                 <artifactId>spring-data-**hadoop</artifactId>
> >                 <version>${spring.hadoop.**version}</version>
> >         </dependency>
> >
> >         <dependency>
> >                 <groupId>org.apache.hadoop</**groupId>
> >                 <artifactId>hadoop-common</**artifactId>
> >                 <version>${hadoop.version}</**version>
> >         </dependency>
> >
> >         <dependency>
> >                 <groupId>org.apache.hadoop</**groupId>
> >                 <artifactId>hadoop-client</**artifactId>
> >                 <version>${hadoop.version}</**version>
> >         </dependency>
> >
> >         <dependency>
> >                 <groupId>org.apache.hadoop</**groupId>
> >                 <artifactId>hadoop-streaming</**artifactId>
> >                 <version>${hadoop.version}</**version>
> >         </dependency>
> >
> >         <dependency>
> >                 <groupId>org.apache.hadoop</**groupId>
> >                 <artifactId>hadoop-test</**artifactId>
> >                 <version>2.0.0-mr1-cdh4.1.3</**version>
> >         </dependency>
> >
> >         <dependency>
> >                 <groupId>org.apache.hadoop</**groupId>
> >                 <artifactId>hadoop-tools</**artifactId>
> >                 <version>2.0.0-mr1-cdh4.1.3</**version>
> >         </dependency>
> >         ...
> > </dependencies>
> > ...
> > <repositories>
> >         <repository>
> >                 <id>cloudera</id>
> >                 <url>https://repository.**cloudera.com/artifactory/**
> >
> cloudera-repos/<https://repository.cloudera.com/artifactory/cloudera-repos/>
> > </url>
> >                 <snapshots>
> >                         <enabled>false</enabled>
> >                 </snapshots>
> >         </repository>
> >
> >         <repository>
> >                 <id>spring-snapshot</id>
> >                 <name>Spring Maven SNAPSHOT Repository</name>
> >                
> <url>http://repo.**springframework.org/snapshot<http://repo.springframework.org/snapshot>
> > </**url>
> >         </repository>
> > </repositories>
> >
> > **Application Context**
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <beans
> xmlns="http://www.**springframework.org/schema/**beans<http://www.springframework.org/schema/beans>
> > "
> >        
> xmlns:xsi="http://www.w3.org/**2001/XMLSchema-instance<http://www.w3.org/2001/XMLSchema-instance>
> > "
> >        
> xmlns:hdp="http://www.**springframework.org/schema/**hadoop<http://www.springframework.org/schema/hadoop>"
> >
> xmlns:context="http://www.**springframework.org/schema/**context<http://www.springframework.org/schema/context>
> > "
> >        
> xmlns:hadoop="http://www.**springframework.org/schema/**hadoop<http://www.springframework.org/schema/hadoop>
> > "
> >         xsi:schemaLocation="
> >
> http://www.springframework.**org/schema/beans<http://www.springframework.org/schema/beans>
> >
> http://www.springframework.**org/schema/beans/spring-beans.**xsd<http://www.springframework.org/schema/beans/spring-beans.xsd>
> >
> http://www.springframework.**org/schema/hadoop<http://www.springframework.org/schema/hadoop>
> >
> http://www.springframework.**org/schema/hadoop/spring-**hadoop.xsd<http://www.springframework.org/schema/hadoop/spring-hadoop.xsd>
> >
> http://www.springframework.**org/schema/context/spring-**context.xsd<http://www.springframework.org/schema/context/spring-context.xsd>
> >
> http://www.springframework.**org/schema/integration<http://www.springframework.org/schema/integration>
> >
> http://www.springframework.**org/schema/context<http://www.springframework.org/schema/context>
> >
> http://www.springframework.**org/schema/context/spring-**context-3.1.xsd<http://www.springframework.org/schema/context/spring-context-3.1.xsd>
> > ">
> >
> >         <context:property-placeholder
> location="classpath:hadoop.**properties"
> > />
> >
> >         <hdp:configuration id="hadoopConfiguration">
> >                 fs.defaultFS=${hd.fs}
> >         </hdp:configuration>
> >
> >         <hdp:job id="wordCountJob" input-path="${input.path}"
> >                 output-path="${output.path}" mapper="com.example.**
> > WordMapper"
> >                 reducer="com.example.**WordReducer" />
> >
> > <hdp:job-runner job-ref="wordCountJob" run-at-startup="true"
> > wait-for-completion="true"/>
> >
> > </beans>
> >
> > **Cluster version**
> >
> > Hadoop 2.0.0-cdh4.1.3
> >
> >
> > **Note:**
> >
> > This small Unittest is running fine with the current configuration:
> >
> > @RunWith(**SpringJUnit4ClassRunner.class)
> > @ContextConfiguration(**locations = {
> "classpath:/**applicationContext.xml"
> > })
> > public class Starter {
> >
> >          @Autowired
> >          private Configuration configuration;
> >
> >          @Test
> >          public void shellOps() {
> >                  Assert.assertNotNull(this.**configuration);
> >                  FsShell fsShell = new FsShell(this.configuration);
> >                  final Collection<FileStatus> coll =
> fsShell.ls("/user");
> >                  System.out.println(coll);
> >          }
> > }
> >
> >
> > It would be nice if someone can give me an example configuration.
> >
> > Best Regards,
> > Christian.
> >
> >
> -- 
> Costin