giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Yost (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (GIRAPH-822) java.lang.ClassCastException: org.apache.giraph.conf.GiraphConfiguration cannot be cast to org.apache.hadoop.mapred.JobConf
Date Wed, 08 Jan 2014 16:58:50 GMT

     [ https://issues.apache.org/jira/browse/GIRAPH-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

John Yost updated GIRAPH-822:
-----------------------------

    Description: 
Hi Everyone,

I've run into this really weird error with the following stacktrace when I attempt to execute
a Giraph job via a custom Tool class:

Exception in thread "main" java.lang.ClassCastException: org.apache.giraph.conf.GiraphConfiguration
cannot be cast to org.apache.hadoop.mapred.JobConf
	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:513)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
	at org.apache.hadoop.mapreduce.Job.connect(Job.java:511)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:499)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
	at com.prototype.analytics.TwitterSpringRunner.run(TwitterSpringRunner.java:58)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at com.prototype.analytics.TwitterSpringRunner.main(TwitterSpringRunner.java:78)

UPDATE:
When I revert back to using GiraphRunner, the job completes successfully, so I must have dorked
up something along the way within my Tool class. 

Also, this ClassCastException is thrown when I make the following method invocation:
job.getInternalJob().waitForCompletion(false);

When I remove this, the job starts, but no tasks are executed

Here's the run method implementation:

	@Override
	public int run(String[] parameters) throws Exception 
	{		
		String jobName     = parameters[0];	
		String zConfig     = parameters[1];
		String inputPath   = parameters[2];
		String outputPath  = parameters[3];
		String seedVertex  = parameters[4];
		int numWorkers     = Integer.parseInt(parameters[5]);
				
		GiraphConfiguration gConf = new GiraphConfiguration(getConf());
		
		gConf.set("mapred.job.tracker","localhost:9001");
		gConf.set("fs.default.name","hdfs://localhost:9000");
		gConf.setVertexClass(com.johnyost.analytics.io.TwitterVertex.class);
		gConf.setMasterComputeClass(TwitterCompute.class);
		gConf.setZooKeeperConfiguration(zConfig);
		gConf.setWorkerConfiguration(numWorkers,numWorkers,100.0f);
		gConf.set("seedVertex",seedVertex);
		gConf.setVertexInputFormatClass((Class<? extends VertexInputFormat>) JsonTwitterInputFormat.class);
		gConf.setVertexOutputFormatClass((Class<? extends VertexOutputFormat>) org.apache.giraph.io.formats.IdWithValueTextOutputFormat.class);
		
		GiraphJob job = new GiraphJob(gConf,jobName);
			
		FileInputFormat.setInputPaths(job.getInternalJob(),new Path(inputPath));
	    FileOutputFormat.setOutputPath(job.getInternalJob(),new Path(outputPath));
	    
	    GiraphFileInputFormat.setVertexInputPath(gConf,new Path(inputPath));   
	    
	    log.info(job.getInternalJob().getConfiguration().get("mapred.job.tracker"));
	    log.info(job.getInternalJob().getConfiguration().get("fs.default.name"));
	    
	    job.getInternalJob().waitForCompletion(false);
	    
	    return job.run(true) ? 0 : -1;
	}

Any ideas as to why my custom Tool class is failing while GiraphRunner is not?  This was working,
but I must have changed something yesterday (not sure what it is, was not in this class) and
now I cannot get the Giraph job to execute.

  was:
Hi Everyone,

I've run into this really weird error with the following stacktrace:

Exception in thread "main" java.lang.ClassCastException: org.apache.giraph.conf.GiraphConfiguration
cannot be cast to org.apache.hadoop.mapred.JobConf
	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:513)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
	at org.apache.hadoop.mapreduce.Job.connect(Job.java:511)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:499)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
	at com.prototype.analytics.TwitterSpringRunner.run(TwitterSpringRunner.java:58)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at com.prototype.analytics.TwitterSpringRunner.main(TwitterSpringRunner.java:78)

Everything was working fine with my locally-built giraph version, but when I added the Giraph
1.0.0 maven dependency, this Exception was thrown.  Any ideas? 

UPDATE:
When I revert back to using GiraphRunner, the job completes successfully, so I must have dorked
up something along the way within my Tool class. 

Also, this ClassCastException is thrown when I make the following method invocation:
job.getInternalJob().waitForCompletion(false);

When I remove this, the job starts, but no tasks are executed

Here's the run method implementation:

	@Override
	public int run(String[] parameters) throws Exception 
	{		
		String jobName     = parameters[0];	
		String zConfig     = parameters[1];
		String inputPath   = parameters[2];
		String outputPath  = parameters[3];
		String seedVertex  = parameters[4];
		int numWorkers     = Integer.parseInt(parameters[5]);
				
		GiraphConfiguration gConf = new GiraphConfiguration(getConf());
		
		gConf.set("mapred.job.tracker","localhost:9001");
		gConf.set("fs.default.name","hdfs://localhost:9000");
		gConf.setVertexClass(com.johnyost.analytics.io.TwitterVertex.class);
		gConf.setMasterComputeClass(TwitterCompute.class);
		gConf.setZooKeeperConfiguration(zConfig);
		gConf.setWorkerConfiguration(numWorkers,numWorkers,100.0f);
		gConf.set("seedVertex",seedVertex);
		gConf.setVertexInputFormatClass((Class<? extends VertexInputFormat>) JsonTwitterInputFormat.class);
		gConf.setVertexOutputFormatClass((Class<? extends VertexOutputFormat>) org.apache.giraph.io.formats.IdWithValueTextOutputFormat.class);
		
		GiraphJob job = new GiraphJob(gConf,jobName);
			
		FileInputFormat.setInputPaths(job.getInternalJob(),new Path(inputPath));
	    FileOutputFormat.setOutputPath(job.getInternalJob(),new Path(outputPath));
	    
	    GiraphFileInputFormat.setVertexInputPath(gConf,new Path(inputPath));   
	    
	    log.info(job.getInternalJob().getConfiguration().get("mapred.job.tracker"));
	    log.info(job.getInternalJob().getConfiguration().get("fs.default.name"));
	    
	    job.getInternalJob().waitForCompletion(false);
	    
	    return job.run(true) ? 0 : -1;
	}

Any ideas as to why my custom Tool class is failing while GiraphRunner is not?


> java.lang.ClassCastException: org.apache.giraph.conf.GiraphConfiguration cannot be cast
to org.apache.hadoop.mapred.JobConf
> ---------------------------------------------------------------------------------------------------------------------------
>
>                 Key: GIRAPH-822
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-822
>             Project: Giraph
>          Issue Type: Bug
>          Components: mapreduce
>    Affects Versions: 1.0.0
>         Environment: CentOS 6.5
>            Reporter: John Yost
>
> Hi Everyone,
> I've run into this really weird error with the following stacktrace when I attempt to
execute a Giraph job via a custom Tool class:
> Exception in thread "main" java.lang.ClassCastException: org.apache.giraph.conf.GiraphConfiguration
cannot be cast to org.apache.hadoop.mapred.JobConf
> 	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:513)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> 	at org.apache.hadoop.mapreduce.Job.connect(Job.java:511)
> 	at org.apache.hadoop.mapreduce.Job.submit(Job.java:499)
> 	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
> 	at com.prototype.analytics.TwitterSpringRunner.run(TwitterSpringRunner.java:58)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> 	at com.prototype.analytics.TwitterSpringRunner.main(TwitterSpringRunner.java:78)
> UPDATE:
> When I revert back to using GiraphRunner, the job completes successfully, so I must have
dorked up something along the way within my Tool class. 
> Also, this ClassCastException is thrown when I make the following method invocation:
> job.getInternalJob().waitForCompletion(false);
> When I remove this, the job starts, but no tasks are executed
> Here's the run method implementation:
> 	@Override
> 	public int run(String[] parameters) throws Exception 
> 	{		
> 		String jobName     = parameters[0];	
> 		String zConfig     = parameters[1];
> 		String inputPath   = parameters[2];
> 		String outputPath  = parameters[3];
> 		String seedVertex  = parameters[4];
> 		int numWorkers     = Integer.parseInt(parameters[5]);
> 				
> 		GiraphConfiguration gConf = new GiraphConfiguration(getConf());
> 		
> 		gConf.set("mapred.job.tracker","localhost:9001");
> 		gConf.set("fs.default.name","hdfs://localhost:9000");
> 		gConf.setVertexClass(com.johnyost.analytics.io.TwitterVertex.class);
> 		gConf.setMasterComputeClass(TwitterCompute.class);
> 		gConf.setZooKeeperConfiguration(zConfig);
> 		gConf.setWorkerConfiguration(numWorkers,numWorkers,100.0f);
> 		gConf.set("seedVertex",seedVertex);
> 		gConf.setVertexInputFormatClass((Class<? extends VertexInputFormat>) JsonTwitterInputFormat.class);
> 		gConf.setVertexOutputFormatClass((Class<? extends VertexOutputFormat>) org.apache.giraph.io.formats.IdWithValueTextOutputFormat.class);
> 		
> 		GiraphJob job = new GiraphJob(gConf,jobName);
> 			
> 		FileInputFormat.setInputPaths(job.getInternalJob(),new Path(inputPath));
> 	    FileOutputFormat.setOutputPath(job.getInternalJob(),new Path(outputPath));
> 	    
> 	    GiraphFileInputFormat.setVertexInputPath(gConf,new Path(inputPath));   
> 	    
> 	    log.info(job.getInternalJob().getConfiguration().get("mapred.job.tracker"));
> 	    log.info(job.getInternalJob().getConfiguration().get("fs.default.name"));
> 	    
> 	    job.getInternalJob().waitForCompletion(false);
> 	    
> 	    return job.run(true) ? 0 : -1;
> 	}
> Any ideas as to why my custom Tool class is failing while GiraphRunner is not?  This
was working, but I must have changed something yesterday (not sure what it is, was not in
this class) and now I cannot get the Giraph job to execute.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message