Mailing-List: contact giraph-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: giraph-user@incubator.apache.org
Message-ID: <4F6976B7.1090100@apache.org>
Date: Tue, 20 Mar 2012 23:35:35 -0700
From: Avery Ching <aching@apache.org>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
 rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2
MIME-Version: 1.0
To: giraph-user@incubator.apache.org
Subject: Re: Problem deploying Giraph job to hadoop cluster:
 onlineZooKeeperServers
 connection failure
References: <6C1FACC6-A54C-4ADC-A743-77C72B4DCCFC@deri.org>
In-Reply-To: <6C1FACC6-A54C-4ADC-A743-77C72B4DCCFC@deri.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Benjamin, my guess is that your jar might not have all the ZooKeeper 
dependencies.  Can you look at the log for the process that was supposed 
to start ZooKeeper?  I'm thinking it didn't start...

Avery

On 3/20/12 1:14 PM, Benjamin Heitmann wrote:
> Hello,
>
> after getting my feet wet with the InternalVertexRunner, I tried packaging a Giraph job as a jar for the first time.
>
> I am getting the following error:
>
> ==========
> 12/03/20 17:21:04 INFO mapred.JobClient: Task Id : attempt_201203201422_0009_m_000000_2, Status : FAILED
> java.lang.IllegalStateException: onlineZooKeeperServers: Failed to connect in 10 tries!
> 	at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:687)
> 	at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:425)
> 	at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:646)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
> attempt_201203201422_0009_m_000000_2: log4j:WARN No appenders could be found for logger (org.apache.giraph.zk.ZooKeeperManager).
> attempt_201203201422_0009_m_000000_2: log4j:WARN Please initialize the log4j system properly.
> =================
>
> Here is some more information, which hopefully might give this mailing list some insight into what is happening,
> because I cant figure it out...
>
> * I am using Hadoop 1.0.1 and giraph svn revision 1293545 (the last one from February)
> * If I run the same Vertex class and Input/OutputFormat using InternalVertexRunner, then everything works fine. (using again Hadoop 1.0.1 and giraph rev 1293545)
> * I package the giraph job as a selfcontaining jar, and it contains the giraph jar, as well as the zookeeper jar in its lib dir
> (I mostly used the recipe from here https://exported.wordpress.com/2010/01/30/building-hadoop-job-jar-with-maven/ )
> * there was an error in which hadoop could not find a class. And I had to fix that error with:
> giraphJob.setJarByClass(SimpleRDFVertex.class);
> * My Vertex class extends HashMapVertex<Text, Text, Text, NullWritable>
> * I followed the code example from SimpleShortestPathVertex regarding the run() method and using the main method to call ToolRunner.run()
>
> Here is the code for my run() method:
>
> ==============
> 	@Override
> 	public int run(String[] args) throws Exception {
> 		// takes 3 args: inputDir outputDir numberOfWorkers
> 		
> 	    GiraphJob job = new GiraphJob(getConf(), getClass().getName());
> 	
> 	    job.setJarByClass(SimpleRDFVertex.class);
> 	
> 	    job.setVertexClass(SimpleRDFVertex.class);
> 	    job.setVertexInputFormatClass(SimpleRDFVertexInputFormat.class);
> 	    job.setVertexOutputFormatClass(SimpleRDFVertexOutputFormat.class);
> 	
> 	    FileInputFormat.addInputPath(job, new Path(args[0]));
> 	    FileOutputFormat.setOutputPath(job, new Path(args[1]));
> 	    job.setWorkerConfiguration(Integer.parseInt(args[2]), Integer.parseInt(args[2]), 100.0f);
>
> 	    return job.run(true) ? 0 : -1;
> 	}
> ==========
>
> Am I constructing the GiraphJob in the wrong way ?
>
> I saw the GiraphRunner class, but the giraph source tree currently does not seem to contain an example of how to use that class.
> Is it safer to use that class for starting a GiraphJob ?
> If yes, how should the job jar be assembled in order to use GiraphRunner ?
>
>
> sincerely, Benjamin Heitmann.
>