giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Silvio Di gregorio <silvio.digrego...@gmail.com>
Subject Re: connected components example in giraph 1.0
Date Mon, 07 Oct 2013 18:39:52 GMT
Hi Matthew
I use giraph 1.1.0 and cdh4 hadoop 0.23

the patch of Marco:

http://www.mail-archive.com/**user@giraph.apache.org/**msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>

and build

mvn clean install -P hadoop_cdh4.1.2



Il giorno 07/ott/2013 20:20, "Matthew Laird" <lairdm@sfu.ca> ha scritto:

> Hello, so I'm watching this thread with interest as well, since that's the
> exact example I'll be trying to run as well, and I see Silvio appears to be
> asking about running Giraph 1.0 against Hadoop 2.0.0.
>
> I'm curious, is that a known working combination, 1.0 vs 2.0.x?  And if
> so, are there patches to apply first and what maven command line should I
> build with? (ie. will a -Dhadoop.version=2.0.5 give the needed MR based API
> or will it use YARN?)  I figure it's better to ask than aimlessly try all
> the combinations. :)
>
> Thanks!
>
> On 13-10-07 11:01 AM, Claudio Martella wrote:
>
>> try to increase the mappers' heap space through mapred.child.java.opts.
>>
>>
>> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>>>
>> wrote:
>>
>>
>>     I think now my issue is another, thanks to all.
>>
>>
>>
>>
>>     java.lang.**IllegalStateException: run: Caught an unrecoverable
>> exception waitFor: ExecutionException occurred while waiting for
>> org.apache.giraph.utils.**ProgressableUtils$**FutureWaitable@73a01e8a
>>         at org.apache.giraph.graph.**GraphMapper.run(GraphMapper.**
>> java:102)
>>
>>
>>
>>         ......
>>     Caused by: java.lang.**IllegalStateException: waitFor:
>> ExecutionException occurred while waiting for org.apache.giraph.utils.**
>> ProgressableUtils$**FutureWaitable@73a01e8a
>>         at org.apache.giraph.utils.**ProgressableUtils.waitFor(**
>> ProgressableUtils.java:151)
>>
>>
>>
>>         ... 7 more
>>     Caused by: java.util.concurrent.**ExecutionException:
>> java.lang.OutOfMemoryError:***Java heap space*
>>         at org.apache.giraph.utils.**ProgressableUtils.waitFor(**
>> ProgressableUtils.java:143)
>>
>>
>>
>>         ... 13 more
>>
>>
>>
>>     2013/10/7 Claudio Martella <claudio.martella@gmail.com
>>     <mailto:claudio.martella@**gmail.com <claudio.martella@gmail.com>>>
>>
>>         OK, thanks.
>>
>>         I really have to push that patch in.
>>
>>
>>         On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di gregorio
>>         <silvio.digregorio@gmail.com
>>         <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>>>
>> wrote:
>>
>>             yes i do,
>>             i have seen this in your post in:
>>             http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00957.html<http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>
>>
>>             excuse me if  i had checked in the mail-achive first I would
>>             have avoided the last post.
>>
>>             Now zk issue are resolved.
>>
>>
>>             2013/10/7 Claudio Martella <claudio.martella@gmail.com
>>             <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >>
>>
>>                 Can you try applying this one first?
>>
>> <<<               < http://www.mail-archive.com/**user@giraph.apache.org/
>> **msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>> <<<<
>>
>>
>>                 On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di gregorio
>>                 <silvio.digregorio@gmail.com
>>                 <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>>>
>> wrote:
>>
>>
>>
>>
>>
>>
>>
>>                     *As i said i have builded*
>>
>>
>>
>>
>>                     /giraph-examples-1.0.0-for-**
>> hadoop-2.0.0-cdh4.1.2-jar-**with-dependencies.jar/
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>                     *for cdh4, successfully. The job start to monitoring
>> the success rate:*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>                     /13/10/07 08:28:45 INFO mapred.JobClient:  map 0%
>> reduce 0%/
>>
>>
>>
>>
>>                     *but then*
>>
>>
>>
>>
>>                     /Error running child
>>                     java.lang.**IllegalStateException: run: Caught an
>> unrecoverable exception java.io.FileNotFoundException: File
>> _bsp/_defaultZkManagerDir/job_**201309181636_0678/_zkServer does not
>> exist.
>>                     /
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>                     /.../
>>
>>
>>
>>
>>
>>
>>
>>                     /Caused by: java.lang.RuntimeException:
>> java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_**201309181636_0678/_zkServer
>> does not exist./
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>                     2013/10/5 Silvio Di gregorio
>>                     <silvio.digregorio@gmail.com
>>                     <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>
>> >>
>>
>>                         I ha ve build w/ hadoop_cdh4.1.2 parameter.
>>                         Something is changed, monday i report the
>>                         result. Now the farm is closed.
>>
>>                         Il giorno 05/ott/2013 14:06, "Claudio Martella"
>>                         <claudio.martella@gmail.com
>>                         <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>>>
>> ha scritto:
>>
>>                             Oh, right, -vof is in trunk. Anyway it looks
>>                             like you built giraph for the wrong profile.
>>                             You mentioned you're running on 2.0, but
>>                             your giraph is built for 0.20.203
>>                             <tel:0.20.203>. try building with a profile
>>                             for your hadoop version.
>>
>>
>>                             On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di
>>                             gregorio <silvio.digregorio@gmail.com
>>                             <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>>>
>> wrote:
>>
>>
>>                                 org.apache.commons.cli.**
>> UnrecognizedOptionException:
>>                                 Unrecognized option: -vof
>>
>>                                 in 1.0 version is
>>                                   -of,--outputFormat <arg>
>>                                 Vertex output format
>>                                   -op,--outputPath <arg>
>>                                 Vertex output path
>>
>>
>>
>>                                 2013/10/4 Claudio Martella
>>                                 <claudio.martella@gmail.com
>>                                 <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >>
>>
>>                                     did you try the argument (-vof) i
>>                                     suggested?
>>
>>
>>                                     On Fri, Oct 4, 2013 at 2:13 PM,
>>                                     Silvio Di gregorio
>>                                     <silvio.digregorio@gmail.com
>>                                     <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>>>
>> wrote:
>>
>>
>>                                         i've specified  -of
>>                                           org.apache.giraph.io.formats.**
>> IdWithValueTextOutputFormat
>>
>>                                         but the same error was produced
>>
>>                                         Exception in thread "main"
>>                                         java.lang.**
>> IncompatibleClassChangeError:
>>                                         Found interface
>>                                         org.apache.hadoop.mapreduce.**
>> JobContext,
>>                                         but class was expected
>>                                                  at
>>                                         org.apache.giraph.bsp.**
>> BspOutputFormat.**checkOutputSpecs(**BspOutputFormat.java:43)
>>                                                  at
>>                                         org.apache.hadoop.mapred.**
>> JobClient$2.run(JobClient.**java:984)
>>                                                  at
>>                                         org.apache.hadoop.mapred.**
>> JobClient$2.run(JobClient.**java:945)
>>                                                  at
>>                                         java.security.**
>> AccessController.doPrivileged(**Native
>>                                         Method)
>>                                                  at
>>                                         javax.security.auth.Subject.**
>> doAs(Subject.java:396)
>>                                                  at
>>                                         org.apache.hadoop.security.**
>> UserGroupInformation.doAs(**UserGroupInformation.java:**1408)
>>                                                  at
>>                                         org.apache.hadoop.mapred.**
>> JobClient.submitJobInternal(**JobClient.java:945)
>>                                                  at
>>                                         org.apache.hadoop.mapreduce.**
>> Job.submit(Job.java:566)
>>                                                  at
>>                                         org.apache.hadoop.mapreduce.**
>> Job.waitForCompletion(Job.**java:596)
>>                                                  at
>>                                         org.apache.giraph.job.**
>> GiraphJob.run(GiraphJob.java:**237)
>>                                                  at
>>                                         org.apache.giraph.**
>> GiraphRunner.run(GiraphRunner.**java:94)
>>                                                  at
>>                                         org.apache.hadoop.util.**
>> ToolRunner.run(ToolRunner.**java:70)
>>                                                  at
>>                                         org.apache.hadoop.util.**
>> ToolRunner.run(ToolRunner.**java:84)
>>                                                  at
>>                                         org.apache.giraph.**
>> GiraphRunner.main(**GiraphRunner.java:124)
>>                                                  at
>>                                         sun.reflect.**
>> NativeMethodAccessorImpl.**invoke0(Native
>>                                         Method)
>>                                                  at
>>                                         sun.reflect.**
>> NativeMethodAccessorImpl.**invoke(**NativeMethodAccessorImpl.java:**39)
>>                                                  at
>>                                         sun.reflect.**
>> DelegatingMethodAccessorImpl.**invoke(**DelegatingMethodAccessorImpl.**
>> java:25)
>>                                                  at
>>                                         java.lang.reflect.Method.**
>> invoke(Method.java:597)
>>                                                  at
>>                                         org.apache.hadoop.util.RunJar.**
>> main(RunJar.java:208)
>>
>>
>>
>>                                         2013/10/4 Claudio Martella
>>                                         <claudio.martella@gmail.com
>>                                         <mailto:claudio.martella@**
>> gmail.com <claudio.martella@gmail.com>>>
>>
>>                                             Hi,
>>
>>                                             you need to specify the
>>                                             vertex outputformat class
>>                                             (-vof option), e.g.
>>                                             org.apache.giraph.io.formats.
>> **IdWithValueTextOutputFormat.
>>
>>
>>                                             On Fri, Oct 4, 2013 at 1:06
>>                                             PM, Silvio Di gregorio
>>                                             <silvio.digregorio@gmail.com
>>                                             <mailto:silvio.digregorio@**
>> gmail.com <silvio.digregorio@gmail.com>>>
>>                                             wrote:
>>
>>
>>                                                 Hi,
>>
>>                                                 I hope I have sent to
>>                                                 the right address.
>>
>>                                                 i have a graph (directed
>>                                                 and unweighted) stored
>>                                                 in hdfs like a adjacency
>>                                                 list (140Milions of
>>                                                 edges 6Milions of vertex)
>>
>>                                                 node<tab>neighbors
>>
>>                                                 23   2   1    3    4    3
>>
>>                                                 1 999 99923
>>                                                 <tel:1%20999%2099923>
>>                                                 909 .......
>>
>>                                                 ......
>>
>>                                                 hadoop version Hadoop
>>                                                 2.0.0-cdh4.3.0 - java 1.6
>>
>>
>>                                                 I have executed the
>>                                                 giraph-1.0 connected
>>                                                 components example, in
>>                                                 this fashion
>>
>>                                                     >  hadoop jar
>>
>> /usr/local/giraph/giraph-**examples/target/giraph-**
>> examples-1.0.0-for-hadoop-0.20.203.0
>>                                                     <tel:0.20.203.0
>> >-jar-with-dependencies.**jar
>>
>> org.apache.giraph.GiraphRunner
>>
>> org.apache.giraph.examples.**ConnectedComponentsVertex
>>                                                     -vif
>>
>> org.apache.giraph.io.formats.**IntIntNullTextInputFormat
>>                                                     -vip
>>
>> /user/hdfs/lista_adj_txt
>>                                                     -op connectedgiraph
>>                                                     --workers 4
>>
>>                                                 and then fail with:
>>
>>                                                 13/10/04 09:28:29 INFO
>>                                                 utils.ConfigurationUtils:
>> No
>>                                                 edge input format
>>                                                 specified. Ensure your
>>                                                 InputFormat does not
>>                                                 require one.
>>
>>                                                 13/10/04 09:28:29 INFO
>>                                                 utils.ConfigurationUtils:
>> No
>>                                                 output format specified.
>>                                                 Ensure your OutputFormat
>>                                                 does not require one.
>>
>>                                                 13/10/04 09:28:30 INFO
>>                                                 job.GiraphJob: run:
>>                                                 Since checkpointing is
>>                                                 disabled (default), do
>>                                                 not allow any task
>>                                                 retries (setting
>>                                                 mapred.map.max.attempts
>>                                                 = 0, old value = 4)
>>
>>                                                 13/10/04 09:28:31 WARN
>>                                                 mapred.JobClient: Use
>>                                                 GenericOptionsParser for
>>                                                 parsing the arguments.
>>                                                 Applications should
>>                                                 implement Tool for the
>> same.
>>
>>                                                 13/10/04 09:28:31 INFO
>>                                                 mapred.JobClient:
>>                                                 Cleaning up the staging
>>                                                 area
>>                                                 hdfs://
>> srv-bigdata-dev-01.int.**sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>>                                                 <
>> http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>> >
>>
>>                                                 Exception in thread
>>                                                 "main"
>>                                                 java.lang.**
>> IncompatibleClassChangeError:
>>                                                 Found interface
>>
>> org.apache.hadoop.mapreduce.**JobContext,
>>                                                 but class was expected
>>
>>                                                          at
>>                                                 org.apache.giraph.bsp.**
>> BspOutputFormat.**checkOutputSpecs(**BspOutputFormat.java:43)
>>
>>                                                 ......
>>
>>                                                 Thanks in advance
>>
>>
>>
>>
>>
>>                                             --
>>                                                 Claudio Martella
>>                                             claudio.martella@gmail.com
>>                                             <mailto:claudio.martella@**
>> gmail.com <claudio.martella@gmail.com>>
>>
>>
>>
>>
>>
>>
>>                                     --
>>                                         Claudio Martella
>>                                     claudio.martella@gmail.com
>>                                     <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>
>>
>>
>>
>>
>>                             --
>>                                 Claudio Martella
>>                             claudio.martella@gmail.com
>>                             <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>
>>
>>
>>
>>
>>                 --
>>                     Claudio Martella
>>                 claudio.martella@gmail.com
>>                 <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>
>>
>>
>>
>>
>>         --
>>             Claudio Martella
>>         claudio.martella@gmail.com <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>
>>
>>
>>
>>
>> --
>>     Claudio Martella
>> claudio.martella@gmail.com <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>
>
> --
> Matthew Laird
> Lead Software Developer, Bioinformatics
> Brinkman Laboratory
> Simon Fraser University, Burnaby, BC, Canada
>

Mime
View raw message