giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Laird <lai...@sfu.ca>
Subject Re: connected components example in giraph 1.0
Date Mon, 07 Oct 2013 18:48:43 GMT
Thanks, which is my next curiosity... if I head over to 
http://giraph.apache.org/ I see at the top is says

"Version: 1.1.0-SNAPSHOT | Last Published: 2013-08-10"

However it's not clickable and the only download links I see are for 
1.0.0.  I'm assuming trunk in git isn't the same as the 1.1.0 snapshot 
since more pushes have occurred since August, and I don't see a tag in 
git for 1.1.0 (https://git-wip-us.apache.org/repos/asf?p=giraph.git). 
So where does one get 1.1.0-SNAPSHOT from?

Thanks.

On 13-10-07 11:39 AM, Silvio Di gregorio wrote:
> Hi Matthew
> I use giraph 1.1.0 and cdh4 hadoop 0.23
>
> the patch of Marco:
>
> http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff
> <http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>
> and build
>
> mvn clean install -P hadoop_cdh4.1.2
>
>
>
> Il giorno 07/ott/2013 20:20, "Matthew Laird" <lairdm@sfu.ca
> <mailto:lairdm@sfu.ca>> ha scritto:
>
>     Hello, so I'm watching this thread with interest as well, since
>     that's the exact example I'll be trying to run as well, and I see
>     Silvio appears to be asking about running Giraph 1.0 against Hadoop
>     2.0.0.
>
>     I'm curious, is that a known working combination, 1.0 vs 2.0.x?  And
>     if so, are there patches to apply first and what maven command line
>     should I build with? (ie. will a -Dhadoop.version=2.0.5 give the
>     needed MR based API or will it use YARN?)  I figure it's better to
>     ask than aimlessly try all the combinations. :)
>
>     Thanks!
>
>     On 13-10-07 11:01 AM, Claudio Martella wrote:
>
>         try to increase the mappers' heap space through
>         mapred.child.java.opts.
>
>
>         On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
>         <silvio.digregorio@gmail.com
>         <mailto:silvio.digregorio@gmail.com>
>         <mailto:silvio.digregorio@__gmail.com
>         <mailto:silvio.digregorio@gmail.com>>> wrote:
>
>
>              I think now my issue is another, thanks to all.
>
>
>
>
>              java.lang.__IllegalStateException: run: Caught an
>         unrecoverable exception waitFor: ExecutionException occurred
>         while waiting for
>         org.apache.giraph.utils.__ProgressableUtils$__FutureWaitable@73a01e8a
>                  at
>         org.apache.giraph.graph.__GraphMapper.run(GraphMapper.__java:102)
>
>
>
>                  ......
>              Caused by: java.lang.__IllegalStateException: waitFor:
>         ExecutionException occurred while waiting for
>         org.apache.giraph.utils.__ProgressableUtils$__FutureWaitable@73a01e8a
>                  at
>         org.apache.giraph.utils.__ProgressableUtils.waitFor(__ProgressableUtils.java:151)
>
>
>
>                  ... 7 more
>              Caused by: java.util.concurrent.__ExecutionException:
>         java.lang.OutOfMemoryError:*__Java heap space*
>                  at
>         org.apache.giraph.utils.__ProgressableUtils.waitFor(__ProgressableUtils.java:143)
>
>
>
>                  ... 13 more
>
>
>
>              2013/10/7 Claudio Martella <claudio.martella@gmail.com
>         <mailto:claudio.martella@gmail.com>
>         <mailto:claudio.martella@__gmail.com
>         <mailto:claudio.martella@gmail.com>>>
>
>                  OK, thanks.
>
>                  I really have to push that patch in.
>
>
>                  On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di gregorio
>         <silvio.digregorio@gmail.com <mailto:silvio.digregorio@gmail.com>
>         <mailto:silvio.digregorio@__gmail.com
>         <mailto:silvio.digregorio@gmail.com>>> wrote:
>
>                      yes i do,
>                      i have seen this in your post in:
>         http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html
>         <http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>
>
>                      excuse me if  i had checked in the mail-achive
>         first I would
>                      have avoided the last post.
>
>                      Now zk issue are resolved.
>
>
>                      2013/10/7 Claudio Martella
>         <claudio.martella@gmail.com <mailto:claudio.martella@gmail.com>
>         <mailto:claudio.martella@__gmail.com
>         <mailto:claudio.martella@gmail.com>>>
>
>                          Can you try applying this one first?
>
>         <<< <
>         http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff
>         <http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff><<<<
>
>
>                          On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di gregorio
>         <silvio.digregorio@gmail.com <mailto:silvio.digregorio@gmail.com>
>         <mailto:silvio.digregorio@__gmail.com
>         <mailto:silvio.digregorio@gmail.com>>> wrote:
>
>
>
>
>
>
>
>                              *As i said i have builded*
>
>
>
>
>
>         /giraph-examples-1.0.0-for-__hadoop-2.0.0-cdh4.1.2-jar-__with-dependencies.jar/
>
>
>
>
>
>
>
>
>
>
>
>
>
>                              *for cdh4, successfully. The job start to
>         monitoring the success rate:*
>
>
>
>
>
>
>
>
>
>
>
>
>
>                              /13/10/07 08:28:45 INFO mapred.JobClient:
>           map 0% reduce 0%/
>
>
>
>
>                              *but then*
>
>
>
>
>                              /Error running child
>                              java.lang.__IllegalStateException: run:
>         Caught an unrecoverable exception java.io.FileNotFoundException:
>         File _bsp/_defaultZkManagerDir/job___201309181636_0678/_zkServer
>         does not exist.
>                              /
>
>
>
>
>
>
>
>
>
>
>
>
>
>                              /.../
>
>
>
>
>
>
>
>                              /Caused by: java.lang.RuntimeException:
>         java.io.FileNotFoundException: File
>         _bsp/_defaultZkManagerDir/job___201309181636_0678/_zkServer does
>         not exist./
>
>
>
>
>
>
>
>
>
>
>                              2013/10/5 Silvio Di gregorio
>         <silvio.digregorio@gmail.com <mailto:silvio.digregorio@gmail.com>
>         <mailto:silvio.digregorio@__gmail.com
>         <mailto:silvio.digregorio@gmail.com>>>
>
>                                  I ha ve build w/ hadoop_cdh4.1.2 parameter.
>                                  Something is changed, monday i report the
>                                  result. Now the farm is closed.
>
>                                  Il giorno 05/ott/2013 14:06, "Claudio
>         Martella"
>         <claudio.martella@gmail.com <mailto:claudio.martella@gmail.com>
>         <mailto:claudio.martella@__gmail.com
>         <mailto:claudio.martella@gmail.com>>> ha scritto:
>
>                                      Oh, right, -vof is in trunk. Anyway
>         it looks
>                                      like you built giraph for the wrong
>         profile.
>                                      You mentioned you're running on
>         2.0, but
>                                      your giraph is built for 0.20.203
>         <tel:0.20.203>
>         <tel:0.20.203 <tel:0.20.203>>. try building with a profile
>                                      for your hadoop version.
>
>
>                                      On Fri, Oct 4, 2013 at 2:35 PM,
>         Silvio Di
>                                      gregorio
>         <silvio.digregorio@gmail.com <mailto:silvio.digregorio@gmail.com>
>         <mailto:silvio.digregorio@__gmail.com
>         <mailto:silvio.digregorio@gmail.com>>> wrote:
>
>
>
>         org.apache.commons.cli.__UnrecognizedOptionException:
>                                          Unrecognized option: -vof
>
>                                          in 1.0 version is
>                                            -of,--outputFormat <arg>
>                                          Vertex output format
>                                            -op,--outputPath <arg>
>                                          Vertex output path
>
>
>
>                                          2013/10/4 Claudio Martella
>         <claudio.martella@gmail.com <mailto:claudio.martella@gmail.com>
>         <mailto:claudio.martella@__gmail.com
>         <mailto:claudio.martella@gmail.com>>>
>
>                                              did you try the argument
>         (-vof) i
>                                              suggested?
>
>
>                                              On Fri, Oct 4, 2013 at 2:13 PM,
>                                              Silvio Di gregorio
>         <silvio.digregorio@gmail.com <mailto:silvio.digregorio@gmail.com>
>         <mailto:silvio.digregorio@__gmail.com
>         <mailto:silvio.digregorio@gmail.com>>> wrote:
>
>
>                                                  i've specified  -of
>
>         org.apache.giraph.io.formats.__IdWithValueTextOutputFormat
>
>                                                  but the same error was
>         produced
>
>                                                  Exception in thread "main"
>
>         java.lang.__IncompatibleClassChangeError:
>                                                  Found interface
>
>         org.apache.hadoop.mapreduce.__JobContext,
>                                                  but class was expected
>                                                           at
>
>         org.apache.giraph.bsp.__BspOutputFormat.__checkOutputSpecs(__BspOutputFormat.java:43)
>                                                           at
>
>         org.apache.hadoop.mapred.__JobClient$2.run(JobClient.__java:984)
>                                                           at
>
>         org.apache.hadoop.mapred.__JobClient$2.run(JobClient.__java:945)
>                                                           at
>
>         java.security.__AccessController.doPrivileged(__Native
>                                                  Method)
>                                                           at
>
>         javax.security.auth.Subject.__doAs(Subject.java:396)
>                                                           at
>
>         org.apache.hadoop.security.__UserGroupInformation.doAs(__UserGroupInformation.java:__1408)
>                                                           at
>
>         org.apache.hadoop.mapred.__JobClient.submitJobInternal(__JobClient.java:945)
>                                                           at
>
>         org.apache.hadoop.mapreduce.__Job.submit(Job.java:566)
>                                                           at
>
>         org.apache.hadoop.mapreduce.__Job.waitForCompletion(Job.__java:596)
>                                                           at
>
>         org.apache.giraph.job.__GiraphJob.run(GiraphJob.java:__237)
>                                                           at
>
>         org.apache.giraph.__GiraphRunner.run(GiraphRunner.__java:94)
>                                                           at
>
>         org.apache.hadoop.util.__ToolRunner.run(ToolRunner.__java:70)
>                                                           at
>
>         org.apache.hadoop.util.__ToolRunner.run(ToolRunner.__java:84)
>                                                           at
>
>         org.apache.giraph.__GiraphRunner.main(__GiraphRunner.java:124)
>                                                           at
>
>         sun.reflect.__NativeMethodAccessorImpl.__invoke0(Native
>                                                  Method)
>                                                           at
>
>         sun.reflect.__NativeMethodAccessorImpl.__invoke(__NativeMethodAccessorImpl.java:__39)
>                                                           at
>
>         sun.reflect.__DelegatingMethodAccessorImpl.__invoke(__DelegatingMethodAccessorImpl.__java:25)
>                                                           at
>
>         java.lang.reflect.Method.__invoke(Method.java:597)
>                                                           at
>
>         org.apache.hadoop.util.RunJar.__main(RunJar.java:208)
>
>
>
>                                                  2013/10/4 Claudio Martella
>         <claudio.martella@gmail.com <mailto:claudio.martella@gmail.com>
>         <mailto:claudio.martella@__gmail.com
>         <mailto:claudio.martella@gmail.com>>>
>
>                                                      Hi,
>
>                                                      you need to specify the
>                                                      vertex outputformat
>         class
>                                                      (-vof option), e.g.
>
>         org.apache.giraph.io.formats.__IdWithValueTextOutputFormat.
>
>
>                                                      On Fri, Oct 4, 2013
>         at 1:06
>                                                      PM, Silvio Di gregorio
>         <silvio.digregorio@gmail.com <mailto:silvio.digregorio@gmail.com>
>         <mailto:silvio.digregorio@__gmail.com
>         <mailto:silvio.digregorio@gmail.com>>>
>                                                      wrote:
>
>
>                                                          Hi,
>
>                                                          I hope I have
>         sent to
>                                                          the right address.
>
>                                                          i have a graph
>         (directed
>                                                          and unweighted)
>         stored
>                                                          in hdfs like a
>         adjacency
>                                                          list (140Milions of
>                                                          edges 6Milions
>         of vertex)
>
>                                                          node<tab>neighbors
>
>                                                          23   2   1    3
>             4    3
>
>         1 999 99923 <tel:1%20999%2099923>
>         <tel:1%20999%2099923>
>                                                          909 .......
>
>                                                          ......
>
>                                                          hadoop version
>         Hadoop
>                                                          2.0.0-cdh4.3.0
>         - java 1.6
>
>
>                                                          I have executed the
>                                                          giraph-1.0
>         connected
>                                                          components
>         example, in
>                                                          this fashion
>
>          >  hadoop jar
>
>         /usr/local/giraph/giraph-__examples/target/giraph-__examples-1.0.0-for-hadoop-0.20.203.0
>         <tel:0.20.203.0>
>         <tel:0.20.203.0 <tel:0.20.203.0>>-jar-with-dependencies.__jar
>
>         org.apache.giraph.GiraphRunner
>
>         org.apache.giraph.examples.__ConnectedComponentsVertex
>                                                              -vif
>
>         org.apache.giraph.io.formats.__IntIntNullTextInputFormat
>                                                              -vip
>
>         /user/hdfs/lista_adj_txt
>                                                              -op
>         connectedgiraph
>                                                              --workers 4
>
>                                                          and then fail with:
>
>                                                          13/10/04
>         09:28:29 INFO
>
>         utils.ConfigurationUtils: No
>                                                          edge input format
>                                                          specified.
>         Ensure your
>                                                          InputFormat
>         does not
>                                                          require one.
>
>                                                          13/10/04
>         09:28:29 INFO
>
>         utils.ConfigurationUtils: No
>                                                          output format
>         specified.
>                                                          Ensure your
>         OutputFormat
>                                                          does not
>         require one.
>
>                                                          13/10/04
>         09:28:30 INFO
>                                                          job.GiraphJob: run:
>                                                          Since
>         checkpointing is
>                                                          disabled
>         (default), do
>                                                          not allow any task
>                                                          retries (setting
>
>         mapred.map.max.attempts
>                                                          = 0, old value = 4)
>
>                                                          13/10/04
>         09:28:31 WARN
>
>         mapred.JobClient: Use
>
>         GenericOptionsParser for
>                                                          parsing the
>         arguments.
>                                                          Applications should
>                                                          implement Tool
>         for the same.
>
>                                                          13/10/04
>         09:28:31 INFO
>                                                          mapred.JobClient:
>                                                          Cleaning up the
>         staging
>                                                          area
>
>         hdfs://srv-bigdata-dev-01.int.__sose.it:8020/user/hdfs/.__staging/job_201309181636_0535
>         <http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>         <http://srv-bigdata-dev-01.__int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535
>         <http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>>
>
>                                                          Exception in thread
>         "main"
>
>         java.lang.__IncompatibleClassChangeError:
>                                                          Found interface
>
>         org.apache.hadoop.mapreduce.__JobContext,
>                                                          but class was
>         expected
>
>                                                                   at
>
>         org.apache.giraph.bsp.__BspOutputFormat.__checkOutputSpecs(__BspOutputFormat.java:43)
>
>                                                          ......
>
>                                                          Thanks in advance
>
>
>
>
>
>                                                      --
>                                                          Claudio Martella
>         claudio.martella@gmail.com <mailto:claudio.martella@gmail.com>
>         <mailto:claudio.martella@__gmail.com
>         <mailto:claudio.martella@gmail.com>>
>
>
>
>
>
>
>                                              --
>                                                  Claudio Martella
>         claudio.martella@gmail.com <mailto:claudio.martella@gmail.com>
>         <mailto:claudio.martella@__gmail.com
>         <mailto:claudio.martella@gmail.com>>
>
>
>
>
>
>                                      --
>                                          Claudio Martella
>         claudio.martella@gmail.com <mailto:claudio.martella@gmail.com>
>         <mailto:claudio.martella@__gmail.com
>         <mailto:claudio.martella@gmail.com>>
>
>
>
>
>
>                          --
>                              Claudio Martella
>         claudio.martella@gmail.com <mailto:claudio.martella@gmail.com>
>         <mailto:claudio.martella@__gmail.com
>         <mailto:claudio.martella@gmail.com>>
>
>
>
>
>
>                  --
>                      Claudio Martella
>         claudio.martella@gmail.com <mailto:claudio.martella@gmail.com>
>         <mailto:claudio.martella@__gmail.com
>         <mailto:claudio.martella@gmail.com>>
>
>
>
>
>
>         --
>              Claudio Martella
>         claudio.martella@gmail.com <mailto:claudio.martella@gmail.com>
>         <mailto:claudio.martella@__gmail.com
>         <mailto:claudio.martella@gmail.com>>
>
>
>     --
>     Matthew Laird
>     Lead Software Developer, Bioinformatics
>     Brinkman Laboratory
>     Simon Fraser University, Burnaby, BC, Canada
>

-- 
Matthew Laird
Lead Software Developer, Bioinformatics
Brinkman Laboratory
Simon Fraser University, Burnaby, BC, Canada

Mime
View raw message