giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: connected components example in giraph 1.0
Date Mon, 07 Oct 2013 18:50:51 GMT
trunk is 1.1.0-SNAPSHOT. I believe the date refers to the last update to
the site.


On Mon, Oct 7, 2013 at 8:48 PM, Matthew Laird <lairdm@sfu.ca> wrote:

> Thanks, which is my next curiosity... if I head over to
> http://giraph.apache.org/ I see at the top is says
>
> "Version: 1.1.0-SNAPSHOT | Last Published: 2013-08-10"
>
> However it's not clickable and the only download links I see are for
> 1.0.0.  I'm assuming trunk in git isn't the same as the 1.1.0 snapshot
> since more pushes have occurred since August, and I don't see a tag in git
> for 1.1.0 (https://git-wip-us.apache.**org/repos/asf?p=giraph.git<https://git-wip-us.apache.org/repos/asf?p=giraph.git>).
> So where does one get 1.1.0-SNAPSHOT from?
>
> Thanks.
>
>
> On 13-10-07 11:39 AM, Silvio Di gregorio wrote:
>
>> Hi Matthew
>> I use giraph 1.1.0 and cdh4 hadoop 0.23
>>
>> the patch of Marco:
>>
>> http://www.mail-archive.com/__**user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>
>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>> >
>>
>> and build
>>
>> mvn clean install -P hadoop_cdh4.1.2
>>
>>
>>
>> Il giorno 07/ott/2013 20:20, "Matthew Laird" <lairdm@sfu.ca
>> <mailto:lairdm@sfu.ca>> ha scritto:
>>
>>
>>     Hello, so I'm watching this thread with interest as well, since
>>     that's the exact example I'll be trying to run as well, and I see
>>     Silvio appears to be asking about running Giraph 1.0 against Hadoop
>>     2.0.0.
>>
>>     I'm curious, is that a known working combination, 1.0 vs 2.0.x?  And
>>     if so, are there patches to apply first and what maven command line
>>     should I build with? (ie. will a -Dhadoop.version=2.0.5 give the
>>     needed MR based API or will it use YARN?)  I figure it's better to
>>     ask than aimlessly try all the combinations. :)
>>
>>     Thanks!
>>
>>     On 13-10-07 11:01 AM, Claudio Martella wrote:
>>
>>         try to increase the mappers' heap space through
>>         mapred.child.java.opts.
>>
>>
>>         On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
>>         <silvio.digregorio@gmail.com
>>         <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>
>> >
>>         <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>
>>         <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>>>>
>> wrote:
>>
>>
>>              I think now my issue is another, thanks to all.
>>
>>
>>
>>
>>              java.lang.__**IllegalStateException: run: Caught an
>>
>>         unrecoverable exception waitFor: ExecutionException occurred
>>         while waiting for
>>         org.apache.giraph.utils.__**ProgressableUtils$__**
>> FutureWaitable@73a01e8a
>>                  at
>>         org.apache.giraph.graph.__**GraphMapper.run(GraphMapper.__**
>> java:102)
>>
>>
>>
>>                  ......
>>              Caused by: java.lang.__**IllegalStateException: waitFor:
>>
>>         ExecutionException occurred while waiting for
>>         org.apache.giraph.utils.__**ProgressableUtils$__**
>> FutureWaitable@73a01e8a
>>                  at
>>         org.apache.giraph.utils.__**ProgressableUtils.waitFor(__**
>> ProgressableUtils.java:151)
>>
>>
>>
>>                  ... 7 more
>>              Caused by: java.util.concurrent.__**ExecutionException:
>>         java.lang.OutOfMemoryError:*__**Java heap space*
>>                  at
>>         org.apache.giraph.utils.__**ProgressableUtils.waitFor(__**
>> ProgressableUtils.java:143)
>>
>>
>>
>>
>>                  ... 13 more
>>
>>
>>
>>              2013/10/7 Claudio Martella <claudio.martella@gmail.com
>>         <mailto:claudio.martella@**gmail.com <claudio.martella@gmail.com>
>> >
>>         <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>>         <mailto:claudio.martella@**gmail.com <claudio.martella@gmail.com>
>> >>>
>>
>>                  OK, thanks.
>>
>>                  I really have to push that patch in.
>>
>>
>>                  On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di gregorio
>>         <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <silvio.digregorio@gmail.com>>
>>          <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>
>>         <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>>>>
>> wrote:
>>
>>                      yes i do,
>>                      i have seen this in your post in:
>>         http://www.mail-archive.com/__**user@giraph.apache.org/__**
>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>>
>>         <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00957.html<http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>
>> >
>>
>>                      excuse me if  i had checked in the mail-achive
>>         first I would
>>                      have avoided the last post.
>>
>>                      Now zk issue are resolved.
>>
>>
>>                      2013/10/7 Claudio Martella
>>         <claudio.martella@gmail.com <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>         <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>>         <mailto:claudio.martella@**gmail.com <claudio.martella@gmail.com>
>> >>>
>>
>>                          Can you try applying this one first?
>>
>>         <<< <
>>         http://www.mail-archive.com/__**user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>
>>         <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>> ><<<<
>>
>>
>>                          On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di
>> gregorio
>>         <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <silvio.digregorio@gmail.com>>
>>          <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>
>>         <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>>>>
>> wrote:
>>
>>
>>
>>
>>
>>
>>
>>                              *As i said i have builded*
>>
>>
>>
>>
>>
>>         /giraph-examples-1.0.0-for-__**hadoop-2.0.0-cdh4.1.2-jar-__**
>> with-dependencies.jar/
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>                              *for cdh4, successfully. The job start to
>>         monitoring the success rate:*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>                              /13/10/07 08:28:45 INFO mapred.JobClient:
>>           map 0% reduce 0%/
>>
>>
>>
>>
>>                              *but then*
>>
>>
>>
>>
>>                              /Error running child
>>                              java.lang.__**IllegalStateException: run:
>>
>>         Caught an unrecoverable exception java.io.FileNotFoundException:
>>         File _bsp/_defaultZkManagerDir/job_**
>> __201309181636_0678/_zkServer
>>
>>         does not exist.
>>                              /
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>                              /.../
>>
>>
>>
>>
>>
>>
>>
>>                              /Caused by: java.lang.RuntimeException:
>>         java.io.FileNotFoundException: File
>>         _bsp/_defaultZkManagerDir/job_**__201309181636_0678/_zkServer
>> does
>>
>>         not exist./
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>                              2013/10/5 Silvio Di gregorio
>>         <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <silvio.digregorio@gmail.com>>
>>          <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>
>>         <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>
>> >>>
>>
>>                                  I ha ve build w/ hadoop_cdh4.1.2
>> parameter.
>>                                  Something is changed, monday i report the
>>                                  result. Now the farm is closed.
>>
>>                                  Il giorno 05/ott/2013 14:06, "Claudio
>>         Martella"
>>         <claudio.martella@gmail.com <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>         <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>>         <mailto:claudio.martella@**gmail.com <claudio.martella@gmail.com>>>>
>> ha scritto:
>>
>>                                      Oh, right, -vof is in trunk. Anyway
>>         it looks
>>                                      like you built giraph for the wrong
>>         profile.
>>                                      You mentioned you're running on
>>         2.0, but
>>                                      your giraph is built for 0.20.203
>>         <tel:0.20.203>
>>         <tel:0.20.203 <tel:0.20.203>>. try building with a profile
>>
>>                                      for your hadoop version.
>>
>>
>>                                      On Fri, Oct 4, 2013 at 2:35 PM,
>>         Silvio Di
>>                                      gregorio
>>         <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <silvio.digregorio@gmail.com>>
>>          <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>         <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>>>>
>> wrote:
>>
>>
>>
>>         org.apache.commons.cli.__**UnrecognizedOptionException:
>>
>>                                          Unrecognized option: -vof
>>
>>                                          in 1.0 version is
>>                                            -of,--outputFormat <arg>
>>                                          Vertex output format
>>                                            -op,--outputPath <arg>
>>                                          Vertex output path
>>
>>
>>
>>                                          2013/10/4 Claudio Martella
>>         <claudio.martella@gmail.com <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>         <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>>         <mailto:claudio.martella@**gmail.com <claudio.martella@gmail.com>
>> >>>
>>
>>                                              did you try the argument
>>         (-vof) i
>>                                              suggested?
>>
>>
>>                                              On Fri, Oct 4, 2013 at 2:13
>> PM,
>>                                              Silvio Di gregorio
>>         <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <silvio.digregorio@gmail.com>>
>>          <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>
>>         <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>>>>
>> wrote:
>>
>>
>>                                                  i've specified  -of
>>
>>         org.apache.giraph.io.formats._**_IdWithValueTextOutputFormat
>>
>>
>>                                                  but the same error was
>>         produced
>>
>>                                                  Exception in thread
>> "main"
>>
>>         java.lang.__**IncompatibleClassChangeError:
>>                                                  Found interface
>>
>>         org.apache.hadoop.mapreduce.__**JobContext,
>>
>>                                                  but class was expected
>>                                                           at
>>
>>         org.apache.giraph.bsp.__**BspOutputFormat.__**checkOutputSpecs(__
>> **BspOutputFormat.java:43)
>>                                                           at
>>
>>         org.apache.hadoop.mapred.__**JobClient$2.run(JobClient.__**
>> java:984)
>>                                                           at
>>
>>         org.apache.hadoop.mapred.__**JobClient$2.run(JobClient.__**
>> java:945)
>>                                                           at
>>
>>         java.security.__**AccessController.doPrivileged(**__Native
>>                                                  Method)
>>                                                           at
>>
>>         javax.security.auth.Subject.__**doAs(Subject.java:396)
>>                                                           at
>>
>>         org.apache.hadoop.security.__**UserGroupInformation.doAs(__**
>> UserGroupInformation.java:__**1408)
>>                                                           at
>>
>>         org.apache.hadoop.mapred.__**JobClient.submitJobInternal(__**
>> JobClient.java:945)
>>                                                           at
>>
>>         org.apache.hadoop.mapreduce.__**Job.submit(Job.java:566)
>>                                                           at
>>
>>         org.apache.hadoop.mapreduce.__**Job.waitForCompletion(Job.__**
>> java:596)
>>                                                           at
>>
>>         org.apache.giraph.job.__**GiraphJob.run(GiraphJob.java:_**_237)
>>                                                           at
>>
>>         org.apache.giraph.__**GiraphRunner.run(GiraphRunner.**__java:94)
>>                                                           at
>>
>>         org.apache.hadoop.util.__**ToolRunner.run(ToolRunner.__**java:70)
>>                                                           at
>>
>>         org.apache.hadoop.util.__**ToolRunner.run(ToolRunner.__**java:84)
>>                                                           at
>>
>>         org.apache.giraph.__**GiraphRunner.main(__**
>> GiraphRunner.java:124)
>>                                                           at
>>
>>         sun.reflect.__**NativeMethodAccessorImpl.__**invoke0(Native
>>                                                  Method)
>>                                                           at
>>
>>         sun.reflect.__**NativeMethodAccessorImpl.__**invoke(__**
>> NativeMethodAccessorImpl.java:**__39)
>>                                                           at
>>
>>         sun.reflect.__**DelegatingMethodAccessorImpl._**_invoke(__**
>> DelegatingMethodAccessorImpl._**_java:25)
>>                                                           at
>>
>>         java.lang.reflect.Method.__**invoke(Method.java:597)
>>                                                           at
>>
>>         org.apache.hadoop.util.RunJar.**__main(RunJar.java:208)
>>
>>
>>
>>
>>                                                  2013/10/4 Claudio
>> Martella
>>         <claudio.martella@gmail.com <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>         <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>>         <mailto:claudio.martella@**gmail.com <claudio.martella@gmail.com>
>> >>>
>>
>>                                                      Hi,
>>
>>                                                      you need to specify
>> the
>>                                                      vertex outputformat
>>         class
>>                                                      (-vof option), e.g.
>>
>>         org.apache.giraph.io.formats._**_IdWithValueTextOutputFormat.
>>
>>
>>
>>                                                      On Fri, Oct 4, 2013
>>         at 1:06
>>                                                      PM, Silvio Di
>> gregorio
>>         <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <silvio.digregorio@gmail.com>>
>>          <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>
>>         <mailto:silvio.digregorio@**gmail.com<silvio.digregorio@gmail.com>
>> >>>
>>                                                      wrote:
>>
>>
>>                                                          Hi,
>>
>>                                                          I hope I have
>>         sent to
>>                                                          the right
>> address.
>>
>>                                                          i have a graph
>>         (directed
>>                                                          and unweighted)
>>         stored
>>                                                          in hdfs like a
>>         adjacency
>>                                                          list (140Milions
>> of
>>                                                          edges 6Milions
>>         of vertex)
>>
>>
>>  node<tab>neighbors
>>
>>                                                          23   2   1    3
>>             4    3
>>
>>         1 999 99923 <tel:1%20999%2099923>
>>         <tel:1%20999%2099923>
>>                                                          909 .......
>>
>>                                                          ......
>>
>>                                                          hadoop version
>>         Hadoop
>>                                                          2.0.0-cdh4.3.0
>>         - java 1.6
>>
>>
>>                                                          I have executed
>> the
>>                                                          giraph-1.0
>>         connected
>>                                                          components
>>         example, in
>>                                                          this fashion
>>
>>          >  hadoop jar
>>
>>         /usr/local/giraph/giraph-__**examples/target/giraph-__**
>> examples-1.0.0-for-hadoop-0.20.203.0
>>         <tel:0.20.203.0>
>>         <tel:0.20.203.0 <tel:0.20.203.0>>-jar-with-dependencies.**__jar
>>
>>         org.apache.giraph.GiraphRunner
>>
>>         org.apache.giraph.examples.__**ConnectedComponentsVertex
>>                                                              -vif
>>
>>         org.apache.giraph.io.formats._**_IntIntNullTextInputFormat
>>
>>                                                              -vip
>>
>>         /user/hdfs/lista_adj_txt
>>                                                              -op
>>         connectedgiraph
>>                                                              --workers 4
>>
>>                                                          and then fail
>> with:
>>
>>                                                          13/10/04
>>         09:28:29 INFO
>>
>>         utils.ConfigurationUtils: No
>>                                                          edge input format
>>                                                          specified.
>>         Ensure your
>>                                                          InputFormat
>>         does not
>>                                                          require one.
>>
>>                                                          13/10/04
>>         09:28:29 INFO
>>
>>         utils.ConfigurationUtils: No
>>                                                          output format
>>         specified.
>>                                                          Ensure your
>>         OutputFormat
>>                                                          does not
>>         require one.
>>
>>                                                          13/10/04
>>         09:28:30 INFO
>>                                                          job.GiraphJob:
>> run:
>>                                                          Since
>>         checkpointing is
>>                                                          disabled
>>         (default), do
>>                                                          not allow any
>> task
>>                                                          retries (setting
>>
>>         mapred.map.max.attempts
>>                                                          = 0, old value =
>> 4)
>>
>>                                                          13/10/04
>>         09:28:31 WARN
>>
>>         mapred.JobClient: Use
>>
>>         GenericOptionsParser for
>>                                                          parsing the
>>         arguments.
>>                                                          Applications
>> should
>>                                                          implement Tool
>>         for the same.
>>
>>                                                          13/10/04
>>         09:28:31 INFO
>>                                                          mapred.JobClient:
>>                                                          Cleaning up the
>>         staging
>>                                                          area
>>
>>         hdfs://srv-bigdata-dev-01.int.**__sose.it:8020/user/hdfs/.__**
>> staging/job_201309181636_0535<http://sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>         <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>> >
>>         <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__**
>> staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>
>>         <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>> >**>
>>
>>                                                          Exception in
>> thread
>>         "main"
>>
>>         java.lang.__**IncompatibleClassChangeError:
>>                                                          Found interface
>>
>>         org.apache.hadoop.mapreduce.__**JobContext,
>>
>>                                                          but class was
>>         expected
>>
>>                                                                   at
>>
>>         org.apache.giraph.bsp.__**BspOutputFormat.__**checkOutputSpecs(__
>> **BspOutputFormat.java:43)
>>
>>
>>                                                          ......
>>
>>                                                          Thanks in advance
>>
>>
>>
>>
>>
>>                                                      --
>>                                                          Claudio Martella
>>         claudio.martella@gmail.com <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>         <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>>         <mailto:claudio.martella@**gmail.com <claudio.martella@gmail.com>
>> >>
>>
>>
>>
>>
>>
>>
>>                                              --
>>                                                  Claudio Martella
>>         claudio.martella@gmail.com <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>         <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>>         <mailto:claudio.martella@**gmail.com <claudio.martella@gmail.com>
>> >>
>>
>>
>>
>>
>>
>>                                      --
>>                                          Claudio Martella
>>         claudio.martella@gmail.com <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>         <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>>         <mailto:claudio.martella@**gmail.com <claudio.martella@gmail.com>
>> >>
>>
>>
>>
>>
>>
>>                          --
>>                              Claudio Martella
>>         claudio.martella@gmail.com <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>         <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>>         <mailto:claudio.martella@**gmail.com <claudio.martella@gmail.com>
>> >>
>>
>>
>>
>>
>>
>>                  --
>>                      Claudio Martella
>>         claudio.martella@gmail.com <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>         <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>>         <mailto:claudio.martella@**gmail.com <claudio.martella@gmail.com>
>> >>
>>
>>
>>
>>
>>
>>         --
>>              Claudio Martella
>>         claudio.martella@gmail.com <mailto:claudio.martella@**gmail.com<claudio.martella@gmail.com>
>> >
>>         <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>>         <mailto:claudio.martella@**gmail.com <claudio.martella@gmail.com>
>> >>
>>
>>
>>     --
>>     Matthew Laird
>>     Lead Software Developer, Bioinformatics
>>     Brinkman Laboratory
>>     Simon Fraser University, Burnaby, BC, Canada
>>
>>
> --
> Matthew Laird
> Lead Software Developer, Bioinformatics
> Brinkman Laboratory
> Simon Fraser University, Burnaby, BC, Canada
>



-- 
   Claudio Martella
   claudio.martella@gmail.com

Mime
View raw message