giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: connected components example in giraph 1.0
Date Mon, 07 Oct 2013 18:01:09 GMT
try to increase the mappers' heap space through mapred.child.java.opts.


On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio <
silvio.digregorio@gmail.com> wrote:

> I think now my issue is another, thanks to all.
>
>
>
> java.lang.IllegalStateException: run: Caught an unrecoverable exception waitFor: ExecutionException
occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@73a01e8a
> 	at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
>
>
> 	......
> Caused by: java.lang.IllegalStateException: waitFor: ExecutionException occurred while
waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@73a01e8a
> 	at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:151)
>
>
> 	... 7 more
> Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: *Java
heap space*
> 	at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:143)
>
>
> 	... 13 more
>
>
>
> 2013/10/7 Claudio Martella <claudio.martella@gmail.com>
>
>> OK, thanks.
>>
>> I really have to push that patch in.
>>
>>
>> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di gregorio <
>> silvio.digregorio@gmail.com> wrote:
>>
>>> yes i do,
>>> i have seen this in your post in:
>>> http://www.mail-archive.com/user@giraph.apache.org/msg00957.html
>>>
>>> excuse me if  i had checked in the mail-achive first I would have
>>> avoided the last post.
>>>
>>> Now zk issue are resolved.
>>>
>>>
>>> 2013/10/7 Claudio Martella <claudio.martella@gmail.com>
>>>
>>>> Can you try applying this one first?
>>>>
>>>> http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff
>>>>
>>>>
>>>> On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di gregorio <
>>>> silvio.digregorio@gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *As i said i have builded*
>>>>>
>>>>>
>>>>>
>>>>> *giraph-examples-1.0.0-for-hadoop-2.0.0-cdh4.1.2-jar-with-dependencies.jar*
>>>>>
>>>>>
>>>>>
>>>>> *for cdh4, successfully. The job start to monitoring the success rate:*
>>>>>
>>>>>
>>>>>
>>>>> *13/10/07 08:28:45 INFO mapred.JobClient:  map 0% reduce 0%*
>>>>>
>>>>>
>>>>>
>>>>> *but then*
>>>>>
>>>>>
>>>>>
>>>>> *Error running child
>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable exception
java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer
does not exist.
>>>>> *
>>>>>
>>>>>
>>>>>
>>>>> *...*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist.*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/10/5 Silvio Di gregorio <silvio.digregorio@gmail.com>
>>>>>
>>>>>> I ha ve build w/ hadoop_cdh4.1.2 parameter.
>>>>>> Something is changed, monday i report the result. Now the farm is
>>>>>> closed.
>>>>>> Il giorno 05/ott/2013 14:06, "Claudio Martella" <
>>>>>> claudio.martella@gmail.com> ha scritto:
>>>>>>
>>>>>>  Oh, right, -vof is in trunk. Anyway it looks like you built giraph
>>>>>>> for the wrong profile. You mentioned you're running on 2.0, but
your giraph
>>>>>>> is built for 0.20.203. try building with a profile for your hadoop
>>>>>>> version.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di gregorio <
>>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized
>>>>>>>> option: -vof
>>>>>>>>
>>>>>>>> in 1.0 version is
>>>>>>>>  -of,--outputFormat <arg>               Vertex output
format
>>>>>>>>  -op,--outputPath <arg>                 Vertex output
path
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/10/4 Claudio Martella <claudio.martella@gmail.com>
>>>>>>>>
>>>>>>>>> did you try the argument (-vof) i suggested?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Oct 4, 2013 at 2:13 PM, Silvio Di gregorio <
>>>>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> i've specified  -of
>>>>>>>>>>  org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>>>>>>>>>>
>>>>>>>>>> but the same error was produced
>>>>>>>>>>
>>>>>>>>>> Exception in thread "main"
>>>>>>>>>> java.lang.IncompatibleClassChangeError: Found interface
>>>>>>>>>> org.apache.hadoop.mapreduce.JobContext, but class
was expected
>>>>>>>>>>         at
>>>>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>>>>>>>>>>         at java.security.AccessController.doPrivileged(Native
>>>>>>>>>> Method)
>>>>>>>>>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>>>>>>>>>>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>>>>>>>>>>         at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>>>> Method)
>>>>>>>>>>         at
>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>>>>>         at
>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/10/4 Claudio Martella <claudio.martella@gmail.com>
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> you need to specify the vertex outputformat class
(-vof option),
>>>>>>>>>>> e.g. org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Oct 4, 2013 at 1:06 PM, Silvio Di gregorio
<
>>>>>>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I hope I have sent to the right address.
>>>>>>>>>>>>
>>>>>>>>>>>> i have a graph (directed and unweighted)
stored in hdfs like a
>>>>>>>>>>>> adjacency list (140Milions of edges 6Milions
of vertex)
>>>>>>>>>>>>
>>>>>>>>>>>> node<tab>neighbors
>>>>>>>>>>>>
>>>>>>>>>>>> 23   2   1    3    4    3
>>>>>>>>>>>>
>>>>>>>>>>>> 1 999 99923 909 .......
>>>>>>>>>>>>
>>>>>>>>>>>> ......
>>>>>>>>>>>>
>>>>>>>>>>>> hadoop version Hadoop 2.0.0-cdh4.3.0 - java
1.6
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have executed the giraph-1.0 connected
components example, in
>>>>>>>>>>>> this fashion
>>>>>>>>>>>>
>>>>>>>>>>>>> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
>>>>>>>>>>>>> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>>>>>>>>>>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
>>>>>>>>>>>>> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat
>>>>>>>>>>>>> -vip /user/hdfs/lista_adj_txt -op connectedgiraph
--workers 4
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> and then fail with:
>>>>>>>>>>>>
>>>>>>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils:
No edge input
>>>>>>>>>>>> format specified. Ensure your InputFormat
does not require one.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils:
No output
>>>>>>>>>>>> format specified. Ensure your OutputFormat
does not require one.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 13/10/04 09:28:30 INFO job.GiraphJob: run:
Since checkpointing
>>>>>>>>>>>> is disabled (default), do not allow any task
retries (setting
>>>>>>>>>>>> mapred.map.max.attempts = 0, old value =
4)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 13/10/04 09:28:31 WARN mapred.JobClient:
Use
>>>>>>>>>>>> GenericOptionsParser for parsing the arguments.
Applications should
>>>>>>>>>>>> implement Tool for the same.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 13/10/04 09:28:31 INFO mapred.JobClient:
Cleaning up the
>>>>>>>>>>>> staging area hdfs://
>>>>>>>>>>>> srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
>>>>>>>>>>>>
>>>>>>>>>>>> Exception in thread "main"
>>>>>>>>>>>> java.lang.IncompatibleClassChangeError: Found
interface
>>>>>>>>>>>> org.apache.hadoop.mapreduce.JobContext, but
class was expected
>>>>>>>>>>>>
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>>>>>>>
>>>>>>>>>>>>         ......
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>    Claudio Martella
>>>>>>>>>>>    claudio.martella@gmail.com
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>    Claudio Martella
>>>>>>>>>    claudio.martella@gmail.com
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>    Claudio Martella
>>>>>>>    claudio.martella@gmail.com
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>    Claudio Martella
>>>>    claudio.martella@gmail.com
>>>>
>>>
>>>
>>
>>
>> --
>>    Claudio Martella
>>    claudio.martella@gmail.com
>>
>
>


-- 
   Claudio Martella
   claudio.martella@gmail.com

Mime
View raw message