giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Waagen <awaa...@gmail.com>
Subject Re: Shortest paths example in giraf 1.1.0.
Date Wed, 31 Jul 2013 18:34:52 GMT
I think the problem may have something to do with the fact that we are
using cloudera. Is there anyone who has experience with cloudera who may
have some idea how to fix this?


On Wed, Jul 31, 2013 at 6:59 AM, Alex Waagen <awaagen@gmail.com> wrote:

> I tried running both the shortest paths and pagerank examples, but I am
> still getting the same errors.
>
>
> On Tue, Jul 30, 2013 at 12:49 PM, Ryan Compton <compton.ryan@gmail.com>wrote:
>
>> Hey Alex,
>>
>> Our cluster uses Cloudera for Hadoop admin stuff. To the best of my
>> knowledge, we need to do our configs through the Cloudera GUI (which
>> is a royal pain).
>>
>> The "mapred.tasktracker.map.tasks.maximum" should be at 16 everywhere
>> (save for a few of the higher nodes which already had it at 32).
>> However, I can't figure how to get at "mapred.map.tasks" from the
>> Cloudera GUI. Can you give it another try?
>>
>> -Ryan
>>
>> On Tue, Jul 30, 2013 at 8:58 AM, Kyle Orlando <kyle.r.orlando@gmail.com>
>> wrote:
>> > Ah, that might be your problem.
>> >
>> > Try adding this between the <configuration> and </configuration>:
>> >
>> > <property>
>> >   <name>mapred.tasktracker.map.tasks.maximum</name>
>> >   <value>4</value>
>> > </property>
>> >
>> > <property>
>> >   <name>mapred.map.tasks</name>
>> >   <value>4</value>
>> > </property>
>> >
>> > See if it works, that's really the only thing I can think of.
>> >
>> > By default, the max number of map tasks and reduce tasksfor Hadoop is
>> > 2.  This changes the max number of map tasks to 4, and "hints" to
>> > Hadoop (whatever that means) that it should utilize 4 map tasks.  I
>> > believe that Giraph workers hijack map tasks, so reduce tasks are
>> > unneeded, but someone who is more familiar with Giraph will have to
>> > tell you more.
>> >
>> > On Tue, Jul 30, 2013 at 11:38 AM, Alex Waagen <awaagen@gmail.com>
>> wrote:
>> >> Here is the file. It is almost empty.
>> >>
>> >> <?xml version="1.0"?>
>> >> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> >>
>> >> <!-- Put site-specific property overrides in this file. -->
>> >>
>> >> <configuration>
>> >>
>> >> </configuration>
>> >>
>> >>
>> >>
>> >> On Tue, Jul 30, 2013 at 8:33 AM, Kyle Orlando <
>> kyle.r.orlando@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hmmm, could you post the contents of your mapred-site.xml in
>> >>> $HADOOP_HOME/conf?  You may need to increase the number of map tasks.
>> >>>
>> >>> On Tue, Jul 30, 2013 at 11:02 AM, Alex Waagen <awaagen@gmail.com>
>> wrote:
>> >>> > I am having some trouble getting these examples running. I’m
using
>> >>> > giraph
>> >>> > version 1.1.0 and hadoop 0.20.2. I am using the following json
file
>> as
>> >>> > input:
>> >>> >
>> >>> > [0,0,[[1,1],[3,3]]]
>> >>> > [1,0,[[0,1],[2,2],[3,1]]]
>> >>> > [2,0,[[1,2],[4,4]]]
>> >>> > [3,0,[[0,3],[1,1],[4,4]]]
>> >>> > [4,0,[[3,4],[2,4]]]
>> >>> >
>> >>> > The command I use is:
>> >>> >
>> >>> > hadoop jar
>> >>> >
>> >>> >
>> /path-to-giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>> >>> > org.apache.giraph.GiraphRunner
>> >>> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> >>> >
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> >>> > -vip
>> >>> > /path-to-input/input_file.json -of
>> >>> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> >>> > /path-to-output/outShortest -w 1
>> >>> >
>> >>> >
>> >>> > I see the following output.
>> >>> >
>> >>> > 13/07/29 14:36:06 INFO utils.ConfigurationUtils: No edge input
>> format
>> >>> > specified. Ensure your InputFormat does not require one.
>> >>> > 13/07/29 14:36:06 INFO job.GiraphJob: run: Since checkpointing
is
>> >>> > disabled
>> >>> > (default), do not allow any task retries (setting
>> >>> > mapred.map.max.attempts =
>> >>> > 0, old value = 4)
>> >>> > 13/07/29 14:36:20 INFO mapred.JobClient: Running job:
>> >>> > job_201307232135_0588
>> >>> > 13/07/29 14:36:21 INFO mapred.JobClient: map 0% reduce 0%
>> >>> > 13/07/29 14:36:52 INFO mapred.JobClient: map 50% reduce 0%
>> >>> > 13/07/29 14:47:24 INFO mapred.JobClient: map 0% reduce 0%
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Job complete:
>> >>> > job_201307232135_0588
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Counters: 6
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Job Counters
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=670508
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all
>> reduces
>> >>> > waiting after reserving slots (ms)=0
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Total time spent by all
>> maps
>> >>> > waiting after reserving slots (ms)=0
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Launched map tasks=2
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>> >>> > 13/07/29 14:47:39 INFO mapred.JobClient: Failed map tasks=1
>> >>> >
>> >>> >
>> >>> > When I check the job tracker, I see that two map jobs were killed,
>> with
>> >>> > the
>> >>> > following errors:
>> >>> >
>> >>> > java.lang.Throwable: Child Error
>> >>> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
>> >>> > Caused by: java.io.IOException: Task process exit with nonzero
>> status of
>> >>> > 1.
>> >>> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>> >>> >
>> >>> > java.lang.IllegalStateException: run: Caught an unrecoverable
>> exception
>> >>> > exists: Failed to check
>> >>> >
>> >>> >
>> /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
>> >>> > after 3 tries!
>> >>> > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
>> >>> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>> >>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>> >>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >>> > at java.security.AccessController.doPrivileged(Native Method)
>> >>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> >>> > at
>> >>> >
>> >>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
>> >>> > at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >>> > Caused by: java.lang.IllegalStateException: exists: Failed to check
>> >>> >
>> >>> >
>> /_hadoopBsp/job_201307232135_0588/_applicationAttemptsDir/0/_superstepDir/-1/_addressesAndPartitions
>> >>> > after 3 tries!
>> >>> > at org.apache.giraph.zk.ZooKeeperExt.exists(ZooKeeperExt.java:369)
>> >>> > at org.apache.giraph.worker.BspServiceWorker.s
>> >>> > Task attempt_201307232135_0588_m_000001_0 failed to report status
>> for
>> >>> > 600
>> >>> > seconds. Killing!
>> >>> >
>> >>> > Any idea what the problem is?
>> >>> > Thanks in advance.
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Kyle Orlando
>> >>> Computer Engineering Major
>> >>> University of Maryland
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Kyle Orlando
>> > Computer Engineering Major
>> > University of Maryland
>>
>
>

Mime
View raw message