giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Schelter <...@apache.org>
Subject Re: Trying to run the Connected Components example.
Date Tue, 07 Aug 2012 07:07:42 GMT
Can you check what the mappers where doing via the web interface of
Hadoop? Can you run 4 mappers at once?



On 07.08.2012 01:46, Vishal Patel wrote:
> I'm seeing a strange behavior that I can't explain.
> 
> 
> hadoop jar giraph-0.1-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.ConnectedComponentsVertex --inputFormat
> org.apache.giraph.examples.IntIntNullIntTextInputFormat --inputPath
> /user/vpatel/graph_in/elist.txt --outputFormat
> org.apache.giraph.examples.VertexWithComponentTextOutputFormat --outputPath
> hdfs:///user/vpatel/giraph_out/1 --workers 4 --combiner
> org.apache.giraph.examples.MinimumIntCombiner
> Warning: $HADOOP_HOME is deprecated.
> 
> 12/08/06 16:16:40 INFO mapred.JobClient: Running job: job_201208031459_0591
> 12/08/06 16:16:41 INFO mapred.JobClient:  map 0% reduce 0%
> 12/08/06 16:16:59 INFO mapred.JobClient:  map 20% reduce 0%
> 12/08/06 16:17:05 INFO mapred.JobClient:  map 40% reduce 0%
> 12/08/06 16:17:08 INFO mapred.JobClient:  map 100% reduce 0%
> 12/08/06 16:17:11 INFO mapred.JobClient:  map 80% reduce 0%
> 12/08/06 16:17:16 INFO mapred.JobClient: Task Id :
> attempt_201208031459_0591_m_000000_0, Status : FAILED
> *java.lang.Throwable: Child Error
>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
> *
> 
> I either get the above error, which I can avoid if I decrease my number of
> workers (based on previous post on the mailing list).
> 
> However when I do specify lesser workers (say 2) or sometimes I don't get
> the above error: the result is missing for one part in the hdfs.
> i.e. when I did workers=2, I got two parts. One of them had 5,000 out of
> the 10k nodes and other part was blank. This happens when I did workers=4,5
> etc as well.
> 
> There are no errors in the log.
> 
> Just to be clear, the input format is adjacency list,
> i.e if a -> b, a ->c and b -> d then
> a b c
> b a d
> c a
> d b
> 
> Since the graph is undirected. Any idea what could be wrong?
> 
> Here is the log when I do workers=1
> 
> Finally loaded a total of *(v=10000, e=19996)*
> 2012-08-06 16:39:13,902 INFO org.apache.giraph.graph.BspService:
> process: inputSplitsAllDoneChanged (all vertices sent from input
> splits)
> 2012-08-06 16:39:13,904 INFO
> org.apache.giraph.comm.BasicRPCCommunications: flush: starting for
> superstep -1 totalMem = 191.6875M, maxMem = 191.6875M, freeMem =
> 164.6044M
> 2012-08-06 16:39:13,906 INFO
> org.apache.giraph.comm.BasicRPCCommunications: flush: ended for
> superstep -1 totalMem = 191.6875M, maxMem = 191.6875M, freeMem =
> 164.60431M
> 2012-08-06 16:39:13,906 INFO org.apache.giraph.graph.BspServiceWorker:
> finishSuperstep: Superstep -1 totalMem = 191.6875M, maxMem =
> 191.6875M, freeMem = 164.60431M
> 2012-08-06 16:39:13,922 INFO org.apache.giraph.graph.BspService:
> process: superstepFinished signaled
> 2012-08-06 16:39:13,924 INFO org.apache.giraph.graph.BspServiceWorker:
> finishSuperstep: Completed superstep -1 with global stats
> (vtx=0,finVtx=0,edges=0,msgCount=0)
> 2012-08-06 16:39:13,924 INFO org.apache.giraph.graph.GraphMapper:
> cleanup: Starting for WORKER_ONLY
> 2012-08-06 16:39:13,925 INFO org.apache.giraph.graph.BspServiceWorker:
> processEvent: Job state changed, checking to see if it needs to
> restart
> 2012-08-06 16:39:13,926 INFO org.apache.giraph.graph.BspService:
> getJobState: Job state already exists
> (/_hadoopBsp/job_201208031459_0621/_masterJobState)
> 2012-08-06 16:39:13,929 INFO org.apache.giraph.graph.BspServiceWorker:
> cleanup: Notifying master its okay to cleanup with
> /_hadoopBsp/job_201208031459_0621/_cleanedUpDir/1_worker
> 2012-08-06 16:39:13,930 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x138fe1c4699003d closed
> 2012-08-06 16:39:13,930 INFO
> org.apache.giraph.comm.BasicRPCCommunications: close: shutting down
> RPC server
> 2012-08-06 16:39:13,930 INFO org.apache.zookeeper.ClientCnxn:
> EventThread shut down
> 2012-08-06 16:39:13,930 INFO org.apache.hadoop.ipc.Server: Stopping
> server on 30003
> 2012-08-06 16:39:13,930 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 30003: exiting
> 2012-08-06 16:39:13,930 INFO org.apache.hadoop.ipc.Server: Stopping
> IPC Server listener on 30003
> 2012-08-06 16:39:13,930 INFO
> org.apache.hadoop.ipc.metrics.RpcInstrumentation: shut down
> 2012-08-06 16:39:13,931 INFO org.apache.hadoop.ipc.Server: Stopping
> IPC Server Responder
> 2012-08-06 16:39:13,931 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 30003: exiting
> 2012-08-06 16:39:13,931 INFO org.apache.giraph.zk.ZooKeeperManager:
> createZooKeeperClosedStamp: Creating my filestamp
> _bsp/_defaultZkManagerDir/job_201208031459_0621/_task/1.COMPUTATION_DONE
> 2012-08-06 16:39:13,934 INFO org.apache.hadoop.mapred.Task:
> Task:attempt_201208031459_0621_m_000001_0 is done. And is in the
> process of commiting
> 2012-08-06 16:39:15,026 INFO org.apache.hadoop.mapred.Task: Task
> attempt_201208031459_0621_m_000001_0 is allowed to commit now
> 2012-08-06 16:39:15,036 INFO
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved
> output of task 'attempt_201208031459_0621_m_000001_0' to
> hdfs:/user/vpatel/giraph_out/one
> 2012-08-06 16:39:16,068 INFO org.apache.hadoop.mapred.Task: Task
> 'attempt_201208031459_0621_m_000001_0' done.
> 2012-08-06 16:39:16,087 INFO
> org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs'
> truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2012-08-06 16:39:16,117 INFO org.apache.hadoop.io.nativeio.NativeIO:
> Initialized cache for UID to User mapping with a cache timeout of
> 14400 seconds.
> 2012-08-06 16:39:16,118 INFO org.apache.hadoop.io.nativeio.NativeIO:
> Got UserName vpatel for UID 10020 from the native implementation
> 
> 
> 
> 
> On Mon, Aug 6, 2012 at 3:05 PM, Sebastian Schelter <ssc@apache.org> wrote:
> 
>> The job expects the input data in adjacency list format, each line
>> should look like:
>>
>> vertex neighbor1 neighbor2 ....
>>
>> --sebastian
>>
>>
>> On 07.08.2012 00:02, Vishal Patel wrote:
>>> Thanks Sebastian, it runs fine now. However, the output comes back as
>>>
>>> 0       0
>>> 1       1
>>> 2       2
>>> 3       3
>>> 4       4
>>> 5       5
>>> 6       6
>>> ..
>>>
>>> I have an unsorted edge file with just int values.
>>> http://www.ics.uci.edu/~vishalrp/public/testg.txt
>>>
>>> My test graph (head below) has 10,000 nodes ( from 0 to 9999) and 9998
>>> edges. There are 4 connected components in the graph.
>>>
>>> 0       5800
>>> 0       5981
>>> 1       1239
>>> 1       2989
>>> 1       3961
>>> 2       5417
>>> 2       7350
>>>
>>> What am I doing wrong? Also, in general does the graph have to have int
>>> values for nodes? Or can I have strings?
>>>
>>> Appreciate your help!
>>>
>>> Vishal
>>>
>>>
>>>
>>>
>>> On Mon, Aug 6, 2012 at 2:22 PM, Sebastian Schelter <ssc@apache.org>
>> wrote:
>>>
>>>> You cannot run the vertex class directly. Instead you can use
>>>> GiraphRunner, e.g.
>>>>
>>>> hadoop jar giraph-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner
>>>> org.apache.giraph.examples.ConnectedComponentsVertex --inputFormat
>>>> org.apache.giraph.examples.IntIntNullIntTextInputFormat --inputPath
>>>> hdfs:///path/to/input --outputFormat
>>>> org.apache.giraph.examples.VertexWithComponentTextOutputFormat
>>>> --outputPath hdfs:///path/to/output --workers numWorkers --combiner
>>>> org.apache.giraph.examples.MinimumIntCombiner
>>>>
>>>> --sebastian
>>>>
>>>>
>>>> 2012/8/6 Vishal Patel <write2vishal@gmail.com>:
>>>>> Hi, I am trying to run the connected-components example. I have giraph
>>>>> installed, all the test pass on a 3 node cluster running hadoop-1.0.3/
>>>>>
>>>>> The main method is missing in the ConnectedComponentsVertex class
>>>>>
>>>>> cd target/classes
>>>>> hadoop jar ../giraph-0.1-jar-with-dependencies.jar
>>>>> org.apache.giraph.examples.ConnectedComponentsVertex
>>>>>
>>>>> Exception in thread "main" java.lang.NoSuchMethodException:
>>>>>
>>>>
>> org.apache.giraph.examples.ConnectedComponentsVertex.main([Ljava.lang.String;)
>>>>>         at java.lang.Class.getMethod(Class.java:1622)
>>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:150)
>>>>>
>>>>> Can someone please help me with running this example?
>>>>>
>>>>> Vishal
>>>>>
>>>>
>>>
>>
>>
> 


Mime
View raw message