giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yi Lu <luyi0...@gmail.com>
Subject How to specify parameters in order to run giraph job in parallel
Date Thu, 17 Oct 2013 15:04:22 GMT
Hi,

I have a computer cluster consisting of 15 slave machines and 1 master
machine.

On each slave machine, there are two Xeon E5-2620 CPUs. With the help of
HT, there are 24 threads.

I am wondering how to specify parameters in order to run giraph job in
parallel on my cluster.

I am using the following parameters to run a pagerank algorithm.

hadoop jar ~/giraph-examples.jar org.apache.giraph.GiraphRunner
SimplePageRank -vif PageRankInputFormat -vip /input -vof
PageRankOutputFormat -op /pagerank -w 1 -mc
SimplePageRank\$SimplePageRankMasterCompute -wc
SimplePageRank\$SimplePageRankWorkerContext

In particular,

1)I know I can use “-w” to specify the number of workers. In my opinion,
the number of workers equals to the number of mappers in hadoop except
zookeeper. Therefore, in my case(15 slave machine), which number should be
chosen? Is 15 a good choice? Since, I find if I input a large number, e.g.
100, the mappers will hang.

2)I know I can use “-Dgiraph.numComputeThreads=1” to specify vertex
computing thread number. However, if I specify it to 10, the total runtime
is much longer than default. I think the default is 1, which is found in
the source code. I wonder if I want to use this parameter, which number
should be chosen.

3)When the giraph job is running, I use “top” command to monitor my cpu
usage on slave machines. I find that the java process can use 200%-300% cpu
resource. However, if I change the number of vertex computing threads to
10, the java process can use 800% cpu resource. I think it is not a linear
relation and I want to know why.


Thanks for your help.

Best,

-Yi

Mime
View raw message