giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jyoti Yadav <rao.jyoti26ya...@gmail.com>
Subject Running out of space..
Date Sun, 02 Feb 2014 16:25:42 GMT
Hi folks..

While executing one algorithm with enabling checkpointing feature on,system
is running out of memory. I gave following command


hadoop jar
$GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner -Dmapred.job.map.memory.mb=1500
-Dgiraph.useSuperstepCounters=false -Dgiraph.useOutOfCoreMessages=true
-Dgiraph.checkpointFrequency=1
org.apache.giraph.examples.MyShortestDistance -vif
org.apache.giraph.examples.io.formats.MyShortestDistanceVertexInputFormat
-vip /user/hduser/big_input/clustering -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/hduser/sp_output537/sd_output -w 1 -mc
org.apache.giraph.examples.MyShortestDistance\$MyMasterCompute


By executing this command   /app/hadoop/tmp/dfs/name/data/current directory
is consuming lots of space,approx  90GB(or may be more than this)
This job execution is involving thousands of supersteps..If i do not enable
checkpoint feature  on then algorithm fails due to worker failure error
after 19000 around supersteps.

Pls suggest me how to deal with this issue??

Thanks
Jyoti

Mime
View raw message