giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Sirotin <sirot...@web.de>
Subject Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
Date Wed, 20 Aug 2014 16:02:17 GMT
Hi Vikalp,

finally I got to run Giraph successfully :-)

Following steps:
#Preparation

  *

    su

  *

    rm -rf /usr/local/giraph; mkdir /usr/local/giraph

  *

    chown hadoop:hadoop /usr/local/giraph

  *

    mkdir ~/setup; cd ~/setup

#Sourcecode (Version 1.1.0)

  *

    git clone https://git-wip-us.apache.org/repos/asf/giraph.git

  *

    mv giraph giraph_src; cd giraph_src

  *

    mvn clean

  *

    mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package

#Installation (little ugly)

  *

    rm -rf myGiraph; mkdir -p myGiraph/lib; mkdir myGiraph/source

  *

    find . -name \*.jar -exec cp {} myGiraph/lib \;

  *

    cp -r conf myGiraph

  *

    cp -r bin myGiraph

  *

    cp CH* CO* RE* LI* NO* myGiraph

  *

    cd myGiraph

  *

    mv lib/*source*.jar source

  *

    mv lib/giraph*.jar .

  *

    rename 's/-SNAPSHOT-for//' *.jar

  *

    rm -rf ../myGiraph; cd ..; mv myGiraph ..

  *

    cd ../myGiraph; rm -rf /usr/local/giraph/*; cp -r * /usr/local/giraph

Now you have to put a variable into hadoop-env.sh (at the end is ok):

      o

        /MY_LIBS="$HADOOP_HOME/share/myLib/*.jar"/

      o

        /export HADOOP_TASKTRACKER_OPTS="-classpath ${MY_LIBS}
        ${HADOOP_TASKTRACKER_OPTS}"/

The folder myLib you have to create and put the Giraph-Jar:

  * mkdir /usr/local/hadoop/share/myLib

  *

    cp
    /usr/local/giraph/giraph-examples-1.1.0-hadoop-2.4.0-jar-with-dependencies.jar
    /usr/local/hadoop/share/myLib

#Configuration
Put
<property>
<name>mapred.map.tasks</name>
<value>4</value>
</property>

into mapred-site.xml

  *

    echo export PATH=\$PATH:/usr/local/giraph/bin >> ~/.bashrc

  *

    echo export GIRAPH_HOME=/usr/local/giraph >> ~/.bashrc

  *

    source ~/.bashrc

Now you can test:
#creating testdata

  *

    rm -rf ~/giraph; mkdir ~/giraph; cd ~/giraph

  * cat << EOF > tiny_graph.txt
  * [0,0,[[1,1],[3,3]]]
    [1,0,[[0,1],[2,2],[3,1]]]
    [2,0,[[1,2],[4,4]]]
    [3,0,[[0,3],[1,1],[4,4]]]
    [4,0,[[3,4],[2,4]]]
  * EOF

#Run example-Jar

  *

    jar=giraph-examples-1.1.0-hadoop-2.4.0-jar-with-dependencies.jar

  *

    runner=org.apache.giraph.GiraphRunner

  *

    computation=org.apache.giraph.examples.SimpleShortestPathsComputation

  *

    informat=org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat

  *

    outformat=org.apache.giraph.io.formats.IdWithValueTextOutputFormat

  *

    hdfs dfs -rm -r shortestpaths

  *

    yarn jar $GIRAPH_HOME/$jar $runner -Dgiraph.yarn.task.heap.mb=3000
    $computation -vif $informat -vip tiny_graph.txt -vof $outformat -op
    shortestpaths -w 1 -yj $jar

  *

    hdfs dfs -cat shortestpaths/*


Finished!! Hope it functions on your site.

Best regards,
Alexander




On 12.08.2014 15:52, Vikalp Handa wrote:
> Hi Alexander,
>
> I looked into my gam-stderr.log file and found *Error: Could not find 
> or load main class org.apache.giraph.yarn.GiraphApplicationMaster*
> As you have already mentioned in your reply about that for loop in 
> *hadoop-env.sh. *
> So can you please tell me what is *mylib *and***share/myLib/giraph** ? 
> *Because I am having only doc and hadoop directories 
> inside  $HADOOP_HOME/share/
>
> Also how to get *giraph-1.1.0-hadoop-2.4.0.jar 
> *and*giraph-examples-1.1.0-hadoop-*2.4.0.jar (without the 
> dependencies) as I am having only 
> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.4.0-jar-with-dependencies.jar 
> inside giraph-examples/target/munged/.
>
>
>
>


Mime
View raw message