giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jyotirmoy Sundi <sundi...@gmail.com>
Subject giraph with oozie, unable to start tasks
Date Wed, 02 Oct 2013 18:22:21 GMT
Hi,
    Giraph is running awesomely. But it cant start the tasks when run from
oozie.

I am able to run from the terminal using fat jar without any issues. I
placed the code and examples jar in the libpath. Now I am trying to run a
job through oozie. The GiraphRunner starts the oozie job, but fails to
start any tasks. When i see the TT machines, there are no logs, the
counters remain at 0 and all get killed after few seconds., its displays
like this:
task_201309260044_1170_m_000000<http://had24.rsk.admobius.com:50030/taskdetails.jsp?tipid=task_201309260044_1170_m_000000>
0.00%

2-Oct-2013 17:28:23


0<http://had24.rsk.admobius.com:50030/taskstats.jsp?tipid=task_201309260044_1170_m_000000>
task_201309260044_1170_m_000001<http://had24.rsk.admobius.com:50030/taskdetails.jsp?tipid=task_201309260044_1170_m_000001>
0.00%

2-Oct-2013 17:28:23


0<http://had24.rsk.admobius.com:50030/taskstats.jsp?tipid=task_201309260044_1170_m_000001>
task_201309260044_1170_m_000002<http://had24.rsk.admobius.com:50030/taskdetails.jsp?tipid=task_201309260044_1170_m_000002>
0.00%

2-Oct-2013 17:28:23


0<http://had24.rsk.admobius.com:50030/taskstats.jsp?tipid=task_201309260044_1170_m_000002>
<http://had24.rsk.admobius.com:50030/taskdetails.jsp?tipid=task_201309260044_1170_m_000003>
*
*
*The logs are here:*

2013-10-02 18:06:00,744 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
2013-10-02 18:06:00,929 INFO
org.apache.giraph.utils.ConfigurationUtils: No edge input format
specified. Ensure your InputFormat does not require one.
2013-10-02 18:06:00,990 INFO org.apache.giraph.job.GiraphJob: run:
Since checkpointing is disabled (default), do not allow any task
retries (setting mapred.map.max.attempts = 0, old value = 4)
2013-10-02 18:06:01,040 WARN org.apache.hadoop.mapred.JobClient: Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2013-10-02 18:06:01,232 INFO org.apache.hadoop.mapred.JobClient:
Running job: job_201309260044_1176
2013-10-02 18:06:02,238 INFO org.apache.hadoop.mapred.JobClient:  map
0% reduce 0%
2013-10-02 18:06:11,276 INFO org.apache.hadoop.mapred.JobClient: Job
complete: job_201309260044_1176
2013-10-02 18:06:11,279 INFO org.apache.hadoop.mapred.JobClient: Counters: 4
2013-10-02 18:06:11,282 INFO org.apache.hadoop.mapred.JobClient:   Job Counters
2013-10-02 18:06:11,284 INFO org.apache.hadoop.mapred.JobClient:
Total time spent by all maps in occupied slots (ms)=9390
2013-10-02 18:06:11,284 INFO org.apache.hadoop.mapred.JobClient:
Total time spent by all reduces in occupied slots (ms)=0
2013-10-02 18:06:11,284 INFO org.apache.hadoop.mapred.JobClient:
Total time spent by all maps waiting after reserving slots (ms)=0
2013-10-02 18:06:11,284 INFO org.apache.hadoop.mapred.JobClient:
Total time spent by all reduces waiting after reserving slots (ms)=0
2013-10-02 18:06:11,305 INFO org.apache.hadoop.mapred.Task:
Task:attempt_201309260044_1175_m_000000_0 is done. And is in the
process of commiting
2013-10-02 18:06:12,399 INFO org.apache.hadoop.mapred.Task: Task
attempt_201309260044_1175_m_000000_0 is allowed to commit now
2013-10-02 18:06:12,427 INFO
org.apache.hadoop.mapred.FileOutputCommitter: Saved output of task
'attempt_201309260044_1175_m_000000_0' to hdfs:
/user/sundi133/oozie-oozi/0000243-130928181649085-oozie-oozi-W/fingerprint-05-giraph--java/output
2013-10-02 18:06:12,431 INFO org.apache.hadoop.mapred.Task: Task
'attempt_201309260044_1175_m_000000_0' done.
2013


*
*
*Terminal command : success*
hadoop jar
/home/sundi133/giraph/giraph/giraph-examples/target/giraph-examples-1.0.0-for-hadoop-2.0.3-alpha-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner -Dgiraph.zKForceSync=true
-Dgiraph.zkJavaOpts="-Xmx2g -XX:ParallelGCThreads=4 -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=70 -XX:MaxGCPauseMillis=100"
-Dmapred.child.java.opts="-Xmx64g" -Dgiraph.zkSessionMsecTimeout=600000
-Dgiraph.useInputSplitLocality=false -Dgiraph.zkList=
mymac23.rsk.admobs.com:2181,mymac16.rsk.admobs.com:2181,mymac
22.rsk.admobs.com:2181 -Dgiraph.maxVerticesPerTransfer=100000
-Dgiraph.maxEdgesPerTransfer=200000 -Dgiraph.isStaticGraph=true
org.apache.giraph.examples.ConnectedComponentsVertexAdmobius -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormatAdmobius
-vip
/data/production/batched/fingerprint/fp_match_matrix/batch_id=2013090614/pa*
-op /data/production/batched/fingerprint/hashed_cc/batch_id=2013090639 -of
org.apache.giraph.examples.VertexWithDoubleValueDoubleEdgeTextOutputFormatAdmobius
-c org.apache.giraph.combiner.MinimumLongCombiner -w 115

*
*
*Oozie xml:*
*
 <action name="fingerprint-05-giraph">

                <java>
                        <job-tracker>${ createCluster == true ?
wf:actionData('start-new-cluster')['jobTracker'] : jobTracker }
                        </job-tracker>
                        <name-node>${ createCluster == true ?
wf:actionData('start-new-cluster')['nameNode'] : nameNode }
                        </name-node>

<job-xml>${nameNode}/user/${wf:user()}/hive/${env}/job.xml</job-xml>
                        <configuration>
                                <property>
                                        <name>mapred.job.queue.name</name>
                                        <value>${queueName}</value>
                                </property>
                        </configuration>

<main-class>org.apache.giraph.GiraphRunner</main-class>

<arg>org.apache.giraph.examples.ConnectedComponentsVertexAdmobius</arg>
                        <arg>-vif</arg>

<arg>org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormatAdmobius</arg>
                        <arg>-vip</arg>

<arg>${nameNode}/data/production/batched/fingerprint/fp_match_matrix/batch_id=${batchId}/pa*</arg>
                        <arg>-op</arg>

<arg>${nameNode}/data/production/batched/fingerprint/hashed_cc/batch_id=${batchId}</arg>
                        <arg>-of</arg>

<arg>org.apache.giraph.examples.VertexWithDoubleValueDoubleEdgeTextOutputFormatAdmobius</arg>
                        <arg>-c</arg>

<arg>org.apache.giraph.combiner.MinimumLongCombiner</arg>
                        <arg>-w</arg>
                        <arg>118</arg>
                        <file>giraph-site.xml</file>
                        <capture-output />
                </java>

                <ok to="fingerprint-update-connnectedcomponents" />
                <error to="fail" />
        </action>


Any pointers over it would be appreciated.
*
-- 
Best Regards,
Jyotirmoy Sundi
Data Engineer,
Admobius

San Francisco, CA 94158

Mime
View raw message