giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Reisman <apache.mail...@gmail.com>
Subject Re: giraph with oozie, unable to start tasks
Date Sun, 10 Nov 2013 20:24:30 GMT
You can add the key/values you are setting with -D in hadoop jar to your
giraph-site.xml and load those values in your job?

Unrelated, but it doesn't look like you need to set the ZK java opts in
your -D args since you also specify a ZK server list which tells Giraph to
assume the quorum is already up and running and to use those addresses to
connect to it.



On Wed, Oct 2, 2013 at 11:22 AM, Jyotirmoy Sundi <sundi133@gmail.com> wrote:

> Hi,
>     Giraph is running awesomely. But it cant start the tasks when run from
> oozie.
>
> I am able to run from the terminal using fat jar without any issues. I
> placed the code and examples jar in the libpath. Now I am trying to run a
> job through oozie. The GiraphRunner starts the oozie job, but fails to
> start any tasks. When i see the TT machines, there are no logs, the
> counters remain at 0 and all get killed after few seconds., its displays
> like this:
> task_201309260044_1170_m_000000<http://had24.rsk.admobius.com:50030/taskdetails.jsp?tipid=task_201309260044_1170_m_000000>
> 0.00%
>
> 2-Oct-2013 17:28:23
>
>
> 0<http://had24.rsk.admobius.com:50030/taskstats.jsp?tipid=task_201309260044_1170_m_000000>
> task_201309260044_1170_m_000001<http://had24.rsk.admobius.com:50030/taskdetails.jsp?tipid=task_201309260044_1170_m_000001>
> 0.00%
>
> 2-Oct-2013 17:28:23
>
>
> 0<http://had24.rsk.admobius.com:50030/taskstats.jsp?tipid=task_201309260044_1170_m_000001>
> task_201309260044_1170_m_000002<http://had24.rsk.admobius.com:50030/taskdetails.jsp?tipid=task_201309260044_1170_m_000002>
> 0.00%
>
> 2-Oct-2013 17:28:23
>
>
> 0<http://had24.rsk.admobius.com:50030/taskstats.jsp?tipid=task_201309260044_1170_m_000002>
> <http://had24.rsk.admobius.com:50030/taskdetails.jsp?tipid=task_201309260044_1170_m_000003>
>
> *The logs are here:*
>
> 2013-10-02 18:06:00,744 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0
> 2013-10-02 18:06:00,929 INFO org.apache.giraph.utils.ConfigurationUtils: No edge input
format specified. Ensure your InputFormat does not require one.
> 2013-10-02 18:06:00,990 INFO org.apache.giraph.job.GiraphJob: run: Since checkpointing
is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0,
old value = 4)
> 2013-10-02 18:06:01,040 WARN org.apache.hadoop.mapred.JobClient: Use GenericOptionsParser
for parsing the arguments. Applications should implement Tool for the same.
> 2013-10-02 18:06:01,232 INFO org.apache.hadoop.mapred.JobClient: Running job: job_201309260044_1176
> 2013-10-02 18:06:02,238 INFO org.apache.hadoop.mapred.JobClient:  map 0% reduce 0%
> 2013-10-02 18:06:11,276 INFO org.apache.hadoop.mapred.JobClient: Job complete: job_201309260044_1176
> 2013-10-02 18:06:11,279 INFO org.apache.hadoop.mapred.JobClient: Counters: 4
> 2013-10-02 18:06:11,282 INFO org.apache.hadoop.mapred.JobClient:   Job Counters
> 2013-10-02 18:06:11,284 INFO org.apache.hadoop.mapred.JobClient:     Total time spent
by all maps in occupied slots (ms)=9390
> 2013-10-02 18:06:11,284 INFO org.apache.hadoop.mapred.JobClient:     Total time spent
by all reduces in occupied slots (ms)=0
> 2013-10-02 18:06:11,284 INFO org.apache.hadoop.mapred.JobClient:     Total time spent
by all maps waiting after reserving slots (ms)=0
> 2013-10-02 18:06:11,284 INFO org.apache.hadoop.mapred.JobClient:     Total time spent
by all reduces waiting after reserving slots (ms)=0
> 2013-10-02 18:06:11,305 INFO org.apache.hadoop.mapred.Task: Task:attempt_201309260044_1175_m_000000_0
is done. And is in the process of commiting
> 2013-10-02 18:06:12,399 INFO org.apache.hadoop.mapred.Task: Task attempt_201309260044_1175_m_000000_0
is allowed to commit now
> 2013-10-02 18:06:12,427 INFO org.apache.hadoop.mapred.FileOutputCommitter: Saved output
of task 'attempt_201309260044_1175_m_000000_0' to hdfs: /user/sundi133/oozie-oozi/0000243-130928181649085-oozie-oozi-W/fingerprint-05-giraph--java/output
> 2013-10-02 18:06:12,431 INFO org.apache.hadoop.mapred.Task: Task 'attempt_201309260044_1175_m_000000_0'
done.
> 2013
>
>
>
> *Terminal command : success*
> hadoop jar
> /home/sundi133/giraph/giraph/giraph-examples/target/giraph-examples-1.0.0-for-hadoop-2.0.3-alpha-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner -Dgiraph.zKForceSync=true
> -Dgiraph.zkJavaOpts="-Xmx2g -XX:ParallelGCThreads=4 -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxGCPauseMillis=100"
> -Dmapred.child.java.opts="-Xmx64g" -Dgiraph.zkSessionMsecTimeout=600000
> -Dgiraph.useInputSplitLocality=false -Dgiraph.zkList=
> mymac23.rsk.admobs.com:2181,mymac16.rsk.admobs.com:2181,mymac
> 22.rsk.admobs.com:2181 -Dgiraph.maxVerticesPerTransfer=100000
> -Dgiraph.maxEdgesPerTransfer=200000 -Dgiraph.isStaticGraph=true
> org.apache.giraph.examples.ConnectedComponentsVertexAdmobius -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormatAdmobius
> -vip /data/production/batched/fingerprint/fp_match_matrix/batch_id=
> 2013090614/pa* -op
> /data/production/batched/fingerprint/hashed_cc/batch_id=2013090639 -of
> org.apache.giraph.examples.VertexWithDoubleValueDoubleEdgeTextOutputFormatAdmobius
> -c org.apache.giraph.combiner.MinimumLongCombiner -w 115
>
>
> *Oozie xml:*
>
>
>
>
> * <action name="fingerprint-05-giraph">                 <java>
>             <job-tracker>${ createCluster == true ?
> wf:actionData('start-new-cluster')['jobTracker'] : jobTracker }
>             </job-tracker>                        <name-node>${
> createCluster == true ? wf:actionData('start-new-cluster')['nameNode'] :
> nameNode }                         </name-node>
> <job-xml>${nameNode}/user/${wf:user()}/hive/${env}/job.xml</job-xml>
>                   <configuration>                                <property>
>                                         <name>mapred.job.queue.name
> <http://mapred.job.queue.name></name>
>   <value>${queueName}</value>                                 </property>
>                       </configuration>
> <main-class>org.apache.giraph.GiraphRunner</main-class>
>     <arg>org.apache.giraph.examples.ConnectedComponentsVertexAdmobius</arg>
>                         <arg>-vif</arg>
> <arg>org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormatAdmobius</arg>
>                         <arg>-vip</arg>
> <arg>${nameNode}/data/production/batched/fingerprint/fp_match_matrix/batch_id=${batchId}/pa*</arg>
>                         <arg>-op</arg>
> <arg>${nameNode}/data/production/batched/fingerprint/hashed_cc/batch_id=${batchId}</arg>
>                         <arg>-of</arg>
> <arg>org.apache.giraph.examples.VertexWithDoubleValueDoubleEdgeTextOutputFormatAdmobius</arg>
>                         <arg>-c</arg>
> <arg>org.apache.giraph.combiner.MinimumLongCombiner</arg>
>       <arg>-w</arg>                        <arg>118</arg>
>       <file>giraph-site.xml</file>                        <capture-output
> />                 </java>                 <ok
> to="fingerprint-update-connnectedcomponents" />                 <error
> to="fail" />        </action> Any pointers over it would be appreciated.*
> --
> Best Regards,
> Jyotirmoy Sundi
> Data Engineer,
> Admobius
>
> San Francisco, CA 94158
>

Mime
View raw message