pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Asim Zafir <asim.za...@gmail.com>
Subject Pig jobs running in LocalJobRunner Mode once Migrated to YARN (MRV2)
Date Wed, 01 May 2019 19:01:38 GMT
** appologies for sending multiple emails as the previous once got
truncated some how **
Hi,
I am struggling to find the root cause of pig jobs not running in Cluster
Mode after we upgraded the cluster to use YARN (MRV2) from MRV1. When the
job is submitted, I see the following message as the most obvious
difference from the old cluster where the job was running fine on Cluster
mode. :

On the new cluster (where Yarn is running) I see the following: It appears
the job is not getting submitted to the RM. I have ensure the PIG_HOME and
HADOOP_MAPRED_HOME is set right but despite of many tries this is what I am
getting. Since the job admission isn't successful on YARN its failing back
to LocalJobRunner Mode.

Pig jobs were running fine with MRv1 but ever since the cluster was
upgraded to use YARN, this problem is being noticed. I have ensured the
classPATHs and environment variable are set right on this environment. We
are using Cloudera Distribution (CDH 5.10) and pig version 0.12 (Apache Pig
version 0.12.0-cdh5.10.2)

Please assist with a solution - also do you think is there
any incompatibility with PIG running with YARN in the recent or previous
release of PIG or YARN?

Thanks
Asim Zafir.

2019-05-01 17:37:23,485 [main] DEBUG org.apache.pig.impl.util.JarManager -
Adding entry pigContext to job jar

2019-05-01 17:37:23,721 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- jar file Job2836672534872456251.jar created

2019-05-01 17:37:23,721 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.jar is
deprecated. Instead, use mapreduce.job.jar

2019-05-01 17:37:23,742 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job

2019-05-01 17:37:23,745 [main] DEBUG
org.apache.pig.data.SchemaTupleFrontend - Nothing registered to generate.

*2019-05-01 17:37:23,773 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.*

2019-05-01 17:37:23,774 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation -
mapred.job.tracker.http.address is deprecated. Instead, use
mapreduce.jobtracker.http.address

2019-05-01 17:37:23,774 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is
deprecated. Instead, use mapreduce.jobtracker.address

2019-05-01 17:37:23,776 [JobControl] DEBUG
org.apache.pig.backend.hadoop23.PigJobControl - Checking state of job job
name: PigLatin:testrun.pig

job id: job_pigexec_0

job state: WAITING

job mapred id: null

job message: just initialized

job has no depending job:

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message