hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ajit Kumar Shreevastava <Ajit.Shreevast...@hcl.com>
Subject RE: hive mapred problem
Date Mon, 08 Oct 2012 12:00:22 GMT
Hi Nitin,



Job tracker log:-->



2012-10-08 15:43:29,331 WARN org.apache.hadoop.conf.Configuration: /usr/local/hadoopstorage/mapred/local/jobTracker/job_201210051717_0015.xml:a
attempt to override fina

l parameter: mapred.reduce.tasks;  Ignoring.

2012-10-08 15:43:29,331 WARN org.apache.hadoop.conf.Configuration: /usr/local/hadoopstorage/mapred/local/jobTracker/job_201210051717_0015.xml:a
attempt to override fina

l parameter: mapred.map.tasks;  Ignoring.

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobInProgress: job_201210051717_0015:
nMaps=10 nReduces=2 max=-1

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobTracker: Job job_201210051717_0015
added successfully for user 'hadoop' to queue 'default'

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.AuditLogger: USER=hadoop  IP=10.99.42.9
  OPERATION=SUBMIT_JOB    TARGET=job_201210051717_0015    RESULT=SUCCESS

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201210051717_0015

2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201210051717_0015

2012-10-08 15:43:29,350 INFO org.apache.hadoop.mapred.JobInProgress: jobToken generated and
stored with users keys in /usr/local/hadoopstorage/mapred/system/job_2012100

51717_0015/jobToken

2012-10-08 15:43:29,352 ERROR org.apache.hadoop.mapred.JobTracker: Job initialization failed:

java.io.IOException: Number of maps in JobConf doesn't match number of recieved splits for
job job_201210051717_0015! numMapTasks=10, #splits=1

        at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:703)

        at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207)

        at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:662)



2012-10-08 15:43:29,352 INFO org.apache.hadoop.mapred.JobTracker: Failing job job_201210051717_0015

2012-10-08 15:43:29,352 INFO org.apache.hadoop.mapred.JobInProgress$JobSummary: jobId=job_201210051717_0015,submitTime=1349691209316,launchTime=0,,finishTime=1349691209

352,numMaps=0,numSlotsPerMap=1,numReduces=0,numSlotsPerReduce=1,user=hadoop,queue=default,status=FAILED,mapSlotSeconds=0,reduceSlotsSeconds=0,clusterMapCapacity=14,clus

terReduceCapacity=14,jobName=select count(1) from pokes(Stage-1)

2012-10-08 15:43:29,353 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop/history/job_201210051717_0015_1349691209316_hadoop_select+count%281%29+fro

m+pokes%28Stage-1%29 to file:/var/log/hadoop/history/done/version-1/NHCLT-PC44-2_1349437647390_/2012/10/08/000000

2012-10-08 15:43:29,356 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop/history/job_201210051717_0015_conf.xml
to file:/var/log/hadoop/history/don

e/version-1/NHCLT-PC44-2_1349437647390_/2012/10/08/000000

2012-10-08 15:43:30,362 INFO org.apache.hadoop.mapred.JobTracker: Killing job job_201210051717_0015

2012-10-08 17:17:35,641 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Updating the current master key for generating delegation

tokens





and hive log:-->



2012-10-08 15:43:08,597 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/tmp/hive-default-1260952199191868105.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:08,606 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:08,606 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:09,000 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/tmp/hive-default-1260952199191868105.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:09,004 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:09,004 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:09,014 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/tmp/hive-default-1260952199191868105.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:09,018 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:09,018 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:17,309 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/tmp/hive-default-7458429923592584125.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:17,319 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:17,320 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:17,759 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/tmp/hive-default-7458429923592584125.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:17,766 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:17,766 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:17,783 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/tmp/hive-default-7458429923592584125.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:17,786 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:17,787 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core"
requires "org.eclipse.core.resources" but it cannot be resolved.

2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core"
requires "org.eclipse.core.resources" but it cannot be resolved.

2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core"
requires "org.eclipse.core.runtime" but it cannot be resolved.

2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core"
requires "org.eclipse.core.runtime" but it cannot be resolved.

2012-10-08 15:43:24,120 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core"
requires "org.eclipse.text" but it cannot be resolved.

2012-10-08 15:43:24,120 ERROR DataNucleus.Plugin (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core"
requires "org.eclipse.text" but it cannot be resolved.

2012-10-08 15:43:24,734 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/tmp/hive-default-7458429923592584125.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:24,750 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:24,751 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:28,535 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/tmp/hive-default-7458429923592584125.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:28,537 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:28,537 WARN  conf.Configuration (Configuration.java:loadResource(1245)) -
file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final parameter: mapred.reduce.tasks;
 Ignoring.

2012-10-08 15:43:29,113 WARN  mapred.JobClient (JobClient.java:copyAndConfigureFiles(667))
- Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for
the same.

2012-10-08 15:43:29,243 WARN  snappy.LoadSnappy (LoadSnappy.java:<clinit>(46)) - Snappy
native library not loaded

2012-10-08 15:43:30,354 ERROR exec.Task (SessionState.java:printError(400)) - Ended Job =
job_201210051717_0015 with errors

2012-10-08 15:43:30,356 ERROR exec.Task (SessionState.java:printError(400)) - Error during
job, obtaining debugging information...

2012-10-08 15:43:30,369 ERROR ql.Driver (SessionState.java:printError(400)) - FAILED: Execution
Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask





Regards,

Ajit



-----Original Message-----
From: Nitin Pawar [mailto:nitinpawar432@gmail.com]
Sent: Monday, October 08, 2012 4:19 PM
To: user@hive.apache.org
Subject: Re: hive mapred problem



Can you provide with job logs and failed task logs?





On Mon, Oct 8, 2012 at 4:13 PM, Ajit Kumar Shreevastava

<Ajit.Shreevastava@hcl.com> wrote:

> Hi,

>

>

>

> When I run the query "select count(1) from pokes;" it fails with the message

> as below.

>

>

>

> hive> select count(1) from pokes;

>

> Total MapReduce jobs = 1

>

> Launching Job 1 out of 1

>

> Number of reduce tasks determined at compile time: 1

>

> In order to change the average load for a reducer (in bytes):

>

>   set hive.exec.reducers.bytes.per.reducer=<number>

>

> In order to limit the maximum number of reducers:

>

>   set hive.exec.reducers.max=<number>

>

> In order to set a constant number of reducers:

>

>   set mapred.reduce.tasks=<number>

>

> Starting Job = job_201210051717_0015, Tracking URL =

> http://NHCLT-PC44-2:50030/jobdetails.jsp?jobid=job_201210051717_0015

>

> Kill Command = /home/hadoop/hadoop-1.0.3/bin/hadoop job  -kill

> job_201210051717_0015

>

> Hadoop job information for Stage-1: number of mappers: 0; number of

> reducers: 0

>

> 2012-10-08 15:43:30,351 Stage-1 map = 100%,  reduce = 100%

>

> Ended Job = job_201210051717_0015 with errors

>

> Error during job, obtaining debugging information...

>

> FAILED: Execution Error, return code 2 from

> org.apache.hadoop.hive.ql.exec.MapRedTask

>

> MapReduce Jobs Launched:

>

> Job 0:  HDFS Read: 0 HDFS Write: 0 FAIL

>

> Total MapReduce CPU Time Spent: 0 msec

>

>

>

> Thanks and Regards

>

> Ajit Kumar Shreevastava

>

> ADCOE (App Development Center Of Excellence )

>

> Mobile: 9717775634

>

>

>

>

>

> ::DISCLAIMER::

> ----------------------------------------------------------------------------------------------------------------------------------------------------

>

> The contents of this e-mail and any attachment(s) are confidential and

> intended for the named recipient(s) only.

> E-mail transmission is not guaranteed to be secure or error-free as

> information could be intercepted, corrupted,

> lost, destroyed, arrive late or incomplete, or may contain viruses in

> transmission. The e mail and its contents

> (with or without referred errors) shall therefore not attach any liability

> on the originator or HCL or its affiliates.

> Views or opinions, if any, presented in this email are solely those of the

> author and may not necessarily reflect the

> views or opinions of HCL or its affiliates. Any form of reproduction,

> dissemination, copying, disclosure, modification,

> distribution and / or publication of this message without the prior written

> consent of authorized representative of

> HCL is strictly prohibited. If you have received this email in error please

> delete it and notify the sender immediately.

> Before opening any email and/or attachments, please check them for viruses

> and other defects.

>

> ----------------------------------------------------------------------------------------------------------------------------------------------------







--

Nitin Pawar

Mime
View raw message