Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 332A5D4B1 for ; Tue, 9 Oct 2012 11:30:06 +0000 (UTC) Received: (qmail 99611 invoked by uid 500); 9 Oct 2012 11:30:04 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 99346 invoked by uid 500); 9 Oct 2012 11:29:59 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 99301 invoked by uid 99); 9 Oct 2012 11:29:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Oct 2012 11:29:58 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Ajit.Shreevastava@hcl.com designates 203.105.185.23 as permitted sender) Received: from [203.105.185.23] (HELO gws05.hcl.com) (203.105.185.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Oct 2012 11:29:49 +0000 X-IronPort-AV: E=Sophos;i="4.80,560,1344191400"; d="scan'208,217";a="13843922" Received: from unknown (HELO NDA-HCLIN-HT02.CORP.HCL.IN) ([10.248.64.36]) by gws05.hcl.com with ESMTP/TLS/AES128-SHA; 09 Oct 2012 16:55:25 +0530 Received: from NDA-HCLT-HT02.HCLT.CORP.HCL.IN (10.98.10.212) by NDA-HCLIN-HT02.CORP.HCL.IN (10.248.64.36) with Microsoft SMTP Server (TLS) id 8.3.245.1; Tue, 9 Oct 2012 16:59:22 +0530 Received: from nda-hclt-evs06.HCLT.CORP.HCL.IN ([10.98.11.179]) by NDA-HCLT-HT02.HCLT.CORP.HCL.IN ([::1]) with mapi; Tue, 9 Oct 2012 16:59:22 +0530 From: Ajit Kumar Shreevastava To: "user@hive.apache.org" Date: Tue, 9 Oct 2012 16:59:20 +0530 Subject: RE: hive mapred problem Thread-Topic: hive mapred problem Thread-Index: Ac2mCYL5QST15toXSvqJWo9ynSn2mAAB14Sw Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_F0232037B611204D9A7090EC62CF180B06A7979C3Dndahcltevs06H_" MIME-Version: 1.0 --_000_F0232037B611204D9A7090EC62CF180B06A7979C3Dndahcltevs06H_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Nitin, Thanks for your reply. I want configuration file of hive with respect to distributed hadoop. Again My jps command is also failed in datanode. When I run command on master node:--> [hadoop@NHCLT-PC44-2 hadoop]$ jps 5694 NameNode 5992 SecondaryNameNode 5843 DataNode 6224 TaskTracker 6085 JobTracker 6701 Jps 4785 RunJar But when I run the same command on slave node:--> [hadoop@NHCLT-PC44-4 hadoop]$ jps -bash: jps: command not found Can you please help me in this regard. -----Original Message----- From: Nitin Pawar [mailto:nitinpawar432@gmail.com] Sent: Tuesday, October 09, 2012 4:03 PM To: user@hive.apache.org Subject: Re: hive mapred problem I am not sure what you mean by distibuted hive. but a multinode hadoop cluster setup you can setup using following link http://ankitasblogger.blogspot.in/2011/01/hadoop-cluster-setup.html if you run into hic-ups feel free to reach me out on my email (dont want hive userlist to be bugged for hadoop installations) On Tue, Oct 9, 2012 at 3:54 PM, Ajit Kumar Shreevastava wrote: > Hi Nitin > > Sorry Nitin, Actually I mean Fully distributed mode( hadoop on multimode)= . > I want configuration file for both hadoop and hive. > > > > -----Original Message----- > From: Nitin Pawar [mailto:nitinpawar432@gmail.com] > Sent: Tuesday, October 09, 2012 2:46 PM > To: user@hive.apache.org > Subject: Re: hive mapred problem > > I did not get the distributed mode for hadoop and hive question. can > you explain what exactly what you want to achieve ? > > Thanks, > Nitin > > On Tue, Oct 9, 2012 at 2:42 PM, Ajit Kumar Shreevastava > wrote: >> Hi Nitin, >> >> >> >> Thanks for your reply... >> >> >> >> Now my query is running but output is like :--> >> >> >> >> hive> select count(1) from pokes; >> >> Total MapReduce jobs =3D 1 >> >> Launching Job 1 out of 1 >> >> Number of reduce tasks determined at compile time: 1 >> >> In order to change the average load for a reducer (in bytes): >> >> set hive.exec.reducers.bytes.per.reducer=3D >> >> In order to limit the maximum number of reducers: >> >> set hive.exec.reducers.max=3D >> >> In order to set a constant number of reducers: >> >> set mapred.reduce.tasks=3D >> >> Starting Job =3D job_201210091435_0001, Tracking URL =3D >> http://NHCLT-PC44-2:50030/jobdetails.jsp?jobid=3Djob_201210091435_0001 >> >> Kill Command =3D /home/hadoop/hadoop-1.0.3/bin/hadoop job -kill >> job_201210091435_0001 >> >> Hadoop job information for Stage-1: number of mappers: 1; number of >> reducers: 2 >> >> 2012-10-09 14:37:14,587 Stage-1 map =3D 0%, reduce =3D 0% >> >> 2012-10-09 14:37:20,609 Stage-1 map =3D 100%, reduce =3D 0%, Cumulative= CPU >> 0.47 sec >> >> 2012-10-09 14:37:21,613 Stage-1 map =3D 100%, reduce =3D 0%, Cumulative= CPU >> 0.47 sec >> >> 2012-10-09 14:37:22,620 Stage-1 map =3D 100%, reduce =3D 0%, Cumulative= CPU >> 0.47 sec >> >> 2012-10-09 14:37:23,625 Stage-1 map =3D 100%, reduce =3D 0%, Cumulative= CPU >> 0.47 sec >> >> 2012-10-09 14:37:24,630 Stage-1 map =3D 100%, reduce =3D 0%, Cumulative= CPU >> 0.47 sec >> >> 2012-10-09 14:37:25,634 Stage-1 map =3D 100%, reduce =3D 0%, Cumulative= CPU >> 0.47 sec >> >> 2012-10-09 14:37:26,638 Stage-1 map =3D 100%, reduce =3D 0%, Cumulative= CPU >> 0.47 sec >> >> 2012-10-09 14:37:27,642 Stage-1 map =3D 100%, reduce =3D 0%, Cumulative= CPU >> 0.47 sec >> >> 2012-10-09 14:37:28,650 Stage-1 map =3D 100%, reduce =3D 0%, Cumulative= CPU >> 0.47 sec >> >> 2012-10-09 14:37:29,654 Stage-1 map =3D 100%, reduce =3D 0%, Cumulative= CPU >> 0.47 sec >> >> 2012-10-09 14:37:30,658 Stage-1 map =3D 100%, reduce =3D 0%, Cumulative= CPU >> 0.47 sec >> >> 2012-10-09 14:37:31,662 Stage-1 map =3D 100%, reduce =3D 0%, Cumulative= CPU >> 0.47 sec >> >> 2012-10-09 14:37:32,667 Stage-1 map =3D 100%, reduce =3D 50%, Cumulativ= e CPU >> 1.66 sec >> >> 2012-10-09 14:37:33,672 Stage-1 map =3D 100%, reduce =3D 50%, Cumulativ= e CPU >> 1.66 sec >> >> 2012-10-09 14:37:34,678 Stage-1 map =3D 100%, reduce =3D 100%, Cumulati= ve CPU >> 3.0 sec >> >> 2012-10-09 14:37:35,682 Stage-1 map =3D 100%, reduce =3D 100%, Cumulati= ve CPU >> 3.0 sec >> >> 2012-10-09 14:37:36,686 Stage-1 map =3D 100%, reduce =3D 100%, Cumulati= ve CPU >> 3.0 sec >> >> 2012-10-09 14:37:37,690 Stage-1 map =3D 100%, reduce =3D 100%, Cumulati= ve CPU >> 3.0 sec >> >> 2012-10-09 14:37:38,694 Stage-1 map =3D 100%, reduce =3D 100%, Cumulati= ve CPU >> 3.0 sec >> >> 2012-10-09 14:37:39,698 Stage-1 map =3D 100%, reduce =3D 100%, Cumulati= ve CPU >> 3.0 sec >> >> 2012-10-09 14:37:40,702 Stage-1 map =3D 100%, reduce =3D 100%, Cumulati= ve CPU >> 3.0 sec >> >> MapReduce Total cumulative CPU time: 3 seconds 0 msec >> >> Ended Job =3D job_201210091435_0001 >> >> MapReduce Jobs Launched: >> >> Job 0: Map: 1 Reduce: 2 Cumulative CPU: 3.0 sec HDFS Read: 6034 HDF= S >> Write: 6 SUCCESS >> >> Total MapReduce CPU Time Spent: 3 seconds 0 msec >> >> OK >> >> 500 >> >> 0 >> >> Time taken: 35.161 seconds >> >> >> >> Can you do one favor for me? I want configuration file template for >> distributed mode for both hadoop and hive. >> >> >> >> Regards >> >> Ajit >> >> >> >> >> >> -----Original Message----- >> From: Nitin Pawar [mailto:nitinpawar432@gmail.com] >> Sent: Monday, October 08, 2012 5:52 PM >> To: user@hive.apache.org >> Subject: Re: hive mapred problem >> >> >> >> from the error looks like you have some incorrect hive settings which >> >> are failing the job initialization. >> >> >> >> this is the error >> >>>java.io.IOException: Number of maps in JobConf doesn't match number of >> >>> recieved splits for job job_201210051717_0015! numMapTasks=3D10 >> >> >> >> can you tell us if you are setting in hive variables before firing up >> >> the query? something like split size or # no maps etc >> >> >> >> On Mon, Oct 8, 2012 at 5:30 PM, Ajit Kumar Shreevastava >> >> wrote: >> >>> Hi Nitin, >> >>> >> >>> >> >>> >> >>> Job tracker log:--> >> >>> >> >>> >> >>> >> >>> 2012-10-08 15:43:29,331 WARN org.apache.hadoop.conf.Configuration: >> >>> >>> /usr/local/hadoopstorage/mapred/local/jobTracker/job_201210051717_0015.= xml:a >> >>> attempt to override fina >> >>> >> >>> l parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:29,331 WARN org.apache.hadoop.conf.Configuration: >> >>> >>> /usr/local/hadoopstorage/mapred/local/jobTracker/job_201210051717_0015.= xml:a >> >>> attempt to override fina >> >>> >> >>> l parameter: mapred.map.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> job_201210051717_0015: nMaps=3D10 nReduces=3D2 max=3D-1 >> >>> >> >>> 2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobTracker: Job >> >>> job_201210051717_0015 added successfully for user 'hadoop' to queue >> >>> 'default' >> >>> >> >>> 2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.AuditLogger: >> >>> USER=3Dhadoop IP=3D10.99.42.9 OPERATION=3DSUBMIT_JOB >> >>> TARGET=3Djob_201210051717_0015 RESULT=3DSUCCESS >> >>> >> >>> 2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobTracker: >> >>> Initializing job_201210051717_0015 >> >>> >> >>> 2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> Initializing job_201210051717_0015 >> >>> >> >>> 2012-10-08 15:43:29,350 INFO org.apache.hadoop.mapred.JobInProgress: >> >>> jobToken generated and stored with users keys in >> >>> /usr/local/hadoopstorage/mapred/system/job_2012100 >> >>> >> >>> 51717_0015/jobToken >> >>> >> >>> 2012-10-08 15:43:29,352 ERROR org.apache.hadoop.mapred.JobTracker: Job >> >>> initialization failed: >> >>> >> >>> java.io.IOException: Number of maps in JobConf doesn't match number of >> >>> recieved splits for job job_201210051717_0015! numMapTasks=3D10, #split= s=3D1 >> >>> >> >>> at >> >>> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:703= ) >> >>> >> >>> at >>> org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4207) >> >>> >> >>> at >> >>> >>> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(Ea= gerTaskInitializationListener.java:79) >> >>> >> >>> at >> >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecut= or.java:886) >> >>> >> >>> at >> >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j= ava:908) >> >>> >> >>> at java.lang.Thread.run(Thread.java:662) >> >>> >> >>> >> >>> >> >>> 2012-10-08 15:43:29,352 INFO org.apache.hadoop.mapred.JobTracker: Faili= ng >> >>> job job_201210051717_0015 >> >>> >> >>> 2012-10-08 15:43:29,352 INFO >> >>> org.apache.hadoop.mapred.JobInProgress$JobSummary: >> >>> >>> jobId=3Djob_201210051717_0015,submitTime=3D1349691209316,launchTime=3D0= ,,finishTime=3D1349691209 >> >>> >> >>> >>> 352,numMaps=3D0,numSlotsPerMap=3D1,numReduces=3D0,numSlotsPerReduce=3D1= ,user=3Dhadoop,queue=3Ddefault,status=3DFAILED,mapSlotSeconds=3D0,reduceSlo= tsSeconds=3D0,clusterMapCapacity=3D14,clus >> >>> >> >>> terReduceCapacity=3D14,jobName=3Dselect count(1) from pokes(Stage-1) >> >>> >> >>> 2012-10-08 15:43:29,353 INFO org.apache.hadoop.mapred.JobHistory: Movin= g >> >>> >>> file:/var/log/hadoop/history/job_201210051717_0015_1349691209316_hadoop= _select+count%281%29+fro >> >>> >> >>> m+pokes%28Stage-1%29 to >> >>> >>> file:/var/log/hadoop/history/done/version-1/NHCLT-PC44-2_1349437647390_= /2012/10/08/000000 >> >>> >> >>> 2012-10-08 15:43:29,356 INFO org.apache.hadoop.mapred.JobHistory: Movin= g >> >>> file:/var/log/hadoop/history/job_201210051717_0015_conf.xml to >> >>> file:/var/log/hadoop/history/don >> >>> >> >>> e/version-1/NHCLT-PC44-2_1349437647390_/2012/10/08/000000 >> >>> >> >>> 2012-10-08 15:43:30,362 INFO org.apache.hadoop.mapred.JobTracker: Killi= ng >> >>> job job_201210051717_0015 >> >>> >> >>> 2012-10-08 17:17:35,641 INFO >> >>> >>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecr= etManager: >> >>> Updating the current master key for generating delegation >> >>> >> >>> tokens >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> and hive log:--> >> >>> >> >>> >> >>> >> >>> 2012-10-08 15:43:08,597 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/tmp/hive-default-1260952199191868105.xml:a attempt to override fi= nal >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:08,606 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:08,606 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:09,000 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/tmp/hive-default-1260952199191868105.xml:a attempt to override fi= nal >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:09,004 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:09,004 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:09,014 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/tmp/hive-default-1260952199191868105.xml:a attempt to override fi= nal >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:09,018 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:09,018 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:17,309 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/tmp/hive-default-7458429923592584125.xml:a attempt to override fi= nal >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:17,319 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:17,320 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:17,759 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/tmp/hive-default-7458429923592584125.xml:a attempt to override fi= nal >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:17,766 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:17,766 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:17,783 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/tmp/hive-default-7458429923592584125.xml:a attempt to override fi= nal >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:17,786 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:17,787 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin >> >>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires >> >>> "org.eclipse.core.resources" but it cannot be resolved. >> >>> >> >>> 2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin >> >>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires >> >>> "org.eclipse.core.resources" but it cannot be resolved. >> >>> >> >>> 2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin >> >>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires >> >>> "org.eclipse.core.runtime" but it cannot be resolved. >> >>> >> >>> 2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin >> >>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires >> >>> "org.eclipse.core.runtime" but it cannot be resolved. >> >>> >> >>> 2012-10-08 15:43:24,120 ERROR DataNucleus.Plugin >> >>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires >> >>> "org.eclipse.text" but it cannot be resolved. >> >>> >> >>> 2012-10-08 15:43:24,120 ERROR DataNucleus.Plugin >> >>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core" requires >> >>> "org.eclipse.text" but it cannot be resolved. >> >>> >> >>> 2012-10-08 15:43:24,734 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/tmp/hive-default-7458429923592584125.xml:a attempt to override fi= nal >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:24,750 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:24,751 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:28,535 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/tmp/hive-default-7458429923592584125.xml:a attempt to override fi= nal >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:28,537 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:28,537 WARN conf.Configuration >> >>> (Configuration.java:loadResource(1245)) - >> >>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final >> >>> parameter: mapred.reduce.tasks; Ignoring. >> >>> >> >>> 2012-10-08 15:43:29,113 WARN mapred.JobClient >> >>> (JobClient.java:copyAndConfigureFiles(667)) - Use GenericOptionsParser = for >> >>> parsing the arguments. Applications should implement Tool for the same. >> >>> >> >>> 2012-10-08 15:43:29,243 WARN snappy.LoadSnappy >> >>> (LoadSnappy.java:(46)) - Snappy native library not loaded >> >>> >> >>> 2012-10-08 15:43:30,354 ERROR exec.Task >>> (SessionState.java:printError(400)) >> >>> - Ended Job =3D job_201210051717_0015 with errors >> >>> >> >>> 2012-10-08 15:43:30,356 ERROR exec.Task >>> (SessionState.java:printError(400)) >> >>> - Error during job, obtaining debugging information... >> >>> >> >>> 2012-10-08 15:43:30,369 ERROR ql.Driver >>> (SessionState.java:printError(400)) >> >>> - FAILED: Execution Error, return code 2 from >> >>> org.apache.hadoop.hive.ql.exec.MapRedTask >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> Regards, >> >>> >> >>> Ajit >> >>> >> >>> >> >>> >> >>> -----Original Message----- >> >>> From: Nitin Pawar [mailto:nitinpawar432@gmail.com] >> >>> Sent: Monday, October 08, 2012 4:19 PM >> >>> To: user@hive.apache.org >> >>> Subject: Re: hive mapred problem >> >>> >> >>> >> >>> >> >>> Can you provide with job logs and failed task logs? >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> On Mon, Oct 8, 2012 at 4:13 PM, Ajit Kumar Shreevastava >> >>> >> >>> wrote: >> >>> >> >>>> Hi, >> >>> >> >>>> >> >>> >> >>>> >> >>> >> >>>> >> >>> >> >>>> When I run the query "select count(1) from pokes;" it fails with the >> >>>> message >> >>> >> >>>> as below. >> >>> >> >>>> >> >>> >> >>>> >> >>> >> >>>> >> >>> >> >>>> hive> select count(1) from pokes; >> >>> >> >>>> >> >>> >> >>>> Total MapReduce jobs =3D 1 >> >>> >> >>>> >> >>> >> >>>> Launching Job 1 out of 1 >> >>> >> >>>> >> >>> >> >>>> Number of reduce tasks determined at compile time: 1 >> >>> >> >>>> >> >>> >> >>>> In order to change the average load for a reducer (in bytes): >> >>> >> >>>> >> >>> >> >>>> set hive.exec.reducers.bytes.per.reducer=3D >> >>> >> >>>> >> >>> >> >>>> In order to limit the maximum number of reducers: >> >>> >> >>>> >> >>> >> >>>> set hive.exec.reducers.max=3D >> >>> >> >>>> >> >>> >> >>>> In order to set a constant number of reducers: >> >>> >> >>>> >> >>> >> >>>> set mapred.reduce.tasks=3D >> >>> >> >>>> >> >>> >> >>>> Starting Job =3D job_201210051717_0015, Tracking URL =3D >> >>> >> >>>> http://NHCLT-PC44-2:50030/jobdetails.jsp?jobid=3Djob_201210051717_0015 >> >>> >> >>>> >> >>> >> >>>> Kill Command =3D /home/hadoop/hadoop-1.0.3/bin/hadoop job -kill >> >>> >> >>>> job_201210051717_0015 >> >>> >> >>>> >> >>> >> >>>> Hadoop job information for Stage-1: number of mappers: 0; number of >> >>> >> >>>> reducers: 0 >> >>> >> >>>> >> >>> >> >>>> 2012-10-08 15:43:30,351 Stage-1 map =3D 100%, reduce =3D 100% >> >>> >> >>>> >> >>> >> >>>> Ended Job =3D job_201210051717_0015 with errors >> >>> >> >>>> >> >>> >> >>>> Error during job, obtaining debugging information... >> >>> >> >>>> >> >>> >> >>>> FAILED: Execution Error, return code 2 from >> >>> >> >>>> org.apache.hadoop.hive.ql.exec.MapRedTask >> >>> >> >>>> >> >>> >> >>>> MapReduce Jobs Launched: >> >>> >> >>>> >> >>> >> >>>> Job 0: HDFS Read: 0 HDFS Write: 0 FAIL >> >>> >> >>>> >> >>> >> >>>> Total MapReduce CPU Time Spent: 0 msec >> >>> >> >>>> >> >>> >> >>>> >> >>> >> >>>> >> >>> >> >>>> Thanks and Regards >> >>> >> >>>> >> >>> >> >>>> Ajit Kumar Shreevastava >> >>> >> >>>> >> >>> >> >>>> ADCOE (App Development Center Of Excellence ) >> >>> >> >>>> >> >>> >> >>>> Mobile: 9717775634 >> >>> >> >>>> >> >>> >> >>>> >> >>> >> >>>> >> >>> >> >>>> >> >>> >> >>>> >> >>> >> >>>> ::DISCLAIMER:: >> >>> >> >>>> >> >>>> >>>> ----------------------------------------------------------------------= ---------------------------------------------------------------------------= --- >> >>> >> >>>> >> >>> >> >>>> The contents of this e-mail and any attachment(s) are confidential and >> >>> >> >>>> intended for the named recipient(s) only. >> >>> >> >>>> E-mail transmission is not guaranteed to be secure or error-free as >> >>> >> >>>> information could be intercepted, corrupted, >> >>> >> >>>> lost, destroyed, arrive late or incomplete, or may contain viruses in >> >>> >> >>>> transmission. The e mail and its contents >> >>> >> >>>> (with or without referred errors) shall therefore not attach any >>>> liability >> >>> >> >>>> on the originator or HCL or its affiliates. >> >>> >> >>>> Views or opinions, if any, presented in this email are solely those of >>>> the >> >>> >> >>>> author and may not necessarily reflect the >> >>> >> >>>> views or opinions of HCL or its affiliates. Any form of reproduction, >> >>> >> >>>> dissemination, copying, disclosure, modification, >> >>> >> >>>> distribution and / or publication of this message without the prior >> >>>> written >> >>> >> >>>> consent of authorized representative of >> >>> >> >>>> HCL is strictly prohibited. If you have received this email in error >> >>>> please >> >>> >> >>>> delete it and notify the sender immediately. >> >>> >> >>>> Before opening any email and/or attachments, please check them for >>>> viruses >> >>> >> >>>> and other defects. >> >>> >> >>>> >> >>> >> >>>> >> >>>> >>>> ----------------------------------------------------------------------= ---------------------------------------------------------------------------= --- >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> >> >>> Nitin Pawar >> >> >> >> >> >> >> >> -- >> >> Nitin Pawar > > > > -- > Nitin Pawar -- Nitin Pawar --_000_F0232037B611204D9A7090EC62CF180B06A7979C3Dndahcltevs06H_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Hi Nitin,

 

Thanks for your reply.

I want co= nfiguration file of hive with respect to distributed hadoop.

=

 

Again = My jps command is also failed in datanode.

 

When I run comman= d on master node:=E0

 

<= b>[hadoop@NHCLT-PC44-2 hadoop]$ jps

5694 NameNode

5992 SecondaryNameNo= de

5843 DataNode

6224 TaskTracker

60= 85 JobTracker

6701 Jps

=

4785 RunJar

<= o:p> 

But when I run the same command= on slave node:=E0<= /p>

[hadoop@NHCLT-PC44-4 hadoop]$ jps<= /b>

-bash: jps: command not found

=

 

Can yo= u please help me in this regard.

 

 

-----Original Message-----
From: Nitin Pawar [mailto:nitinpaw= ar432@gmail.com]
Sent: Tuesday, October 09, 2012 4:03 PM
To: user@hi= ve.apache.org
Subject: Re: hive mapred problem

 

I am not sure wha= t you mean by distibuted hive.

&= nbsp;

but a multinode hadoop cluster setup= you can setup using following link

h= ttp://ankitasblogger.blogspot.in/2011/01/hadoop-cluster-setup.html

 

= if you run into hic-ups feel free to reach me out on my email (dont

want hive userlist to be bugged for hadoop i= nstallations)

 

 

&nb= sp;

 

On Tue, Oct 9, 2012 at 3:54 PM, Ajit Kumar Shreevastava

<Ajit.Shreevastava@hcl.com> wrote:

> Hi Nitin

> 

> Sorry Nitin, A= ctually I mean Fully distributed mode( hadoop on multimode).

=

> I want configuration file for both hadoop and = hive.

> 

> 

> 

> -----Original Message-----

> From: Nitin Pawar [mailto:nitinpaw= ar432@gmail.com]

> Sent: Tuesday, = October 09, 2012 2:46 PM

> To: use= r@hive.apache.org

> Subject: Re: h= ive mapred problem

> 

> I did not get the distributed mode for = hadoop and hive question. can

> yo= u explain what exactly what you want to achieve ?

> 

> Thanks,<= o:p>

> Nitin

> 

> On Tue, O= ct 9, 2012 at 2:42 PM, Ajit Kumar Shreevastava

> <Ajit.Shreevastava@hcl.com> wrote:

>> Hi Nitin,

>> 

>> 

>> 

>> Thanks for your reply...

>> 

>>&n= bsp;

>> 

>> Now my query is running but output is like :-->=

>> 

>> 

>&= gt; 

>> hive> select co= unt(1) from pokes;

>> = ;

>> Total MapReduce jobs =3D 1=

>> 

>> Launching Job 1 out of 1

>> 

>> Numb= er of reduce tasks determined at compile time: 1

>> 

>> I= n order to change the average load for a reducer (in bytes):

=

>> 

>>=A0=A0 set hive.exec.reducers.bytes.per.reducer=3D<number><= o:p>

>> 

>> In order to limit the maximum number of reducers:<= o:p>

>> 

>>=A0=A0 set hive.exec.reducers.max=3D<number><= o:p>

>> 

>> In order to set a constant number of reducers:

>> 

>>=A0=A0 set mapred.reduce.tasks=3D<number>

>> 

>> Starting Job =3D job_201210091435_0001, Tracking URL =3D=

>> http://NHCLT-PC44-2:50030/j= obdetails.jsp?jobid=3Djob_201210091435_0001

>> 

>> Kill C= ommand =3D /home/hadoop/hadoop-1.0.3/bin/hadoop job=A0 -kill

=

>> job_201210091435_0001

>> 

>= > Hadoop job information for Stage-1: number of mappers: 1; number of

>> reducers: 2

>> 

&g= t;> 2012-10-09 14:37:14,587 Stage-1 map =3D 0%,=A0 reduce =3D 0%

>> 

>> 2012-10-09 14:37:20,609 Stage-1 map =3D 100%,=A0 reduce = =3D 0%, Cumulative CPU

>> 0.47 = sec

>> 

>> 2012-10-09 14:37:21,613 Stage-1 map =3D 100%,= =A0 reduce =3D 0%, Cumulative CPU

>= ;> 0.47 sec

>> 

>> 2012-10-09 14:37:22,620 Stage-1 map= =3D 100%,=A0 reduce =3D 0%, Cumulative CPU

>> 0.47 sec

>> 

>> 2012-10-09 14:37:23,625 S= tage-1 map =3D 100%,=A0 reduce =3D 0%, Cumulative CPU

>> 0.47 sec

&g= t;> 

>> 2012-10-09 14:3= 7:24,630 Stage-1 map =3D 100%,=A0 reduce =3D 0%, Cumulative CPU<= /p>

>> 0.47 sec

>> 

>> 2012-= 10-09 14:37:25,634 Stage-1 map =3D 100%,=A0 reduce =3D 0%, Cumulative CPU

>> 0.47 sec

>> 

>= > 2012-10-09 14:37:26,638 Stage-1 map =3D 100%,=A0 reduce =3D 0%, Cumula= tive CPU

>> 0.47 sec=

>> 

>> 2012-10-09 14:37:27,642 Stage-1 map =3D 100%,=A0 reduce =3D = 0%, Cumulative CPU

>> 0.47 sec<= o:p>

>> 

>> 2012-10-09 14:37:28,650 Stage-1 map =3D 100%,=A0 r= educe =3D 0%, Cumulative CPU

>>= 0.47 sec

>> 

>> 2012-10-09 14:37:29,654 Stage-1 map =3D = 100%,=A0 reduce =3D 0%, Cumulative CPU

>> 0.47 sec

>>&nbs= p;

>> 2012-10-09 14:37:30,658 Stage-= 1 map =3D 100%,=A0 reduce =3D 0%, Cumulative CPU

>> 0.47 sec

>>= ; 

>> 2012-10-09 14:37:31,= 662 Stage-1 map =3D 100%,=A0 reduce =3D 0%, Cumulative CPU

>> 0.47 sec

>> 

>> 2012-10-09= 14:37:32,667 Stage-1 map =3D 100%,=A0 reduce =3D 50%, Cumulative CPU<= /o:p>

>> 1.66 sec

>> 

>&g= t; 2012-10-09 14:37:33,672 Stage-1 map =3D 100%,=A0 reduce =3D 50%, Cumulat= ive CPU

>> 1.66 sec<= /p>

>> 

>> 2012-10-09 14:37:34,678 Stage-1 map =3D 100%,=A0 reduce =3D 1= 00%, Cumulative CPU

>> 3.0 sec<= o:p>

>> 

>> 2012-10-09 14:37:35,682 Stage-1 map =3D 100%,=A0 r= educe =3D 100%, Cumulative CPU

>&g= t; 3.0 sec

>> <= /p>

>> 2012-10-09 14:37:36,686 Stage-1 map =3D= 100%,=A0 reduce =3D 100%, Cumulative CPU

>> 3.0 sec

>>&n= bsp;

>> 2012-10-09 14:37:37,690 Stag= e-1 map =3D 100%,=A0 reduce =3D 100%, Cumulative CPU

>> 3.0 sec

>= > 

>> 2012-10-09 14:37:= 38,694 Stage-1 map =3D 100%,=A0 reduce =3D 100%, Cumulative CPU<= /p>

>> 3.0 sec

>> 

>> 2012-1= 0-09 14:37:39,698 Stage-1 map =3D 100%,=A0 reduce =3D 100%, Cumulative CPU<= o:p>

>> 3.0 sec

>> 

>= > 2012-10-09 14:37:40,702 Stage-1 map =3D 100%,=A0 reduce =3D 100%, Cumu= lative CPU

>> 3.0 sec

>> 

>> MapReduce Total cumulative CPU time: 3 seconds 0 msec<= /o:p>

>> 

>> Ended Job =3D job_201210091435_0001

>> 

>= ;> MapReduce Jobs Launched:

>&g= t; 

>> Job 0: Map: 1=A0 Re= duce: 2=A0=A0 Cumulative CPU: 3.0 sec=A0=A0 HDFS Read: 6034 HDFS=

>> Write: 6 SUCCESS

>> 

>&= gt; Total MapReduce CPU Time Spent: 3 seconds 0 msec

>> 

>&g= t; OK

>> 

>> 500

&g= t;> 

>> 0

>> 

>> Time taken: 35.161 seconds

>> 

>> <= /o:p>

>> 

>> Can you do one favor for me? I want configuration file = template for

>> distributed mod= e for both hadoop and hive.

>><= o:p> 

>> 

>> 

= >> Regards

>> <= /o:p>

>> Ajit

>> 

>> 

>> 

>> 

>= ;> 

>> -----Original Me= ssage-----

>> From: Nitin Pawar= [mailto:nitinpawar432@gmail.com]

>= ;> Sent: Monday, October 08, 2012 5:52 PM

>> To: user@hive.apache.org

>> Subject: Re: hive mapred problem

>> 

>> 

>> 

>> from the error looks like you have some incorre= ct hive settings which

>>&= nbsp;

>> are failing the job initial= ization.

>> 

>> 

>> 

>> this is th= e error

>> 

=

>>>java.io.IOException: Number of maps in = JobConf doesn't match number of

>&= gt; 

>>> recieved split= s for job job_201210051717_0015! numMapTasks=3D10

>> 

>><= o:p> 

>> 

>> can you tell us if you are setting in hive v= ariables before firing up

>> 

>> the query?=A0 something = like split size or # no maps etc

>= > 

>> <= /p>

>> 

>> On Mon, Oct 8, 2012 at 5:30 PM, Ajit Kumar Shreevastava<= /o:p>

>> 

>> <Ajit.Shreevastava@hcl.com> wrote:

=

>> 

>>> Hi Nitin,

>> 

>>> 

<= p class=3DMsoPlainText>>> 

>>> 

>>&nbs= p;

>>> 

>> 

>= >> Job tracker log:-->

>&= gt; 

>>> 

>> 

>>> 

>> 

>>> 

>> 

>>> 2012-10-08 15:43:29,331 WARN org.apache.hadoop.conf.Configu= ration:

>> 

=

>>> 

>>> /usr/local/hadoopstorage/mapred/local/jobTracker/job_201= 210051717_0015.xml:a

>>&nb= sp;

>>> attempt to override fina<= o:p>

>> 

>>> 

&g= t;> 

>>> l parameter= : mapred.reduce.tasks;=A0 Ignoring.

&= gt;> 

>>> =

>> 

>>> 2012-10-08 15:43:29,331 WARN org.apache.hadoop.con= f.Configuration:

>> <= /o:p>

>>> 

>>> /usr/local/hadoopstorage/mapred/local/jobTrack= er/job_201210051717_0015.xml:a

>&g= t; 

>>> attempt to over= ride fina

>> 

>>> 

>> 

>>> l= parameter: mapred.map.tasks;=A0 Ignoring.

>> 

>>> 

>> 

>>> 2012-10-08 15:43:29,332 INFO org.apache.had= oop.mapred.JobInProgress:

>> 

>>> job_201210051717_001= 5: nMaps=3D10 nReduces=3D2 max=3D-1

&= gt;> 

>>> =

>> 

>>> 2012-10-08 15:43:29,332 INFO org.apache.hadoop.map= red.JobTracker: Job

>>&nbs= p;

>>> job_201210051717_0015 adde= d successfully for user 'hadoop' to queue

>> 

>>> 'def= ault'

>> 

>>> 

>> 

>>> 2012-= 10-08 15:43:29,332 INFO org.apache.hadoop.mapred.AuditLogger:

>> 

>>> USER=3Dhadoop=A0 IP=3D10.99.42.9=A0=A0 OPERATION=3DSUBMIT_J= OB

>> 

>>> TARGET=3Djob_201210051717_0015=A0=A0=A0 RES= ULT=3DSUCCESS

>> 

>>> 

>> 

>>&g= t; 2012-10-08 15:43:29,332 INFO org.apache.hadoop.mapred.JobTracker:

>> 

>>> Initializing job_201210051717_0015

>> 

&= gt;>> 

>> =

>>> 2012-10-08 15:43:29,332 INFO= org.apache.hadoop.mapred.JobInProgress:

>> 

>>> Initi= alizing job_201210051717_0015

>>= ; 

>>> =

>> 

>>> 2012-10-08 15:43:29,350 INFO org.apache.hadoop.mapred.Jo= bInProgress:

>> 

>>> jobToken generated and stored wit= h users keys in

>> 

>>> /usr/local/hadoopstorage/mapre= d/system/job_2012100

>>&nb= sp;

>>> 

>> 

>= ;>> 51717_0015/jobToken

>>= ; 

>>> =

>> 

>>> 2012-10-08 15:43:29,352 ERROR org.apache.hadoop.mapred.J= obTracker: Job

>> 

>>> initialization failed:

>> 

>>> 

>> 

>>> java.io.IOException= : Number of maps in JobConf doesn't match number of

>> 

>&g= t;> recieved splits for job job_201210051717_0015! numMapTasks=3D10, #sp= lits=3D1

>> 

>>> 

>> 

>>>=A0= =A0=A0=A0=A0=A0=A0=A0 at

>> 

>>> org.apache.hadoop.map= red.JobInProgress.initTasks(JobInProgress.java:703)

>> 

>&g= t;> 

>> 

>>>=A0=A0=A0=A0=A0=A0=A0=A0 at

>>> org.apache.hadoop.mapred.JobTr= acker.initJob(JobTracker.java:4207)

&= gt;> 

>>> =

>> 

>>>=A0=A0=A0=A0=A0=A0=A0=A0 at

>> 

>&g= t;> 

>>> org.apache.= hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitiali= zationListener.java:79)

>>=  

>>> 

>> 

= >>>=A0=A0=A0=A0=A0=A0=A0=A0 at

>> 

>>>&n= bsp;

>>> java.util.concurrent.Thr= eadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

<= p class=3DMsoPlainText>>> 

>>> 

>>&nbs= p;

>>>=A0=A0=A0=A0=A0=A0=A0=A0 at=

>> 

>>> 

&= gt;>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolEx= ecutor.java:908)

>> <= /o:p>

>>> 

>> 

>&g= t;>=A0=A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thread.java:662)

>> 

>>> 

>&= gt; 

>>> 

>> 

>>> 

>> 

>>> 2012-10-08 15:43:29= ,352 INFO org.apache.hadoop.mapred.JobTracker: Failing

>> 

>= >> job job_201210051717_0015

&g= t;> 

>>> <= /o:p>

>> 

>>> 2012-10-08 15:43:29,352 INFO

>> 

>&g= t;> org.apache.hadoop.mapred.JobInProgress$JobSummary:

>> 

&= gt;>> 

>>> jobId= =3Djob_201210051717_0015,submitTime=3D1349691209316,launchTime=3D0,,finishT= ime=3D1349691209

>> <= /o:p>

>>> 

>> 

>&g= t;> 

>>> 352,numMaps= =3D0,numSlotsPerMap=3D1,numReduces=3D0,numSlotsPerReduce=3D1,user=3Dhadoop,= queue=3Ddefault,status=3DFAILED,mapSlotSeconds=3D0,reduceSlotsSeconds=3D0,c= lusterMapCapacity=3D14,clus

>><= o:p> 

>>> 

>> 

>>> terReduceCapacity=3D14,jobName=3Dselect count(1) from poke= s(Stage-1)

>> <= /p>

>>> 

>> 

>>> = 2012-10-08 15:43:29,353 INFO org.apache.hadoop.mapred.JobHistory: Moving

>> 

>>> 

&g= t;>> file:/var/log/hadoop/history/job_201210051717_0015_1349691209316= _hadoop_select+count%281%29+fro

>&= gt; 

>>> 

>> 

>>> m+pokes%28Stage-1%29 to

>> 

>>> 

>>> file:/var/log/hadoo= p/history/done/version-1/NHCLT-PC44-2_1349437647390_/2012/10/08/000000=

>> 

>>> 

>&g= t; 

>>> 2012-10-08 15:4= 3:29,356 INFO org.apache.hadoop.mapred.JobHistory: Moving

>> 

&= gt;>> file:/var/log/hadoop/history/job_201210051717_0015_conf.xml to<= o:p>

>> 

>>> file:/var/log/hadoop/history/don

>> 

>>> 

>>&n= bsp;

>>> e/version-1/NHCLT-PC44-2= _1349437647390_/2012/10/08/000000

>= ;> 

>>> 

>> 

>>> 2012-10-08 15:43:30,362 INFO org.apache.hadoop.mapre= d.JobTracker: Killing

>>&n= bsp;

>>> job job_201210051717_001= 5

>> 

>>> 

= >> 

>>> 2012-10-0= 8 17:17:35,641 INFO

>>&nbs= p;

>>> 

>>> org.apache.hadoop.security.token.delegation.= AbstractDelegationTokenSecretManager:

>> 

>>> Updating= the current master key for generating delegation

>> 

>>&= gt; 

>> 

>>> tokens

>> 

>>><= o:p> 

>> 

>>> 

>> 

>>>&= nbsp;

>> 

>>> 

&= gt;> 

>>> =

>> 

>>> and hive log:-->

>> 

>>> 

>> 

>>> 

>> 

>>>&n= bsp;

>> 

>>> 2012-10-08 15:43:08,597 WARN=A0 conf.Configura= tion

>> 

>>> (Configuration.java:loadResource(1245)) -=

>> 

>>> file:/tmp/hive-default-1260952199191868105.xm= l:a attempt to override final

>>= ; 

>>> parameter: mapre= d.reduce.tasks;=A0 Ignoring.

>>=  

>>> <= /p>

>> 

>>> 2012-10-08 15:43:08,606 WARN=A0 conf.Configuration

>> 

>>> (Configuration.java:loadResource(1245)) -=

>> 

>>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to o= verride final

>> 

>>> parameter: mapred.reduce.tasks;= =A0 Ignoring.

>> 

>>> 

>> 

>>&g= t; 2012-10-08 15:43:08,606 WARN=A0 conf.Configuration

>> 

>&= gt;> (Configuration.java:loadResource(1245)) -

>> 

>>&= gt; file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final

>> 

>>> parameter: mapred.reduce.tasks;=A0 Ignoring.

>> 

>>> 

&g= t;> 

>>> 2012-10-08 = 15:43:09,000 WARN=A0 conf.Configuration

>> 

>>> (Confi= guration.java:loadResource(1245)) -

&= gt;> 

>>> file:/tmp/= hive-default-1260952199191868105.xml:a attempt to override final=

>> 

>>> parameter: mapred.reduce.tasks;=A0 Ignoring.<= /p>

>> 

>>> 

>>=  

>>> 2012-10-08 15:43:09,00= 4 WARN=A0 conf.Configuration

>>=  

>>> (Configuration.ja= va:loadResource(1245)) -

>> 

>>> file:/home/hadoop/Hiv= e/conf/hive-site.xml:a attempt to override final

>> 

>>&g= t; parameter: mapred.reduce.tasks;=A0 Ignoring.

>> 

>>>= ; 

>> 

=

>>> 2012-10-08 15:43:09,004 WARN=A0 conf.C= onfiguration

>> 

>>> (Configuration.java:loadResource(= 1245)) -

>> 

>>> file:/home/hadoop/Hive/conf/hive-site= .xml:a attempt to override final

>= > 

>>> parameter: ma= pred.reduce.tasks;=A0 Ignoring.

>&= gt; 

>>> 

>> 

>>> 2012-10-08 15:43:09,014 WARN=A0 conf.Configuration

>> 

>>> (Configuration.java:loadResource(1245)) -

>> 

>>> file:/tmp/hive-default-1260952199191868105.xml:a atte= mpt to override final

>>&n= bsp;

>>> parameter: mapred.reduce= .tasks;=A0 Ignoring.

>>&nb= sp;

>>> 

>> 

>= ;>> 2012-10-08 15:43:09,018 WARN=A0 conf.Configuration

=

>> 

>>> (Configuration.java:loadResource(1245)) -

>> 

&g= t;>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override = final

>> 

>>> parameter: mapred.reduce.tasks;=A0 Ignor= ing.

>> 

>>> 

>> 

>>> 2012-1= 0-08 15:43:09,018 WARN=A0 conf.Configuration

>> 

>>> (= Configuration.java:loadResource(1245)) -

>> 

>>> file:= /home/hadoop/Hive/conf/hive-site.xml:a attempt to override final=

>> 

>>> parameter: mapred.reduce.tasks;=A0 Ignoring.<= /p>

>> 

>>> 

>>=  

>>> 2012-10-08 15:43:17,30= 9 WARN=A0 conf.Configuration

>>=  

>>> (Configuration.ja= va:loadResource(1245)) -

>> 

>>> file:/tmp/hive-defaul= t-7458429923592584125.xml:a attempt to override final

>> 

>&= gt;> parameter: mapred.reduce.tasks;=A0 Ignoring.

>> 

>&g= t;> 

>> 

>>> 2012-10-08 15:43:17,319 WARN=A0 c= onf.Configuration

>> =

>>> (Configuration.java:loadReso= urce(1245)) -

>> 

>>> file:/home/hadoop/Hive/conf/hive= -site.xml:a attempt to override final

>> 

>>> paramete= r: mapred.reduce.tasks;=A0 Ignoring.

= >> 

>>> = ;

>> 

>>> 2012-10-08 15:43:17,320 WARN=A0 conf.Configuratio= n

>> 

>>> (Configuration.java:loadResource(1245)) -

>> 

>>> file:/home/hadoop/Hive/conf/hive-site.xml:a at= tempt to override final

>>=  

>>> parameter: mapred.redu= ce.tasks;=A0 Ignoring.

>>&= nbsp;

>>> 

>> 

&= gt;>> 2012-10-08 15:43:17,759 WARN=A0 conf.Configuration

>> 

>>> (Configuration.java:loadResource(1245)) -

>> 

= >>> file:/tmp/hive-default-7458429923592584125.xml:a attempt to ov= erride final

>> 

>>> parameter: mapred.reduce.tasks;= =A0 Ignoring.

>> 

>>> 

>> 

>>&g= t; 2012-10-08 15:43:17,766 WARN=A0 conf.Configuration

>> 

>&= gt;> (Configuration.java:loadResource(1245)) -

>> 

>>&= gt; file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override final

>> 

>>> parameter: mapred.reduce.tasks;=A0 Ignoring.

>> 

>>> 

&g= t;> 

>>> 2012-10-08 = 15:43:17,766 WARN=A0 conf.Configuration

>> 

>>> (Confi= guration.java:loadResource(1245)) -

&= gt;> 

>>> file:/home= /hadoop/Hive/conf/hive-site.xml:a attempt to override final

<= p class=3DMsoPlainText>>> 

>>> parameter: mapred.reduce.tasks;=A0 Ignoring.

>> 

= >>> 

>> = ;

>>> 2012-10-08 15:43:17,783 WAR= N=A0 conf.Configuration

>>=  

>>> (Configuration.java:lo= adResource(1245)) -

>>&nbs= p;

>>> file:/tmp/hive-default-745= 8429923592584125.xml:a attempt to override final

>> 

>>&g= t; parameter: mapred.reduce.tasks;=A0 Ignoring.

>> 

>>>= ; 

>> 

=

>>> 2012-10-08 15:43:17,786 WARN=A0 conf.C= onfiguration

>> 

>>> (Configuration.java:loadResource(= 1245)) -

>> 

>>> file:/home/hadoop/Hive/conf/hive-site= .xml:a attempt to override final

>= > 

>>> parameter: ma= pred.reduce.tasks;=A0 Ignoring.

>&= gt; 

>>> 

>> 

>>> 2012-10-08 15:43:17,787 WARN=A0 conf.Configuration

>> 

>>> (Configuration.java:loadResource(1245)) -

>> 

>>> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt t= o override final

>> <= /o:p>

>>> parameter: mapred.reduce.task= s;=A0 Ignoring.

>> 

>>> 

>> 

>&g= t;> 2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin

>> 

>= ;>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core= " requires

>> 

>>> "org.eclipse.core.resourc= es" but it cannot be resolved.

&= gt;> 

>>> =

>> 

>>> 2012-10-08 15:43:24,119 ERROR DataNucleus.Plugin

>> 

>>> (Log4JLogger.java:error(115)) - Bundle "o= rg.eclipse.jdt.core" requires

&g= t;> 

>>> "org.e= clipse.core.resources" but it cannot be resolved.

>> 

>= >> 

>> 

>>> 2012-10-08 15:43:24,119 ERROR D= ataNucleus.Plugin

>> =

>>> (Log4JLogger.java:error(115)= ) - Bundle "org.eclipse.jdt.core" requires

>> 

>&g= t;> "org.eclipse.core.runtime" but it cannot be resolved.=

>> 

>>> 

>&g= t; 

>>> 2012-10-08 15:4= 3:24,119 ERROR DataNucleus.Plugin

>= ;> 

>>> (Log4JLogger= .java:error(115)) - Bundle "org.eclipse.jdt.core" requires

>> 

>>> "org.eclipse.core.runtime" but it cannot b= e resolved.

>> =

>>> 

>> 

>>>= 2012-10-08 15:43:24,120 ERROR DataNucleus.Plugin

>> 

>>&= gt; (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core"= requires

>> 

>>> "org.eclipse.text" but it = cannot be resolved.

>>&nbs= p;

>>> 

>> 

>= >> 2012-10-08 15:43:24,120 ERROR DataNucleus.Plugin

>> 

&= gt;>> (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.co= re" requires

>> =

>>> "org.eclipse.text"= but it cannot be resolved.

>><= o:p> 

>>> 

>> 

>>> 2012-10-08 15:43:24,734 WARN=A0 conf.Configuration

>> 

>>> (Configuration.java:loadResource(1245)) -<= /p>

>> 

>>> file:/tmp/hive-default-7458429923592584125.xml:a attempt = to override final

>> =

>>> parameter: mapred.reduce.tas= ks;=A0 Ignoring.

>> <= /o:p>

>>> 

>> 

>&g= t;> 2012-10-08 15:43:24,750 WARN=A0 conf.Configuration

>> 

&= gt;>> (Configuration.java:loadResource(1245)) -

>> 

>&= gt;> file:/home/hadoop/Hive/conf/hive-site.xml:a attempt to override fin= al

>> 

>>> parameter: mapred.reduce.tasks;=A0 Ignoring= .

>> 

>>> 

= >> 

>>> 2012-10-0= 8 15:43:24,751 WARN=A0 conf.Configuration

>> 

>>> (Con= figuration.java:loadResource(1245)) -

>> 

>>> file:/ho= me/hadoop/Hive/conf/hive-site.xml:a attempt to override final

>> 

>>> parameter: mapred.reduce.tasks;=A0 Ignoring.

=

>> 

>>> 

>>&nb= sp;

>>> 2012-10-08 15:43:28,535 W= ARN=A0 conf.Configuration

>> 

>>> (Configuration.java:= loadResource(1245)) -

>>&n= bsp;

>>> file:/tmp/hive-default-7= 458429923592584125.xml:a attempt to override final

>> 

>&g= t;> parameter: mapred.reduce.tasks;=A0 Ignoring.

>> 

>&g= t;> 

>> 

>>> 2012-10-08 15:43:28,537 WARN=A0 c= onf.Configuration

>> =

>>> (Configuration.java:loadReso= urce(1245)) -

>> 

>>> file:/home/hadoop/Hive/conf/hive= -site.xml:a attempt to override final

>> 

>>> paramete= r: mapred.reduce.tasks;=A0 Ignoring.

= >> 

>>> = ;

>> 

>>> 2012-10-08 15:43:28,537 WARN=A0 conf.Configuratio= n

>> 

>>> (Configuration.java:loadResource(1245)) -

>> 

>>> file:/home/hadoop/Hive/conf/hive-site.xml:a at= tempt to override final

>>=  

>>> parameter: mapred.redu= ce.tasks;=A0 Ignoring.

>>&= nbsp;

>>> 

>> 

&= gt;>> 2012-10-08 15:43:29,113 WARN=A0 mapred.JobClient

=

>> 

>>> (JobClient.java:copyAndConfigureFiles(667)) - Use GenericOpt= ionsParser for

>> 

>>> parsing the arguments. Applicat= ions should implement Tool for the same.

>> 

>>>&= nbsp;

>> 

>>> 2012-10-08 15:43:29,243 WARN=A0 snappy.LoadSn= appy

>> 

>>> (LoadSnappy.java:<clinit>(46)) - Sn= appy native library not loaded

>&g= t; 

>>> 

>> 

>>> 2012-10-08 15:43:30,354 ERROR exec.Task

<= p class=3DMsoPlainText>>>> (SessionState.java:printError(400))

>> 

>>> - Ended Job =3D job_201210051717_0015 with errors=

>> 

>>> 

&= gt;> 

>>> 2012-10-08= 15:43:30,356 ERROR exec.Task

>>= ;> (SessionState.java:printError(400))

>> 

>>> - Er= ror during job, obtaining debugging information...

>> 

>&g= t;> 

>> 

>>> 2012-10-08 15:43:30,369 ERROR ql.= Driver

>>> (SessionState.jav= a:printError(400))

>> = ;

>>> - FAILED: Execution Error, = return code 2 from

>> = ;

>>> org.apache.hadoop.hive.ql.e= xec.MapRedTask

>> 

>>> 

>> 

>>&= gt; 

>> 

>>> 

>> 

>>> 

>> 

>>> 

>> 

>>> Regard= s,

>> 

>>> 

>> 

>>> Ajit

>> 

>>> 

>&= gt; 

>>> 

>> 

>>> 

>> 

>>> -----Original Messa= ge-----

>> 

=

>>> From: Nitin Pawar [mailto:nitinpawar43= 2@gmail.com]

>> 

>>> Sent: Monday, October 08, 2012 4:= 19 PM

>> 

>>> To: user@hive.apache.org

<= p class=3DMsoPlainText>>> 

>>> Subject: Re: hive mapred problem

>> 

>>>=  

>> 

<= p class=3DMsoPlainText>>>> 

>> 

>>>=  

>> 

>>> Can you provide with job logs and failed tas= k logs?

>> 

=

>>> 

>> 

>>> 

>> 

>>> 

>> 

>>>&nbs= p;

>> 

>>> 

&g= t;> 

>>> On Mon, Oct= 8, 2012 at 4:13 PM, Ajit Kumar Shreevastava

>> 

>>> 

>> 

>>> <Ajit.Shreevastava@hcl.com> wrote:<= o:p>

>> 

>>> 

&g= t;> 

>>>> Hi,

>> 

>>> 

>&= gt; 

>>>> =

>> 

>>> 

>&g= t; 

>>>> <= /o:p>

>> 

>>> 

>>= ; 

>>>> 

>> 

>>> 

>>=  

>>>> When I run th= e query "select count(1) from pokes;" it fails with the

>> 

>>>> message

>&= gt; 

>>> 

>> 

>>>> as below.

&g= t;> 

>>> <= /o:p>

>> 

>>>> 

>= ;> 

>>> 

>> 

>>>> 

>= > 

>>> 

>> 

>>>> 

>&= gt; 

>>> 

>> 

>>>> hive> select count(1) from pokes;

>> 

>>> 

>>&n= bsp;

>>>> 

=

>> 

>>> 

>>&nb= sp;

>>>> Total MapReduce jobs = =3D 1

>> 

>>> 

>> 

>>>> 

>> 

>>> 

>> 

>>>> La= unching Job 1 out of 1

>>&= nbsp;

>>> 

>> 

&= gt;>>> 

>>&n= bsp;

>>> 

>> 

&g= t;>>> Number of reduce tasks determined at compile time: 1

>> 

>>> 

>><= o:p> 

>>>> 

>> 

>>> 

>> 

>>>> In order to cha= nge the average load for a reducer (in bytes):

>> 

>>>=  

>> 

<= p class=3DMsoPlainText>>>>> 

>> 

>>><= o:p> 

>> 

>>>>=A0=A0 set hive.exec.reducers.bytes.p= er.reducer=3D<number>

>><= o:p> 

>>> 

>> 

>>>> 

>> 

>>> 

>> 

>>>> In order to limit the maximum number of reducers:<= /o:p>

>> 

>>> 

>>= ; 

>>>> 

>> 

>>> 

>>=  

>>>>=A0=A0 set hiv= e.exec.reducers.max=3D<number>

= >> 

>>> = ;

>> 

>>>> 

&= gt;> 

>>> =

>> 

>>>> In order to set a constant number of reducers:=

>> 

>>> 

&= gt;> 

>>>>&n= bsp;

>> 

>>> 

&g= t;> 

>>>>=A0=A0 s= et mapred.reduce.tasks=3D<number>

>> 

>>>&n= bsp;

>> 

>>>> 

>> 

>>>&nb= sp;

>> 

>>>> Starting Job =3D job_201210051717_0015, Tr= acking URL =3D

>> 

>>> 

>> 

>>&= gt;> http://NHCLT-PC44-2:50030/jobdetails.jsp?jobid=3Djob_201210051717_0= 015

>> 

>>> 

>> 

>>>> 

>> 

>>> 

>> 

>>>> Kill= Command =3D /home/hadoop/hadoop-1.0.3/bin/hadoop job=A0 -kill

>> 

>>> 

>>&= nbsp;

>>>> job_201210051717_00= 15

>> 

>>> 

>> 

>>>>=  

>> 

>>> 

= >> 

>>>> Hadoo= p job information for Stage-1: number of mappers: 0; number of

>> 

>>> 

>>&= nbsp;

>>>> reducers: 0

>> 

>>> 

>><= o:p> 

>>>> 

>> 

>>> 

>> 

>>>> 2012-10-08 15:4= 3:30,351 Stage-1 map =3D 100%,=A0 reduce =3D 100%

>> 

>>&= gt; 

>> 

>>>> 

>> 

>>&g= t; 

>> 

>>>> Ended Job =3D job_201210051717_00= 15 with errors

>> 

>>> 

>> 

>>&= gt;> 

>> 

>>> 

>> 

>>&g= t;> Error during job, obtaining debugging information...

<= p class=3DMsoPlainText>>> 

>>> 

>>&nbs= p;

>>>> 

>> 

= >>> 

>> = ;

>>>> FAILED: Execution Error= , return code 2 from

>>&nb= sp;

>>> 

>> 

>= ;>>> org.apache.hadoop.hive.ql.exec.MapRedTask

>> 

>= ;>> 

>> 

>>>> 

>> 

>= >> 

>> 

>>>> MapReduce Jobs Launched:

>> 

>>> 

&g= t;> 

>>>>&nb= sp;

>> 

>>> 

&g= t;> 

>>>> Job 0:= =A0 HDFS Read: 0 HDFS Write: 0 FAIL

&= gt;> 

>>> =

>> 

>>>> 

&g= t;> 

>>> <= /o:p>

>> 

>>>> Total MapReduce CPU Time Spent: 0 msec

>> 

>>> 

>> 

>>>> 

>> 

>>> 

>> 

>>>> =

>> 

>>> 

>> 

>>>> <= /p>

>> 

>>> 

>>=  

>>>> Thanks and Regards=

>> 

>>> 

&= gt;> 

>>>>&n= bsp;

>> 

>>> 

&g= t;> 

>>>> Ajit Ku= mar Shreevastava

>> <= /o:p>

>>> 

>> 

>&g= t;>> 

>> <= /o:p>

>>> 

>> 

>&g= t;>> ADCOE (App Development Center Of Excellence )

>> 

&g= t;>> 

>> <= /o:p>

>>>> 

>> 

>= ;>> 

>> 

>>>> Mobile: 9717775634

>> 

>>> 

>><= o:p> 

>>>> 

>> 

>>> 

>> 

>>>> 

>> 

>>> 

>> 

>>>> =

>> 

>>> 

>> 

>>>> <= /p>

>> 

>>> 

>>=  

>>>> 

>> 

>>> 

>>&= nbsp;

>>>> ::DISCLAIMER::=

>> 

>>> 

>&g= t; 

>>>> <= /o:p>

>> 

>>>> 

>= ;>>> -------------------------------------------------------------= ---------------------------------------------------------------------------= ------------

>> 

>>> 

>> 

>>>= ;> 

>> =

>>> 

>> 

>>>= > The contents of this e-mail and any attachment(s) are confidential and=

>> 

>>> 

&= gt;> 

>>>> intend= ed for the named recipient(s) only.

&= gt;> 

>>> =

>> 

>>>> E-mail transmission is not guaranteed to be se= cure or error-free as

>>&n= bsp;

>>> 

>> 

&g= t;>>> information could be intercepted, corrupted,

<= p class=3DMsoPlainText>>> 

>>> 

>>&nbs= p;

>>>> lost, destroyed, arriv= e late or incomplete, or may contain viruses in

>> 

>>>= ; 

>> 

=

>>>> transmission. The e mail and its c= ontents

>> 

=

>>> 

>> 

>>>>= (with or without referred errors) shall therefore not attach any

>>>> liability

>> 

>= ;>> 

>> 

>>>> on the originator or HCL o= r its affiliates.

>> =

>>> 

>> 

>&g= t;>> Views or opinions, if any, presented in this email are solely th= ose of

>>>> the

>> 

>>> 

>> 

>>>> author and may n= ot necessarily reflect the

>> 

>>> 

>> 

>>>> views or opinions of HCL or its affiliates. Any form of= reproduction,

>> 

>>> 

>> 

>>&= gt;> dissemination, copying, disclosure, modification,

>> 

&= gt;>> 

>> =

>>>> distribution and / or pu= blication of this message without the prior

>> 

>>>>= ; written

>> 

>>> 

>> 

>>>&g= t; consent of authorized representative of

>> 

>>> 

>> 

>>>> HCL is strictly prohibited. If you have= received this email in error

>>= ; 

>>>> please<= /o:p>

>> 

>>> 

>>= ; 

>>>> delete it an= d notify the sender immediately.

>= > 

>>> 

>> 

>>>> Before opening any email and/or attachments, plea= se check them for

>>>> vi= ruses

>> 

>>> 

>> 

>>>> a= nd other defects.

>> =

>>> 

>> 

>&g= t;>> 

>> <= /o:p>

>>> 

>> 

>&g= t;>> 

>> <= /o:p>

>>>> 

>>>> ---------------------------------------= ---------------------------------------------------------------------------= ----------------------------------

&g= t;> 

>>> <= /o:p>

>> 

>>> 

>>= ; 

>>> =

>> 

>>> 

>> 

>>> 

<= p class=3DMsoPlainText>>> 

>>> 

>>&nbs= p;

>>> 

>> 

>= >> --

>> =

>>> 

>> 

>>>= Nitin Pawar

>> 

>> 

>> 

>>&n= bsp;

>> 

>> 

>&g= t; 

>> --

>> 

= >> Nitin Pawar

> <= /o:p>

> 

> 

> --<= /p>

> Nitin Pawar

 

 

 

-- =

Nitin Pawar

= --_000_F0232037B611204D9A7090EC62CF180B06A7979C3Dndahcltevs06H_--