Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C8639200BBB for ; Thu, 27 Oct 2016 00:35:15 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id C70C2160B02; Wed, 26 Oct 2016 22:35:15 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5084E160AEE for ; Thu, 27 Oct 2016 00:35:14 +0200 (CEST) Received: (qmail 1998 invoked by uid 500); 26 Oct 2016 22:35:13 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 1988 invoked by uid 99); 26 Oct 2016 22:35:13 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Oct 2016 22:35:13 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 8EC07C0DE9 for ; Wed, 26 Oct 2016 22:35:12 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.779 X-Spam-Level: ** X-Spam-Status: No, score=2.779 tagged_above=-999 required=6.31 tests=[FSL_HELO_BARE_IP_2=1.499, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id mElXdfyn0s1N for ; Wed, 26 Oct 2016 22:35:07 +0000 (UTC) Received: from relayvx12b.securemail.intermedia.net (relayvx12b.securemail.intermedia.net [64.78.52.186]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id EFABD5FAF9 for ; Wed, 26 Oct 2016 22:35:05 +0000 (UTC) Received: from emg-ca-1-2.securemail.intermedia.net (localhost [127.0.0.1]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by emg-ca-1-2.localdomain (Postfix) with ESMTPS id B547D53EA8 for ; Wed, 26 Oct 2016 15:34:57 -0700 (PDT) Subject: Re: Error with flush_length File in Orc, in hive 2.1.0 and mr execution engine. MIME-Version: 1.0 x-echoworx-msg-id: 0505cdb8-9103-4c48-9a74-8cb472ad62f0 x-echoworx-emg-received: Wed, 26 Oct 2016 15:34:57.679 -0700 x-echoworx-message-code-hashed: c61deda3a4eae052d26a04f14643f98c924fcddd8e7fc4d0f0fc6fbd84fe7e09 x-echoworx-action: delivered Received: from 10.254.155.16 ([10.254.155.16]) by emg-ca-1-2 (JAMES SMTP Server 2.3.2) with SMTP ID 632 for ; Wed, 26 Oct 2016 15:34:57 -0700 (PDT) Received: from MBX080-W3-CO-1.exch080.serverpod.net (unknown [10.224.117.52]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by emg-ca-1-2.localdomain (Postfix) with ESMTPS id 624AE53EBB for ; Wed, 26 Oct 2016 15:34:57 -0700 (PDT) Received: from MBX080-W3-CO-1.exch080.serverpod.net (10.224.117.52) by MBX080-W3-CO-1.exch080.serverpod.net (10.224.117.52) with Microsoft SMTP Server (TLS) id 15.0.1178.4; Wed, 26 Oct 2016 15:34:56 -0700 Received: from MBX080-W3-CO-1.exch080.serverpod.net ([10.224.117.52]) by MBX080-W3-CO-1.exch080.serverpod.net ([169.254.1.215]) with mapi id 15.00.1178.000; Wed, 26 Oct 2016 15:34:56 -0700 From: Eugene Koifman To: "user@hive.apache.org" Thread-Topic: Error with flush_length File in Orc, in hive 2.1.0 and mr execution engine. Thread-Index: AQHSLyLMSXEYtOwQqE+Bd4r5MC2fLKC6GbsAgAGZLwD//6EoAA== Date: Wed, 26 Oct 2016 22:34:56 +0000 Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [192.175.27.10] x-source-routing-agent: Processed Content-Type: multipart/alternative; boundary="_000_D4367D0A8380Eekoifmanhortonworkscom_" archived-at: Wed, 26 Oct 2016 22:35:16 -0000 --_000_D4367D0A8380Eekoifmanhortonworkscom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable If you can run this, then it=92s safe to ignore =9300094_flush_length=94 me= ssages and the issue is somewhere else select count(*) from mls_public_record_association_snapshot_orc pra left o= uter join mls_listing_snapshot_orc ml on pra.primary_listing_id =3D ml.id left outer join attribute a on a.id =3D ml.sta= ndard_status Eugene From: satyajit vegesna > Date: Wednesday, October 26, 2016 at 2:14 PM To: "user@hive.apache.org" >, Eugene Koifman > Cc: "dev@hive.apache.org" > Subject: Re: Error with flush_length File in Orc, in hive 2.1.0 and mr exec= ution engine. Hi Eugene, PFB Transaction table in green and parquet tables in yellow, INSERT INTO access_logs.crawlstats_dpp PARTITION(day=3D"2016-10-23") select= pra.url as prUrl,pra.url_type as urlType,CAST(pra.created_at AS timestamp)= as prCreated, CAST(pra.updated_at AS timestamp) as prUpdated, CAST(ml.crea= ted_at AS timestamp) as mlCreated, CAST(ml.updated_at AS timestamp) as mlUp= dated, a.name as status, pra.public_record_id as prId, acl.= accesstime as crawledon, pra.id as propId, pra.primary_list= ing_id as listingId, datediff(CAST(acl.accesstime AS timestamp),CAST(ml.cre= ated_at AS timestamp)) as mlcreateage, datediff(CAST(acl.accesstime AS time= stamp),CAST(ml.updated_at AS timestamp)) as mlupdateage, datediff(CAST(acl.= accesstime AS timestamp),CAST(pra.created_at AS timestamp)) as prcreateage,= datediff(CAST(acl.accesstime AS timestamp),CAST(pra.updated_at AS timestam= p)) as prupdateage, (case when (pra.public_record_id is not null and TRIM(= pra.public_record_id) <> '') then (case when (pra.primary_listing_id is nu= ll or TRIM(pra.primary_listing_id) =3D '') then 'PR' else 'PRMLS' END) els= e (case when (pra.primary_listing_id is not null and TRIM(pra.primary_listi= ng_id) <> '') then 'MLS' else 'UNKNOWN' END) END) as listingType, acl.http= statuscode, acl.httpverb, acl.requesttime, acl.upstreamheadertime , acl.u= pstreamresponsetime, acl.page_id, useragent AS user_agent, substring(spl= it(pra.url,'/')[0], 0,length(split(pra.url,'/')[0])-3) as city, substring(= split(pra.url,'/')[0], length(split(pra.url,'/')[0])-1,2) as state, ml.mls= _id FROM access_logs.loadbalancer_accesslogs acl inner join mls_public_re= cord_association_snapshot_orc pra on acl.listing_url =3D pra.url left oute= r join mls_listing_snapshot_orc ml on pra.primary_listing_id =3D ml.id left outer join attribute a on a.id =3D ml.standa= rd_status WHERE acl.accesstimedate=3D"2016-10-23"; Any clue, or something that you would want me to focus on to debug the issu= e. Regards, Satyajit. On Tue, Oct 25, 2016 at 8:49 PM, Eugene Koifman > wrote: Which of your tables are are transactional? Can you provide the DDL? I don=92t think =93File does not exist=94 error is causing your queries to = fail. It=92s an INFO level msg. There should be some other error. Eugene From: satyajit vegesna > Reply-To: "user@hive.apache.org" > Date: Tuesday, October 25, 2016 at 5:46 PM To: "user@hive.apache.org" >, "dev@hive.apache.org" > Subject: Error with flush_length File in Orc, in hive 2.1.0 and mr executio= n engine. HI All, i am using hive 2.1.0 , hadoop 2.7.2 , but when i try running queries like= simple insert, set mapreduce.job.queuename=3Ddefault;set hive.exec.dynamic.partition=3Dtru= e;set hive.exec.dynamic.partition.mode=3Dnonstrict;set hive.exec.max.dynami= c.partitions.pernode=3D400;set hive.exec.max.dynamic.partitions=3D2000;set = mapreduce.map.memory.mb=3D5120;set mapreduce.reduce.memory.mb=3D5120;set ma= pred.tasktracker.map.tasks.maximum=3D30;set mapred.tasktracker.reduce.tasks= .maximum=3D20;set mapred.reduce.child.java.opts=3D-Xmx2048m;set mapred.map.= child.java.opts=3D-Xmx2048m; set hive.support.concurrency=3Dtrue; set hive.= txn.manager=3Dorg.apache.hadoop.hive.ql.lockmgr.DbTxnManager; set hive.comp= actor.initiator.on=3Dfalse; set hive.compactor.worker.threads=3D1;set mapre= duce.job.queuename=3Ddefault;set hive.exec.dynamic.partition=3Dtrue;set hiv= e.exec.dynamic.partition.mode=3Dnonstrict;INSERT INTO access_logs.crawlstat= s_dpp PARTITION(day=3D"2016-10-23") select pra.url as prUrl,pra.url_type as= urlType,CAST(pra.created_at AS timestamp) as prCreated, CAST(pra.updated_a= t AS timestamp) as prUpdated, CAST(ml.created_at AS timestamp) as mlCreated= , CAST(ml.updated_at AS timestamp) as mlUpdated, a.name as s= tatus, pra.public_record_id as prId, acl.accesstime as crawledon, pra.id as propId, pra.primary_listing_id as listingId, datediff(CAST(= acl.accesstime AS timestamp),CAST(ml.created_at AS timestamp)) as mlcreatea= ge, datediff(CAST(acl.accesstime AS timestamp),CAST(ml.updated_at AS timest= amp)) as mlupdateage, datediff(CAST(acl.accesstime AS timestamp),CAST(pra.c= reated_at AS timestamp)) as prcreateage, datediff(CAST(acl.accesstime AS ti= mestamp),CAST(pra.updated_at AS timestamp)) as prupdateage, (case when (pr= a.public_record_id is not null and TRIM(pra.public_record_id) <> '') then = (case when (pra.primary_listing_id is null or TRIM(pra.primary_listing_id) = =3D '') then 'PR' else 'PRMLS' END) else (case when (pra.primary_listing_i= d is not null and TRIM(pra.primary_listing_id) <> '') then 'MLS' else 'UNKN= OWN' END) END) as listingType, acl.httpstatuscode, acl.httpverb, acl.req= uesttime, acl.upstreamheadertime , acl.upstreamresponsetime, acl.page_id, = useragent AS user_agent, substring(split(pra.url,'/')[0], 0,length(split(= pra.url,'/')[0])-3) as city, substring(split(pra.url,'/')[0], length(split= (pra.url,'/')[0])-1,2) as state, ml.mls_id FROM access_logs.loadbalancer_= accesslogs acl inner join mls_public_record_association_snapshot_orc pra o= n acl.listing_url =3D pra.url left outer join mls_listing_snapshot_orc ml = on pra.primary_listing_id =3D ml.id left outer join attribut= e a on a.id =3D ml.standard_status WHERE acl.accesstimedate= =3D"2016-10-23"; i finally end up getting below error, 2016-10-25 17:40:18,725 Stage-2 map =3D 100%, reduce =3D 52%, Cumulative C= PU 1478.96 sec 2016-10-25 17:40:19,761 Stage-2 map =3D 100%, reduce =3D 62%, Cumulative C= PU 1636.58 sec 2016-10-25 17:40:20,794 Stage-2 map =3D 100%, reduce =3D 64%, Cumulative C= PU 1764.97 sec 2016-10-25 17:40:21,820 Stage-2 map =3D 100%, reduce =3D 69%, Cumulative C= PU 1879.61 sec 2016-10-25 17:40:22,842 Stage-2 map =3D 100%, reduce =3D 80%, Cumulative C= PU 2051.38 sec 2016-10-25 17:40:23,872 Stage-2 map =3D 100%, reduce =3D 90%, Cumulative C= PU 2151.49 sec 2016-10-25 17:40:24,907 Stage-2 map =3D 100%, reduce =3D 93%, Cumulative C= PU 2179.67 sec 2016-10-25 17:40:25,944 Stage-2 map =3D 100%, reduce =3D 94%, Cumulative C= PU 2187.86 sec 2016-10-25 17:40:29,062 Stage-2 map =3D 100%, reduce =3D 95%, Cumulative C= PU 2205.22 sec 2016-10-25 17:40:30,107 Stage-2 map =3D 100%, reduce =3D 100%, Cumulative = CPU 2241.25 sec MapReduce Total cumulative CPU time: 37 minutes 21 seconds 250 msec Ended Job =3D job_1477437520637_0009 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-hive-2.1.0-bin/lib/log4j-slf4= j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.2/share/hadoop/common/lib= /slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explana= tion. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFacto= ry] 2016-10-25 17:40:35Starting to launch local task to process map join;maximu= m memory =3D 514850816 Execution failed with exit status: 2 Obtaining error information Task failed! Task ID: Stage-14 Logs: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.= mr.MapredLocalTask MapReduce Jobs Launched: Stage-Stage-1: Map: 106 Reduce: 45 Cumulative CPU: 3390.11 sec HDFS Re= ad: 8060555201 HDFS Write: 757253756 SUCCESS Stage-Stage-2: Map: 204 Reduce: 85 Cumulative CPU: 2241.25 sec HDFS Re= ad: 2407914653 HDFS Write: 805874953 SUCCESS Total MapReduce CPU Time Spent: 0 days 1 hours 33 minutes 51 seconds 360 ms= ec Could not find any errors in logs, but when i check namenode logs , oi get = the following error, 2016-10-25 17:01:51,923 INFO org.apache.hadoop.ipc.Server: IPC Server handl= er 1 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockL= ocations from 192.168.120.133:47114 Call#4 Re= try#0: java.io.FileNotFoundException: File does not exist: /user/hive/wareh= ouse/mls_public_record_association_snapshot_orc/delta_0000002_0000002_0000/= bucket_00094_flush_length 2016-10-25 17:01:52,779 INFO org.apache.hadoop.ipc.Server: IPC Server handl= er 1 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockL= ocations from 192.168.120.132:43008 Call#4 Re= try#0: java.io.FileNotFoundException: File does not exist: /user/hive/wareh= ouse/mls_public_record_association_snapshot_orc/delta_0000002_0000002_0000/= bucket_00095_flush_length 2016-10-25 17:01:52,984 INFO org.apache.hadoop.ipc.Server: IPC Server handl= er 0 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockL= ocations from 192.168.120.133:47260 Call#4 Re= try#0: java.io.FileNotFoundException: File does not exist: /user/hive/wareh= ouse/mls_public_record_association_snapshot_orc/delta_0000002_0000002_0000/= bucket_00096_flush_length 2016-10-25 17:01:53,381 INFO org.apache.hadoop.ipc.Server: IPC Server handl= er 0 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockL= ocations from 192.168.120.132:43090 Call#4 Re= try#0: java.io.FileNotFoundException: File does not exist: /user/hive/wareh= ouse/mls_public_record_association_snapshot_orc/delta_0000002_0000002_0000/= bucket_00097_flush_length 2016-10-25 17:01:53,971 INFO org.apache.hadoop.ipc.Server: IPC Server handl= er 1 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockL= ocations from 192.168.120.134:37444 Call#4 Re= try#0: java.io.FileNotFoundException: File does not exist: /user/hive/wareh= ouse/mls_public_record_association_snapshot_orc/delta_0000002_0000002_0000/= bucket_00098_flush_length 2016-10-25 17:01:54,092 INFO org.apache.hadoop.ipc.Server: IPC Server handl= er 2 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockL= ocations from 192.168.120.133:47300 Call#4 Re= try#0: java.io.FileNotFoundException: File does not exist: /user/hive/wareh= ouse/mls_public_record_association_snapshot_orc/delta_0000002_0000002_0000/= bucket_00099_flush_length 2016-10-25 17:01:55,094 INFO org.apache.hadoop.ipc.Server: IPC Server handl= er 8 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockL= ocations from 192.168.120.134:37540 Call#4 Re= try#0: java.io.FileNotFoundException: File does not exist: /user/hive/wareh= ouse/mls_public_record_association_snapshot_orc/delta_0000002_0000002_0000/= bucket_00012_flush_length 2016-10-25 17:02:11,269 INFO org.apache.hadoop.ipc.Server: IPC Server handl= er 5 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockL= ocations from 192.168.120.133:47378 Call#4 Re= try#0: java.io.FileNotFoundException: File does not exist: /user/hive/wareh= ouse/mls_public_record_association_snapshot_orc/delta_0000002_0000002_0000/= bucket_00075_flush_length i also search for find the flush_length files in the above mentioned locati= on, but i only see buckets but no files ending with flush_length. Any clue or help would be highly appreciated. Regards, Satyajit. --_000_D4367D0A8380Eekoifmanhortonworkscom_ Content-Type: text/html; charset="Windows-1252" Content-ID: <5649EA7B75DB4D42B260E36CEACAD258@exch080.serverpod.net> Content-Transfer-Encoding: quoted-printable
I= f you can run this, then it=92s safe to ignore =9300094_flush_length=94 messages and the issue is somewhere else

select count(*) from mls_public_record_association_snapshot_orc pra  left outer join mls_listing_snapshot_orc ml on pra.primary_listing_id =3D=  ml.id  left out= er join attribute a on a.id = =3D ml.standard_status

Eugene

From: satyajit vegesna <satyajit.apasprk@gmail.com> Date: Wednesday, October 26, 2016 a= t 2:14 PM
To: "user@hive.apache.org" <user@hive.apache.org>, Eugene Koifman <ekoifman@hortonworks.com>
Cc: "dev@hive.apache.org" <dev@hive.apache.org>
Subject: Re: Error with flush_lengt= h File in Orc, in hive 2.1.0 and mr execution engine.

Hi Eugene,

PFB Transaction table in green and parquet tables in yell= ow,

INSERT INTO access_logs.crawlstats_dpp PARTITION(day=3D"2016-10-23") s= elect pra.url as prUrl,pra.url_type as urlType,CAST(pra.created_at AS times= tamp) as prCreated, CAST(pra.updated_at AS timestamp) as prUpdated, CAST(ml= .created_at AS timestamp) as mlCreated, CAST(ml.updated_at AS timestamp) as mlUpdated, a.name as status, pra.public_record_id as prId, acl.accesstime as cra= wledon, pra.id as propId, pra.primary_listing_id as listingId, datediff(CAST(acl.accesstime = AS timestamp),CAST(ml.created_at AS timestamp)) as mlcreateage, datediff(CA= ST(acl.accesstime AS timestamp),CAST(ml.updated_at AS timestamp)) as mlupda= teage, datediff(CAST(acl.accesstime AS timestamp),CAST(pra.created_at AS timestamp)) as prcreateage, datediff(= CAST(acl.accesstime AS timestamp),CAST(pra.updated_at AS timestamp)) as pru= pdateage,  (case when (pra.public_record_id is not null and TRIM(pra.p= ublic_record_id) <> '')  then (case when (pra.primary_listing_id is null or TRIM(pra.primary_listing_id) =3D '') th= en 'PR' else 'PRMLS' END)  else (case when (pra.primary_listing_id is = not null and TRIM(pra.primary_listing_id) <> '') then 'MLS' else 'UNK= NOWN' END) END) as listingType,  acl.httpstatuscode,  acl.httpverb,  acl.requesttime, acl.upstreamheadertime , acl.up= streamresponsetime,  acl.page_id,  useragent AS user_agent,  = ;substring(split(pra.url,'/')[0], 0,length(split(pra.url,'/')[0])-3) as city,  substring(split(pra.url,'/')[0], length(split(pra.url,'/')[0])= -1,= 2) as state,  ml.mls_id  FROM access_logs.loadbalancer_= accesslogs acl  inner join mls_p= ublic_record_association_snapshot_orc pra on acl.listing_url =3D pra.url  left outer join mls_listing_snapshot_orc ml on pra.primary_listing_id =3D ml= .id  left outer join attribute a on = ;a.id =3D ml.standard_status &n= bsp;WHERE acl.accesstimedate=3D"2016-10-23";


Any clue, or something that you would= want me to focus on to debug the issue.

Regards,
Satyajit.



On Tue, Oct 25, 2016 at 8:49 PM, Eugene Koifman = <ekoifman@= hortonworks.com> wrote:
Which of your tables are are transactional?  Can you provide the = DDL?

I don=92t think =93File does not exist=94 error is causing your querie= s to fail.  It=92s an INFO level msg.
There should be some other error.

Eugene


From: satyajit vegesna <satyajit.apasprk@g= mail.com>
Reply-To: "user@hive.apache.org" <<= a href=3D"mailto:user@hive.apache.org" target=3D"_blank">user@hive.apache.o= rg>
Date: Tuesday, October 25, 2016 at = 5:46 PM
To: "user@hive.apache.org" <user@hive.apache.org= >, "dev@hi= ve.apache.org" <dev@hive.apac= he.org>
Subject: Error with flush_length Fi= le in Orc, in hive 2.1.0 and mr execution engine.

HI All,

i am using hive 2.1.0 , hadoop 2.7.2 , but  when i try running qu= eries like simple insert,

set mapreduce.job.queuename=3Ddefault;set hive.exec.dynamic.parti= tion=3Dtrue;set hive.exec.dynamic.partition.mode=3Dnonstrict;set = hive.exec.max.dynamic.partitions.pernode=3D400;set hive.exec.max.dynam= ic.partitions=3D2000;set mapreduce.map.memory.mb=3D5120;set mapreduce.reduce.memory.mb=3D5120;set mapred.tasktracker.map.tasks.maximum=3D30;set mapred.tasktracker.reduce.tasks.maximum=3D20;set m= apred.reduce.child.java.opts=3D-Xmx2048m;set mapred.map.child.java.opt= s=3D-Xmx2048m; set hive.support.concurrency=3Dtrue; set hive.txn.manager=3Dorg.apache.hadoop.hive.ql.lockmgr.DbTxnMa= nager; set hive.compactor.initiator.on=3Dfalse; set hive.compactor.wor= ker.threads=3D1;set mapreduce.job.queuename=3Ddefault;set hive.ex= ec.dynamic.partition=3Dtrue;set hive.exec.dynamic.partition.mode= =3Dnonstrict;INSERT INTO access_logs.crawlstats_dpp PARTITION(day=3D"2016-10-23") se= lect pra.url as prUrl,pra.url_type as urlType,CAST(pra.created_at AS timest= amp) as prCreated, CAST(pra.updated_at AS timestamp) as prUpdated, CAST(ml.= created_at AS timestamp) as mlCreated, CAST(ml.updated_at AS timestamp) as mlUpdated, a.= name as status, pra.public_record_id as prId, acl.accesstime as crawled= on, pra.id as propId, pra.prima= ry_listing_id as listingId, datediff(CAST(acl.accesstime AS timestamp),CAST= (ml.created_at AS timestamp)) as mlcreateage, datediff(CAST(acl.accesstime = AS timestamp),CAST(ml.updated_at AS timestamp)) as mlupdateage, datediff(CAST(acl.accesstime AS timestamp),CAS= T(pra.created_at AS timestamp)) as prcreateage, datediff(CAST(acl.accesstim= e AS timestamp),CAST(pra.updated_at AS timestamp)) as prupdateage,  (c= ase when (pra.public_record_id is not null and TRIM(pra.public_record_id) <> '')  then (case when (pr= a.primary_listing_id is null or TRIM(pra.primary_listing_id) =3D '') then '= PR' else 'PRMLS' END)  else (case when (pra.primary_listing_id is not = null and TRIM(pra.primary_listing_id) <> '') then 'MLS' else 'UNKNOWN' END) END) as listingType,  acl.httpstatuscode, &= nbsp;acl.httpverb,  acl.requesttime, acl.upstreamheadertime , acl.upst= reamresponsetime,  acl.page_id,  useragent AS user_agent,  s= ubstring(split(pra.url,'/')[0], 0,length(split(pra.url,'/')[0])-3= ) as city,  substring(split(pra.url,'/')[0], length(split(pra.url,= '/')[0])-1,2) as state,  ml.mls_id  FROM access_logs.loadbal= ancer_accesslogs acl  inner join mls_public_record_association_snapshot_orc pra on acl.listing_url =3D pra.url  left outer join mls_listing_snapshot_orc ml on pra.primary_listing_id =3D ml.id  left outer join attribute a on a.id =3D ml.standard_status  WHERE acl.accesstimedat= e=3D"2016-10-23";

i finally end up getting below error,
2016-10-25 17:40:18,725 Stage-2 map =3D 100%,  reduce =3D 52%, Cu= mulative CPU 1478.96 sec
2016-10-25 17:40:19,761 Stage-2 map =3D 100%,  reduce =3D 62%, Cu= mulative CPU 1636.58 sec
2016-10-25 17:40:20,794 Stage-2 map =3D 100%,  reduce =3D 64%, Cu= mulative CPU 1764.97 sec
2016-10-25 17:40:21,820 Stage-2 map =3D 100%,  reduce =3D 69%, Cu= mulative CPU 1879.61 sec
2016-10-25 17:40:22,842 Stage-2 map =3D 100%,  reduce =3D 80%, Cu= mulative CPU 2051.38 sec
2016-10-25 17:40:23,872 Stage-2 map =3D 100%,  reduce =3D 90%, Cu= mulative CPU 2151.49 sec
2016-10-25 17:40:24,907 Stage-2 map =3D 100%,  reduce =3D 93%, Cu= mulative CPU 2179.67 sec
2016-10-25 17:40:25,944 Stage-2 map =3D 100%,  reduce =3D 94%, Cu= mulative CPU 2187.86 sec
2016-10-25 17:40:29,062 Stage-2 map =3D 100%,  reduce =3D 95%, Cu= mulative CPU 2205.22 sec
2016-10-25 17:40:30,107 Stage-2 map =3D 100%,  reduce =3D 100%, C= umulative CPU 2241.25 sec
MapReduce Total cumulative CPU time: 37 minutes 21 seconds 250 msec
Ended Job =3D job_1477437520637_0009
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-2.1.0-bin/lib/= log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.cla= ss]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.2/share/hadoop/= common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerB= inder.class]
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jL= oggerFactory]
2016-10-25 17:40:35Starting to launch local task = to process map join;maximum memory =3D 514850816
Execution failed with exit status: 2
Obtaining error information

Task failed!
Task ID:
  Stage-14

Logs:

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.= exec.mr.MapredLocalTask
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 106  Reduce: 45   Cumulative CPU: 3390.1= 1 sec   HDFS Read: 8060555201 HDFS Write: 757253756 SUCCESS
Stage-Stage-2: Map: 204  Reduce: 85   Cumulative CPU: 2241.2= 5 sec   HDFS Read: 240= 7914653 HDFS Write: 805874953 SUCCESS
Total MapReduce CPU Time Spent: 0 days 1 hours 33 minutes 51 seconds 3= 60 msec

Could not find any errors in logs, but when i check namenode logs , oi= get the following error,

2016-10-25 17:01:51,923 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 1 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.133:= 47114 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00094_flush_length
2016-10-25 17:01:52,779 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 1 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.132:= 43008 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00095_flush_length
2016-10-25 17:01:52,984 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 0 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.133:= 47260 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00096_flush_length
2016-10-25 17:01:53,381 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 0 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.132:= 43090 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00097_flush_length
2016-10-25 17:01:53,971 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 1 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.134:= 37444 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00098_flush_length
2016-10-25 17:01:54,092 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 2 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.133:= 47300 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00099_flush_length
2016-10-25 17:01:55,094 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 8 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.134:= 37540 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00012_flush_length
2016-10-25 17:02:11,269 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 5 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.133:= 47378 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00075_flush_length

i also search for find the flush_length files in the above mentioned l= ocation, but i only see buckets but no files ending with flush_length.

Any clue or help would be highly appreciated.

Regards,
Satyajit.


 --_000_D4367D0A8380Eekoifmanhortonworkscom_--