Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0E225200BBB for ; Thu, 27 Oct 2016 01:26:52 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 0CA69160AFD; Wed, 26 Oct 2016 23:26:52 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0E6A2160AEE for ; Thu, 27 Oct 2016 01:26:49 +0200 (CEST) Received: (qmail 22126 invoked by uid 500); 26 Oct 2016 23:26:48 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 22107 invoked by uid 99); 26 Oct 2016 23:26:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Oct 2016 23:26:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 24E6CC0D64; Wed, 26 Oct 2016 23:26:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.38 X-Spam-Level: ** X-Spam-Status: No, score=2.38 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 5BqQqBPs3wrx; Wed, 26 Oct 2016 23:26:41 +0000 (UTC) Received: from mail-qt0-f171.google.com (mail-qt0-f171.google.com [209.85.216.171]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 61BDA5FC5D; Wed, 26 Oct 2016 23:26:41 +0000 (UTC) Received: by mail-qt0-f171.google.com with SMTP id g32so12041238qta.2; Wed, 26 Oct 2016 16:26:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=pubKhqaiLiMS25hqt2JpXFdZ+GbXAFUPdd9/EJDtgUs=; b=ZS/KYrECbIsDejRfb0yLIP5z0OoT2+MKVqDqPHL3SxsTDDO9w9YaniS6WUhmRIFQNU 3Y/uR4dZXT8Gs7mGl784VALs6P6P9A09sqSY7+3I6k0fASw51jPvNH0nqlgn19AdkH4D XccuGMOjdfIV+rqgitzPvvCeep9Yu8fmGmBZrTOTBNESd8wSI4GEn6r0Vvak9wuVWsFc MgnkL9kPP9Pbv/QTfkAVUkabhkHZCpTA5GkY40vCXkig7PP4X08rR7+EFrehMlgDxV0i vVwgVrNjh7YhPo3nLisYY11E4vhIyaHVyInA7EiewEBdphShfx1/Z9f270mauFKwLwjn FmxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=pubKhqaiLiMS25hqt2JpXFdZ+GbXAFUPdd9/EJDtgUs=; b=cGtT3Pvma7VT+au7cqnC0iXKLYX4qCZbARY1rvIykWSQBuV34pj//VaSwz8fLf5Zn/ 5cvpiIfaKekWDxLJ2pwCVnB5aLp//eTE0evcJ37a6eZeR5pzu+TM/ByRi43Hfl2gpy8X 1CpZbG4m3QKOQoHiz64ksSpTvQASuoWlbFCuQyOO3IU2BnFe99lVuAo6NTCtJAn9lfrg vA5g/oDnBal06KPoFBwU1PSO1m7bQOayrEFrIrD+Nj/9zOouape3DKWWNFy7kr0Dmp0X 3sT16oS0726lJATLEwI9ORdA98LASmPJqQ7cPaOzPE7313KFdxiJFG9x7/xxTPnD5+pF Ao8Q== X-Gm-Message-State: ABUngvcEtbATtCWwYcdtMMiptIJpBZDveGc3hvuuA0Lmt7hA20OLGpnbiKs5MQfSEXfKRKz1QK/TxxIKO8uRsg== X-Received: by 10.200.48.146 with SMTP id v18mr3737100qta.49.1477524399250; Wed, 26 Oct 2016 16:26:39 -0700 (PDT) MIME-Version: 1.0 Received: by 10.200.38.135 with HTTP; Wed, 26 Oct 2016 16:26:38 -0700 (PDT) In-Reply-To: References: From: satyajit vegesna Date: Wed, 26 Oct 2016 16:26:38 -0700 Message-ID: Subject: Re: Error with flush_length File in Orc, in hive 2.1.0 and mr execution engine. To: user@hive.apache.org, Eugene Koifman , dev@hive.apache.org Content-Type: multipart/alternative; boundary=001a1141b320f6e9d0053fccf4da archived-at: Wed, 26 Oct 2016 23:26:52 -0000 --001a1141b320f6e9d0053fccf4da Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Eugene, One more observation, in namenode logs when i run select count(*) on individual tables i still see the same error as before, org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 192.168.120.133:47114 Call#4 Retry#0: java.io.FileNotFoundException: File does not exist: /user/hive/warehouse/mls_public_record_association_ snapshot_orc/delta_0000002_0000002_0000/bucket_00094_flush_length but i get the count of the tables and they match well with the source data. So i believe the problem is with joining these tables together. Any specific logs you want me to debug. Regards, Satyajit. On Wed, Oct 26, 2016 at 4:16 PM, satyajit vegesna < satyajit.apasprk@gmail.com> wrote: > Hi Eugene, > > select count(*) from mls_public_record_association_snapshot_orc pra left > outer join mls_listing_snapshot_orc ml on pra.primary_listing_id =3D ml.i= d left > outer join attribute a on a.id =3D ml.standard_status > > ran till end and threw the below exception. > > MapReduce Total cumulative CPU time: 0 days 1 hours 0 minutes 53 seconds > 760 msec > Ended Job =3D job_1477494091659_0024 > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in [jar:file:/opt/apache-hive-2. > 1.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.2/ > share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/ > impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.apache.logging.slf4j. > Log4jLoggerFactory] > 2016-10-26 16:09:01 Starting to launch local task to process map join; ma= ximum > memory =3D 514850816 > Execution failed with exit status: 2 > Obtaining error information > > Task failed! > Task ID: > Stage-9 > > Logs: > > FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql. > exec.mr.MapredLocalTask > MapReduce Jobs Launched: > Stage-Stage-1: Map: 300 Reduce: 121 Cumulative CPU: 3654.02 sec HDFS > Read: 1771032233 HDFS Write: 1917532703 SUCCESS > Total MapReduce CPU Time Spent: 0 days 1 hours 0 minutes 54 seconds 20 ms= ec > > Explain Plan: > > STAGE DEPENDENCIES: > Stage-8 is a root stage , consists of Stage-1 > Stage-1 > Stage-9 depends on stages: Stage-1 > Stage-3 depends on stages: Stage-9 > Stage-0 depends on stages: Stage-3 > > STAGE PLANS: > Stage: Stage-8 > Conditional Operator > > Stage: Stage-1 > Map Reduce > Map Operator Tree: > TableScan > alias: pra > Statistics: Num rows: 99241216 Data size: 9924121600 Basic > stats: COMPLETE Column stats: NONE > Select Operator > expressions: primary_listing_id (type: string) > outputColumnNames: _col0 > Statistics: Num rows: 99241216 Data size: 9924121600 Basic > stats: COMPLETE Column stats: NONE > Reduce Output Operator > key expressions: _col0 (type: string) > sort order: + > Map-reduce partition columns: _col0 (type: string) > Statistics: Num rows: 99241216 Data size: 9924121600 Basi= c > stats: COMPLETE Column stats: NONE > TableScan > alias: ml > Statistics: Num rows: 201432950 Data size: 20949026816 Basic > stats: COMPLETE Column stats: NONE > Select Operator > expressions: id (type: string), standard_status (type: int) > outputColumnNames: _col0, _col1 > Statistics: Num rows: 201432950 Data size: 20949026816 Basi= c > stats: COMPLETE Column stats: NONE > Reduce Output Operator > key expressions: _col0 (type: string) > sort order: + > Map-reduce partition columns: _col0 (type: string) > Statistics: Num rows: 201432950 Data size: 20949026816 > Basic stats: COMPLETE Column stats: NONE > value expressions: _col1 (type: int) > Reduce Operator Tree: > Join Operator > condition map: > Left Outer Join0 to 1 > keys: > 0 _col0 (type: string) > 1 _col0 (type: string) > outputColumnNames: _col2 > Statistics: Num rows: 221576249 Data size: 23043929997 Basic > stats: COMPLETE Column stats: NONE > File Output Operator > compressed: false > table: > input format: org.apache.hadoop.mapred. > SequenceFileInputFormat > output format: org.apache.hadoop.hive.ql.io. > HiveSequenceFileOutputFormat > serde: org.apache.hadoop.hive.serde2. > lazybinary.LazyBinarySerDe > > Stage: Stage-9 --it is failing in the same mapreduce local work. > Map Reduce Local Work > Alias -> Map Local Tables: > $hdt$_2:a > Fetch Operator > limit: -1 > Alias -> Map Local Operator Tree: > $hdt$_2:a > TableScan > alias: a > Statistics: Num rows: 12830 Data size: 51322 Basic stats: > COMPLETE Column stats: NONE > Select Operator > expressions: id (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 12830 Data size: 51322 Basic stats: > COMPLETE Column stats: NONE > HashTable Sink Operator > keys: > 0 _col2 (type: int) > 1 _col0 (type: int) > > Stage: Stage-3 > Map Reduce > Map Operator Tree: > TableScan > Map Join Operator > condition map: > Left Outer Join0 to 1 > keys: > 0 _col2 (type: int) > 1 _col0 (type: int) > Statistics: Num rows: 243733879 Data size: 25348323546 Basi= c > stats: COMPLETE Column stats: NONE > Group By Operator > aggregations: count() > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLET= E > Column stats: NONE > Reduce Output Operator > sort order: > Statistics: Num rows: 1 Data size: 8 Basic stats: > COMPLETE Column stats: NONE > value expressions: _col0 (type: bigint) > Local Work: > Map Reduce Local Work > Reduce Operator Tree: > Group By Operator > aggregations: count(VALUE._col0) > mode: mergepartial > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE > Column stats: NONE > File Output Operator > compressed: false > Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE > Column stats: NONE > table: > input format: org.apache.hadoop.mapred. > SequenceFileInputFormat > output format: org.apache.hadoop.hive.ql.io. > HiveSequenceFileOutputFormat > serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe > > Stage: Stage-0 > Fetch Operator > limit: -1 > Processor Tree: > ListSink > > Any suggestion in debugging this issue is appreciated. > > > Regards, > Satyajit. > > > > > On Wed, Oct 26, 2016 at 3:34 PM, Eugene Koifman > wrote: > >> If you can run this, then it=E2=80=99s safe to ignore =E2=80=9C00094_flu= sh_length=E2=80=9D messages >> and the issue is somewhere else >> >> select count(*) from mls_public_record_association_snapshot_orc pra >> left outer join mls_listing_snapshot_orc ml on pra.primary_listing_id = =3D >> ml.id left outer join attribute a on a.id =3D ml.standard_status >> >> Eugene >> >> From: satyajit vegesna >> Date: Wednesday, October 26, 2016 at 2:14 PM >> To: "user@hive.apache.org" , Eugene Koifman < >> ekoifman@hortonworks.com> >> Cc: "dev@hive.apache.org" >> Subject: Re: Error with flush_length File in Orc, in hive 2.1.0 and mr >> execution engine. >> >> Hi Eugene, >> >> PFB Transaction table in green and parquet tables in yellow, >> >> INSERT INTO access_logs.crawlstats_dpp PARTITION(day=3D"2016-10-23") >> select pra.url as prUrl,pra.url_type as urlType,CAST(pra.created_at AS >> timestamp) as prCreated, CAST(pra.updated_at AS timestamp) as prUpdated, >> CAST(ml.created_at AS timestamp) as mlCreated, CAST(ml.updated_at AS >> timestamp) as mlUpdated, a.name as status, pra.public_record_id as prId, >> acl.accesstime as crawledon, pra.id as propId, pra.primary_listing_id as >> listingId, datediff(CAST(acl.accesstime AS timestamp),CAST(ml.created_at= AS >> timestamp)) as mlcreateage, datediff(CAST(acl.accesstime AS >> timestamp),CAST(ml.updated_at AS timestamp)) as mlupdateage, >> datediff(CAST(acl.accesstime AS timestamp),CAST(pra.created_at AS >> timestamp)) as prcreateage, datediff(CAST(acl.accesstime AS >> timestamp),CAST(pra.updated_at AS timestamp)) as prupdateage, (case whe= n >> (pra.public_record_id is not null and TRIM(pra.public_record_id) <> '') >> then (case when (pra.primary_listing_id is null or >> TRIM(pra.primary_listing_id) =3D '') then 'PR' else 'PRMLS' END) else (= case >> when (pra.primary_listing_id is not null and TRIM(pra.primary_listing_id= ) >> <> '') then 'MLS' else 'UNKNOWN' END) END) as listingType, >> acl.httpstatuscode, acl.httpverb, acl.requesttime, >> acl.upstreamheadertime , acl.upstreamresponsetime, acl.page_id, userag= ent >> AS user_agent, substring(split(pra.url,'/')[0], >> 0,length(split(pra.url,'/')[0])-3) as city, >> substring(split(pra.url,'/')[0], length(split(pra.url,'/')[0])-1,2) as >> state, ml.mls_id FROM access_logs.loadbalancer_accesslogs acl inner >> join mls_public_record_association_snapshot_orc pra on acl.listing_url = =3D >> pra.url left outer join mls_listing_snapshot_orc ml on >> pra.primary_listing_id =3D ml.id left outer join attribute a on a.id = =3D >> ml.standard_status WHERE acl.accesstimedate=3D"2016-10-23"; >> >> >> Any clue, or something that you would want me to focus on to debug the >> issue. >> >> Regards, >> Satyajit. >> >> >> >> On Tue, Oct 25, 2016 at 8:49 PM, Eugene Koifman > > wrote: >> >>> Which of your tables are are transactional? Can you provide the DDL? >>> >>> I don=E2=80=99t think =E2=80=9CFile does not exist=E2=80=9D error is ca= using your queries to >>> fail. It=E2=80=99s an INFO level msg. >>> There should be some other error. >>> >>> Eugene >>> >>> >>> From: satyajit vegesna >>> Reply-To: "user@hive.apache.org" >>> Date: Tuesday, October 25, 2016 at 5:46 PM >>> To: "user@hive.apache.org" , "dev@hive.apache.org= " >>> >>> Subject: Error with flush_length File in Orc, in hive 2.1.0 and mr >>> execution engine. >>> >>> HI All, >>> >>> i am using hive 2.1.0 , hadoop 2.7.2 , but when i try running queries >>> like simple insert, >>> >>> set mapreduce.job.queuename=3Ddefault;set hive.exec.dynamic.partition= =3Dtrue;set >>> hive.exec.dynamic.partition.mode=3Dnonstrict;set >>> hive.exec.max.dynamic.partitions.pernode=3D400;set >>> hive.exec.max.dynamic.partitions=3D2000;set mapreduce.map.memory.mb=3D5= 120;set >>> mapreduce.reduce.memory.mb=3D5120;set mapred.tasktracker.map.tasks.maxi= mum=3D30;set >>> mapred.tasktracker.reduce.tasks.maximum=3D20;set >>> mapred.reduce.child.java.opts=3D-Xmx2048m;set >>> mapred.map.child.java.opts=3D-Xmx2048m; set >>> hive.support.concurrency=3Dtrue; set hive.txn.manager=3Dorg.apache.ha >>> doop.hive.ql.lockmgr.DbTxnManager; set hive.compactor.initiator.on=3Dfa= lse; >>> set hive.compactor.worker.threads=3D1;set mapreduce.job.queuename=3Ddef= ault;set >>> hive.exec.dynamic.partition=3Dtrue;set hive.exec.dynamic.partition.mode= =3Dnonstrict;INSERT >>> INTO access_logs.crawlstats_dpp PARTITION(day=3D"2016-10-23") select pr= a.url >>> as prUrl,pra.url_type as urlType,CAST(pra.created_at AS timestamp) as >>> prCreated, CAST(pra.updated_at AS timestamp) as prUpdated, >>> CAST(ml.created_at AS timestamp) as mlCreated, CAST(ml.updated_at AS >>> timestamp) as mlUpdated, a.name as status, pra.public_record_id as >>> prId, acl.accesstime as crawledon, pra.id as propId, >>> pra.primary_listing_id as listingId, datediff(CAST(acl.accesstime AS >>> timestamp),CAST(ml.created_at AS timestamp)) as mlcreateage, >>> datediff(CAST(acl.accesstime AS timestamp),CAST(ml.updated_at AS >>> timestamp)) as mlupdateage, datediff(CAST(acl.accesstime AS >>> timestamp),CAST(pra.created_at AS timestamp)) as prcreateage, >>> datediff(CAST(acl.accesstime AS timestamp),CAST(pra.updated_at AS >>> timestamp)) as prupdateage, (case when (pra.public_record_id is not nu= ll >>> and TRIM(pra.public_record_id) <> '') then (case when >>> (pra.primary_listing_id is null or TRIM(pra.primary_listing_id) =3D '')= then >>> 'PR' else 'PRMLS' END) else (case when (pra.primary_listing_id is not = null >>> and TRIM(pra.primary_listing_id) <> '') then 'MLS' else 'UNKNOWN' END) = END) >>> as listingType, acl.httpstatuscode, acl.httpverb, acl.requesttime, >>> acl.upstreamheadertime , acl.upstreamresponsetime, acl.page_id, usera= gent >>> AS user_agent, substring(split(pra.url,'/')[0], >>> 0,length(split(pra.url,'/')[0])-3) as city, >>> substring(split(pra.url,'/')[0], length(split(pra.url,'/')[0])-1,2) as >>> state, ml.mls_id FROM access_logs.loadbalancer_accesslogs acl inner >>> join mls_public_record_association_snapshot_orc pra on acl.listing_url >>> =3D pra.url left outer join mls_listing_snapshot_orc ml on >>> pra.primary_listing_id =3D ml.id left outer join attribute a on a.id = =3D >>> ml.standard_status WHERE acl.accesstimedate=3D"2016-10-23"; >>> >>> i finally end up getting below error, >>> >>> 2016-10-25 17:40:18,725 Stage-2 map =3D 100%, reduce =3D 52%, Cumulati= ve >>> CPU 1478.96 sec >>> 2016-10-25 17:40:19,761 Stage-2 map =3D 100%, reduce =3D 62%, Cumulati= ve >>> CPU 1636.58 sec >>> 2016-10-25 17:40:20,794 Stage-2 map =3D 100%, reduce =3D 64%, Cumulati= ve >>> CPU 1764.97 sec >>> 2016-10-25 17:40:21,820 Stage-2 map =3D 100%, reduce =3D 69%, Cumulati= ve >>> CPU 1879.61 sec >>> 2016-10-25 17:40:22,842 Stage-2 map =3D 100%, reduce =3D 80%, Cumulati= ve >>> CPU 2051.38 sec >>> 2016-10-25 17:40:23,872 Stage-2 map =3D 100%, reduce =3D 90%, Cumulati= ve >>> CPU 2151.49 sec >>> 2016-10-25 17:40:24,907 Stage-2 map =3D 100%, reduce =3D 93%, Cumulati= ve >>> CPU 2179.67 sec >>> 2016-10-25 17:40:25,944 Stage-2 map =3D 100%, reduce =3D 94%, Cumulati= ve >>> CPU 2187.86 sec >>> 2016-10-25 17:40:29,062 Stage-2 map =3D 100%, reduce =3D 95%, Cumulati= ve >>> CPU 2205.22 sec >>> 2016-10-25 17:40:30,107 Stage-2 map =3D 100%, reduce =3D 100%, Cumulat= ive >>> CPU 2241.25 sec >>> MapReduce Total cumulative CPU time: 37 minutes 21 seconds 250 msec >>> Ended Job =3D job_1477437520637_0009 >>> SLF4J: Class path contains multiple SLF4J bindings. >>> SLF4J: Found binding in [jar:file:/opt/apache-hive-2.1 >>> .0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/Static >>> LoggerBinder.class] >>> SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.2/sh >>> are/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/im >>> pl/StaticLoggerBinder.class] >>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >>> explanation. >>> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4 >>> jLoggerFactory] >>> 2016-10-25 17:40:35Starting to launch local task to process map join;ma= ximum >>> memory =3D 514850816 >>> Execution failed with exit status: 2 >>> Obtaining error information >>> >>> Task failed! >>> Task ID: >>> Stage-14 >>> >>> Logs: >>> >>> FAILED: Execution Error, return code 2 from >>> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask >>> MapReduce Jobs Launched: >>> Stage-Stage-1: Map: 106 Reduce: 45 Cumulative CPU: 3390.11 sec HDF= S >>> Read: 8060555201 HDFS Write: 757253756 SUCCESS >>> Stage-Stage-2: Map: 204 Reduce: 85 Cumulative CPU: 2241.25 sec HDF= S >>> Read: 2407914653 HDFS Write: 805874953 SUCCESS >>> Total MapReduce CPU Time Spent: 0 days 1 hours 33 minutes 51 seconds 36= 0 >>> msec >>> >>> Could not find any errors in logs, but when i check namenode logs , oi >>> get the following error, >>> >>> 2016-10-25 17:01:51,923 INFO org.apache.hadoop.ipc.Server: IPC Server >>> handler 1 on 9000, call org.apache.hadoop.hdfs.protoco >>> l.ClientProtocol.getBlockLocations from 192.168.120.133:47114 Call#4 >>> Retry#0: java.io.FileNotFoundException: File does not exist: >>> /user/hive/warehouse/mls_public_record_association_snapshot_ >>> orc/delta_0000002_0000002_0000/bucket_00094_flush_length >>> 2016-10-25 17:01:52,779 INFO org.apache.hadoop.ipc.Server: IPC Server >>> handler 1 on 9000, call org.apache.hadoop.hdfs.protoco >>> l.ClientProtocol.getBlockLocations from 192.168.120.132:43008 Call#4 >>> Retry#0: java.io.FileNotFoundException: File does not exist: >>> /user/hive/warehouse/mls_public_record_association_snapshot_ >>> orc/delta_0000002_0000002_0000/bucket_00095_flush_length >>> 2016-10-25 17:01:52,984 INFO org.apache.hadoop.ipc.Server: IPC Server >>> handler 0 on 9000, call org.apache.hadoop.hdfs.protoco >>> l.ClientProtocol.getBlockLocations from 192.168.120.133:47260 Call#4 >>> Retry#0: java.io.FileNotFoundException: File does not exist: >>> /user/hive/warehouse/mls_public_record_association_snapshot_ >>> orc/delta_0000002_0000002_0000/bucket_00096_flush_length >>> 2016-10-25 17:01:53,381 INFO org.apache.hadoop.ipc.Server: IPC Server >>> handler 0 on 9000, call org.apache.hadoop.hdfs.protoco >>> l.ClientProtocol.getBlockLocations from 192.168.120.132:43090 Call#4 >>> Retry#0: java.io.FileNotFoundException: File does not exist: >>> /user/hive/warehouse/mls_public_record_association_snapshot_ >>> orc/delta_0000002_0000002_0000/bucket_00097_flush_length >>> 2016-10-25 17:01:53,971 INFO org.apache.hadoop.ipc.Server: IPC Server >>> handler 1 on 9000, call org.apache.hadoop.hdfs.protoco >>> l.ClientProtocol.getBlockLocations from 192.168.120.134:37444 Call#4 >>> Retry#0: java.io.FileNotFoundException: File does not exist: >>> /user/hive/warehouse/mls_public_record_association_snapshot_ >>> orc/delta_0000002_0000002_0000/bucket_00098_flush_length >>> 2016-10-25 17:01:54,092 INFO org.apache.hadoop.ipc.Server: IPC Server >>> handler 2 on 9000, call org.apache.hadoop.hdfs.protoco >>> l.ClientProtocol.getBlockLocations from 192.168.120.133:47300 Call#4 >>> Retry#0: java.io.FileNotFoundException: File does not exist: >>> /user/hive/warehouse/mls_public_record_association_snapshot_ >>> orc/delta_0000002_0000002_0000/bucket_00099_flush_length >>> 2016-10-25 17:01:55,094 INFO org.apache.hadoop.ipc.Server: IPC Server >>> handler 8 on 9000, call org.apache.hadoop.hdfs.protoco >>> l.ClientProtocol.getBlockLocations from 192.168.120.134:37540 Call#4 >>> Retry#0: java.io.FileNotFoundException: File does not exist: >>> /user/hive/warehouse/mls_public_record_association_snapshot_ >>> orc/delta_0000002_0000002_0000/bucket_00012_flush_length >>> 2016-10-25 17:02:11,269 INFO org.apache.hadoop.ipc.Server: IPC Server >>> handler 5 on 9000, call org.apache.hadoop.hdfs.protoco >>> l.ClientProtocol.getBlockLocations from 192.168.120.133:47378 Call#4 >>> Retry#0: java.io.FileNotFoundException: File does not exist: >>> /user/hive/warehouse/mls_public_record_association_snapshot_ >>> orc/delta_0000002_0000002_0000/bucket_00075_flush_length >>> >>> i also search for find the flush_length files in the above mentioned >>> location, but i only see buckets but no files ending with flush_length. >>> >>> Any clue or help would be highly appreciated. >>> >>> Regards, >>> Satyajit. >>> >>> >> > --001a1141b320f6e9d0053fccf4da Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi=C2=A0Eugene,

<= span style=3D"color:rgb(0,0,0);font-family:calibri,sans-serif;font-size:14p= x">One more observation, in namenode logs when i run select count(*) on ind= ividual tables i still see the same error as before,
<= br>
org.apache.hadoop.ipc= .Server: IPC Server handler 1 on 9000, call org.apache.hadoop.hdfs.<= wbr style=3D"font-size:12.8px">protocol.Cl= ientProtocol.getBlockLocations from=C2=A0192.168.120.133:47= 114=C2=A0Call#4 Retry#0: java.io.FileN= otFoundException: File does not exist: /user/hive/warehouse/mls_public_record_= association_snapshot_orc/delta_0000002_= 0000002_0000/bucket_00094_flush_length=

but i get the count of the tables an= d they match well with the source data.
So i believe the problem is with joining these tables togeth= er.

Any specific logs you want me to debug.

Regards,
Satyajit.

On Wed, Oct 26, 2016 at 4:16 PM, satyajit vegesna <= span dir=3D"ltr"><satyajit.apasprk@gmail.com> wrote:
Hi=C2=A0Eugene,

select count(*) from mls_publ= ic_record_association_snapshot_orc=C2=A0pra =C2=A0left outer jo= in=C2=A0mls_listing_snapsh= ot_orc=C2=A0ml on pra.primary_listing_id =3D=C2=A0ml.id<= span style=3D"font-size:12.8px">=C2=A0=C2=A0left outer join=C2=A0attribute=C2=A0a on=C2=A0a.= id=C2=A0=3D ml.standard_status<= /div>

ran till end and threw the below except= ion.

MapReduce Tot= al cumulative CPU time: 0 days 1 hours 0 minutes 53 seconds 760 msec=
Ended Job =3D job_1477494091659_0024
SLF4J: Class path contains multiple = SLF4J bindings.
SLF4J: Found binding in [j= ar:file:/opt/apache-hive-2.1.0-bin/lib/log4j-slf4j-impl-2.4.1.jar= !/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.2/share/hadoop= /common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLogger= Binder.class]
SLF4J: Actual binding is of type [org.apache.loggi= ng.slf4j.Log4jLoggerFactory]
2= 016-10-26 16:09:01 Starting to launch local task to p= rocess map join; maximum memory =3D 514850816<= /font>
Execution failed with exit statu= s: 2
Obtaining error information

Task failed!<= /span>
Task ID:
=C2=A0 Stage-9
Logs:
=
FAILED: Execution Error, return code = 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask=
MapReduce Jobs Launched:=C2=A0
Stage-Stage-1: Map: 300 =C2=A0Reduce: 121 =C2=A0 Cumulative= CPU: 3654.02 sec =C2=A0 HDFS Read: 1771032233 HDFS Write: 1917532703 SUCCE= SS
Total MapReduce CPU Time Spent: 0 days = 1 hours 0 minutes 54 seconds 20 msec

Explain Plan:

STAGE DEPENDENCIES:
=C2=A0 Stage-8 is a root sta= ge , consists of Stage-1
= =C2=A0 Stage-1
=C2=A0 Stage= -9 depends on stages: Stage-1
=C2=A0 Stage-3 depends on stages: Stage-9
=C2=A0 Stage-0 depends on stages: Stage-3
<= div>
STAGE PLANS:
= =C2=A0 Stage: Stage-8
=C2=A0 =C2=A0 Conditional Operator

= =C2=A0 Stage: Stage-1
=C2= =A0 =C2=A0 Map Reduce
=C2= =A0 =C2=A0 =C2=A0 Map Operator Tree:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 TableScan
<= div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 alias: pra
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Statistics: Num rows: 99241216 Data size: 9= 924121600 Basic stats: COMPLETE Column stats: NONE
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Select Op= erator
=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 expressions: primary_listing_id (type: string)=
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 outputColumnNames: _col0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Sta= tistics: Num rows: 99241216 Data size: 9924121600 Basic stats: COMPLETE Col= umn stats: NONE
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Reduce Output Operator
<= span class=3D"">
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 key expressions: _col0 (type: string)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 sort order: +
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Map-= reduce partition columns: _col0 (type: string)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 Statistics: Num rows: 99241216 Data size: 9924121600 Basic stats: C= OMPLETE Column stats: NONE
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 TableScan
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 alias: ml
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 Statistics: Num rows: 201432950 Data size: 20949026816 Basic = stats: COMPLETE Column stats: NONE
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Select Operator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 expressions: id (type: string), standard_status (type: int)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 outputColumnNames: _col0, _col1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = Statistics: Num rows: 201432950 Data size: 20949026816 Basic stats: COMPLET= E Column stats: NONE
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Reduce Output O= perator
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 key expressions: _col0 (type: string= )
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sort order: +
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Map-reduce partition columns: _col0 (type: string)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 Statistics: Num rows: 201432950 Data size: 20949026816 Basic = stats: COMPLETE Column stats: NONE
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 value expressions: _col1 (type: int)
=C2=A0 =C2=A0 =C2=A0 Reduce Opera= tor Tree:
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 Join Operator
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 condition map:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0Left Outer Join0 to 1
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 keys:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 _col0 (typ= e: string)
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 1 _col0 (type: string)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 outputColumnName= s: _col2
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 Statistics: Num rows: 221576249 Data size: 23043929997 Ba= sic stats: COMPLETE Column stats: NONE
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 File Output= Operator
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 compressed: false
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 table:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 input format: org.apache.hadoop.mapred.SequenceFi= leInputFormat
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 output format: org.apache.hadoop.hive.ql.io.HiveSe= quenceFileOutputFormat
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 serde: org.apache.hado= op.hive.serde2.lazybinary.LazyBinarySerDe

=C2=A0 Stage: Stage-= 9 --it is failing in the same mapreduce local work.
=C2=A0 =C2=A0 Map Reduce L= ocal Work
=C2=A0 =C2=A0 =C2= =A0 Alias -> Map Local Tables:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 $hdt$_2:a=C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 Fetch Operator
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 limit: -1
=C2=A0 =C2=A0 =C2=A0 Alias -> Map Local Operator T= ree:
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 $hdt$_2:a=C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 TableScan
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 alias: a
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 Statistics: Num rows: 12830 Data size: 51322 Basic stats: COM= PLETE Column stats: NONE
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Select Operator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 expressions: id (type: int)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 outputColumnNames: _col0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Statistics: Num rows: 12= 830 Data size: 51322 Basic stats: COMPLETE Column stats: NONE
<= div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 HashTable Sink Operator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 keys:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 _col2 (type: int)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 1 _col0 (type: int)

=C2=A0 Stage: S= tage-3
=C2= =A0 =C2=A0 Map Reduce
=C2= =A0 =C2=A0 =C2=A0 Map Operator Tree:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 TableScan
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Map J= oin Operator
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 condition map:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0Left Outer Join0 to 1
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 keys:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 _col2 (type: int)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 1 _col0 (type: int)
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Statistics: Num rows: 2437338= 79 Data size: 25348323546 Basic stats: COMPLETE Column stats: NONE
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 Group By Operator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 aggregations: = count()
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 mode: hash
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 outputColumnNames: _col0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Statistics: Num r= ows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 Reduce Output Operator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 s= ort order:=C2=A0
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Statistics: Num rows: = 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 value expressions: _col0 (type: bigint)
=C2=A0 =C2=A0 =C2=A0 Local = Work:
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 Map Reduce Local Work
=C2=A0 =C2=A0 =C2=A0 Reduce Operator Tree:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Group By Operator
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 aggregations: count(VALUE._col0)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 mode: mergepartial
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 out= putColumnNames: _col0
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Statistics: Num rows: 1 Data size: 8 Basic = stats: COMPLETE Column stats: NONE
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 File Output Ope= rator
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 compressed: false
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Statistics: N= um rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 table:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 input format: o= rg.apache.hadoop.mapred.SequenceFileInputFormat
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 output format: org.a= pache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
<= /span>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 serde: org.apache.hadoop.hive.serde2.lazy.Laz= ySimpleSerDe

=C2=A0 Stage: Stage-0
=C2=A0 =C2=A0 Fetch Operator
=
=C2=A0 =C2=A0 =C2=A0 limit: -1
=C2=A0 =C2=A0 =C2=A0 Processor Tree= :
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 ListSink

Any suggestion in debugging th= is issue is appreciated.


Regards,=
Satyajit.




On Wed, O= ct 26, 2016 at 3:34 PM, Eugene Koifman <ekoifman@hortonworks.com> wrote:
If = you can run this, then it=E2=80=99s safe to ignore=C2=A0=E2=80=9C00094_flush_length=E2=80=9D=C2=A0messages and the issue is somewhere else

Eugene

From: satyajit vegesna <satyajit.apasprk@g= mail.com>
Date: Wednesday, October 26, 2016 a= t 2:14 PM
To: "user@hive.apache.org" <user@hive.apache.org= >, Eugene Koifman <ekoifman@hortonworks.com>
Cc: "dev@hive.apache.org" <dev@hive.apache.org&g= t;
Subject: Re: Error with flush_lengt= h File in Orc, in hive 2.1.0 and mr execution engine.

Hi=C2=A0Eugene,

PFB Transaction table in green and parquet tables in yellow,

INSERT INTO access_logs.crawlstats_dpp PARTITION(day=3D"2016-10-23") s= elect pra.url as prUrl,pra.url_type as urlType,CAST(pra.created_at AS times= tamp) as prCreated, CAST(pra.updated_at AS timestamp) as prUpdated, CAST(ml= .created_at AS timestamp) as mlCreated, CAST(ml.updated_at AS timestamp) as mlUpdated,=C2=A0a.name=C2=A0as status, pra.public_record_id as prId, acl.accesstime as cr= awledon,=C2=A0pra.id=C2=A0as propId, pra.primary_listing_id as listingId, datediff(CAST(acl.accesstime = AS timestamp),CAST(ml.created_at AS timestamp)) as mlcreateage, datediff(CA= ST(acl.accesstime AS timestamp),CAST(ml.updated_at AS timestamp)) as mlupda= teage, datediff(CAST(acl.accesstime AS timestamp),CAST(pra.created_at AS timestamp)) as prcreateage, datediff(= CAST(acl.accesstime AS timestamp),CAST(pra.updated_at AS timestamp)) as pru= pdateage, =C2=A0(case when (pra.public_record_id is not null and TRIM(pra.p= ublic_record_id) <> '') =C2=A0then (case when (pra.primary_listing_id is null or TRIM(pra.primary_listing_id) =3D '&= #39;) then 'PR' else 'PRMLS' END) =C2=A0else (case when (pr= a.primary_listing_id is not null and TRIM(pra.primary_listing_id) <> = '') then 'MLS' else 'UNKNOWN' END) END) as listingT= ype, =C2=A0acl.httpstatuscode, =C2=A0acl.httpverb, =C2=A0acl.requesttime, acl.upstreamheadertime , acl.up= streamresponsetime, =C2=A0acl.page_id, =C2=A0useragent AS user_agent, =C2= =A0substring(split(pra.url,'/')[0], 0,length(split(pra.url,'/')[0])-3) as city, =C2=A0substring(split(pra.url,'/')[0], length(split(pra.url,'/')[0])-1,2) as state, =C2=A0ml.mls_id =C2=A0FRO= M access_logs.loadbalancer_= accesslogs acl =C2=A0inner join mls_p= ublic_record_association_snapshot_orc pra on acl.listing_url =3D pra.url =C2=A0left outer join mls_listing_snapshot_orc ml on pra.primary_listing_id =3D=C2=A0ml= .id=C2=A0=C2=A0left outer join attribute a on=C2= =A0a.id=C2=A0=3D ml.standard_status = =C2=A0WHERE acl.accesstimedate=3D"2016-10-23";


Any clue, or something that you would= want me to focus on to debug the issue.

Regards,
Satyajit.



On Tue, Oct 25, 2016 at 8:49 PM, Eugene Koifman = <ekoifman@= hortonworks.com> wrote:
Which of your tables are are transactional?=C2=A0 Can you provide the = DDL?

I don=E2=80=99t think =E2=80=9CFile does not exist=E2=80=9D error is c= ausing your queries to fail.=C2=A0 It=E2=80=99s an INFO level msg.
There should be some other error.

Eugene


From: satyajit vegesna <satyajit.apasprk@g= mail.com>
Reply-To: "user@hive.apache.org" <<= a href=3D"mailto:user@hive.apache.org" target=3D"_blank">user@hive.apache.o= rg>
Date: Tuesday, October 25, 2016 at = 5:46 PM
To: "user@hive.apache.org" <user@hive.apache.org= >, "dev@hi= ve.apache.org" <dev@hive.apac= he.org>
Subject: Error with flush_length Fi= le in Orc, in hive 2.1.0 and mr execution engine.

HI All,

i am using hive 2.1.0 , hadoop 2.7.2 , but =C2=A0when i try running qu= eries like simple insert,

set mapreduce.job.queuename=3Ddefault;set hive.exec.dynamic.parti= tion=3Dtrue;set hive.exec.dynamic.partition.mode=3Dnonstrict;set hiv= e.exec.max.dynamic.partitions.pernode=3D400;set hive.exec.max.dynamic.= partitions=3D2000;set mapreduce.map.memory.mb=3D5120;set mapreduce.reduce.memory.mb=3D5120;set mapred.tasktracker.map.tasks.m<= wbr>aximum=3D30;set mapred.tasktracker.reduce.tasks.maximum=3D20;set m= apred.reduce.child.java.opts=3D-Xmx2048m;set mapred.map.child.java.opt= s=3D-Xmx2048m; set hive.support.concurrency=3Dtrue; set hive.txn.manager=3Dorg.apache.hadoop.hive.ql.lockmgr.DbTxnManager; set hive.compactor.initiator.on=3Dfalse; set hive.compactor.wor= ker.threads=3D1;set mapreduce.job.queuename=3Ddefault;set hive.ex= ec.dynamic.partition=3Dtrue;set hive.exec.dynamic.partition.mode=3Dn= onstrict;INSERT INTO access_logs.crawlstats_dpp PARTITION(day=3D"2016-10-23") se= lect pra.url as prUrl,pra.url_type as urlType,CAST(pra.created_at AS timest= amp) as prCreated, CAST(pra.updated_at AS timestamp) as prUpdated, CAST(ml.= created_at AS timestamp) as mlCreated, CAST(ml.updated_at AS timestamp) as mlUpdated, a.= name as status, pra.public_record_id as prId, acl.accesstime as crawled= on, pra.id as propId, pra.prima= ry_listing_id as listingId, datediff(CAST(acl.accesstime AS timestamp),CAST= (ml.created_at AS timestamp)) as mlcreateage, datediff(CAST(acl.accesstime = AS timestamp),CAST(ml.updated_at AS timestamp)) as mlupdateage, datediff(CAST(acl.accesstime AS timestamp),CAS= T(pra.created_at AS timestamp)) as prcreateage, datediff(CAST(acl.accesstim= e AS timestamp),CAST(pra.updated_at AS timestamp)) as prupdateage, =C2=A0(c= ase when (pra.public_record_id is not null and TRIM(pra.public_record_id) <> '') =C2=A0then (case = when (pra.primary_listing_id is null or TRIM(pra.primary_listing_id) =3D &#= 39;') then 'PR' else 'PRMLS' END) =C2=A0else (case when= (pra.primary_listing_id is not null and TRIM(pra.primary_listing_id) <&= gt; '') then 'MLS' else 'UNKNOWN' END) END) as listingType, =C2=A0acl.h= ttpstatuscode, =C2=A0acl.httpverb, =C2=A0acl.requesttime, acl.upstreamheade= rtime , acl.upstreamresponsetime, =C2=A0acl.page_id, =C2=A0useragent AS use= r_agent, =C2=A0substring(split(pra.url,'/')[0], 0,length(split= (pra.url,'/')[0])-3) as city, =C2=A0substring(split(pra.url,'/')[0], length(split(= pra.url,'/')[0])-1,2) as state, =C2=A0ml.mls_id =C2=A0FROM acc= ess_logs.loadbalancer_accesslogs acl =C2=A0inner join mls_public_recor= d_association_snapshot_orc pra on acl.listing_url =3D pra.url =C2=A0le= ft outer join mls_listing_snapshot_orc ml on pra.primary_listing_id =3D ml.id =C2=A0left outer join attribute a on a.id =3D ml.standard_status =C2=A0WHERE acl.accesstimedat= e=3D"2016-10-23";

i finally end up getting below error,
2016-10-25 17:40:18,725 Stage-2 map =3D 100%, =C2=A0reduce =3D 52%, Cu= mulative CPU 1478.96 sec
2016-10-25 17:40:19,761 Stage-2 map =3D 100%, =C2=A0reduce =3D 62%, Cu= mulative CPU 1636.58 sec
2016-10-25 17:40:20,794 Stage-2 map =3D 100%, =C2=A0reduce =3D 64%, Cu= mulative CPU 1764.97 sec
2016-10-25 17:40:21,820 Stage-2 map =3D 100%, =C2=A0reduce =3D 69%, Cu= mulative CPU 1879.61 sec
2016-10-25 17:40:22,842 Stage-2 map =3D 100%, =C2=A0reduce =3D 80%, Cu= mulative CPU 2051.38 sec
2016-10-25 17:40:23,872 Stage-2 map =3D 100%, =C2=A0reduce =3D 90%, Cu= mulative CPU 2151.49 sec
2016-10-25 17:40:24,907 Stage-2 map =3D 100%, =C2=A0reduce =3D 93%, Cu= mulative CPU 2179.67 sec
2016-10-25 17:40:25,944 Stage-2 map =3D 100%, =C2=A0reduce =3D 94%, Cu= mulative CPU 2187.86 sec
2016-10-25 17:40:29,062 Stage-2 map =3D 100%, =C2=A0reduce =3D 95%, Cu= mulative CPU 2205.22 sec
2016-10-25 17:40:30,107 Stage-2 map =3D 100%, =C2=A0reduce =3D 100%, C= umulative CPU 2241.25 sec
MapReduce Total cumulative CPU time: 37 minutes 21 seconds 250 msec
Ended Job =3D job_1477437520637_0009
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apache-hive-2.1.0-bin/lib/= log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.cla= ss]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.2/share/hadoop/= common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerB= inder.class]
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jL= oggerFactory]
2016-10-25 17:40:35Starting to launch local task to process map join;maximum memory =3D 514850816
Execution failed with exit status: 2
Obtaining error information

Task failed!
Task ID:
=C2=A0 Stage-14

Logs:

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.= exec.mr.MapredLocalTask
MapReduce Jobs Launched:=C2=A0
Stage-Stage-1: Map: 106 =C2=A0Reduce: 45 =C2=A0 Cumulative CPU: 3390.1= 1 sec =C2=A0 HDFS Read: 8060555201 HDFS Write: 757253756 SUCCESS
Stage-Stage-2: Map: 204 =C2=A0Reduce: 85 =C2=A0 Cumulative CPU: 2241.2= 5 sec =C2=A0 HDFS Read: 2407914= 653 HDFS Write: 805874953 SUCCESS
Total MapReduce CPU Time Spent: 0 days 1 hours 33 minutes 51 seconds 3= 60 msec

Could not find any errors in logs, but when i check namenode logs , oi= get the following error,

2016-10-25 17:01:51,923 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 1 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.133:= 47114 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00094_flush_length
2016-10-25 17:01:52,779 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 1 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.132:= 43008 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00095_flush_length
2016-10-25 17:01:52,984 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 0 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.133:= 47260 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00096_flush_length
2016-10-25 17:01:53,381 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 0 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.132:= 43090 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00097_flush_length
2016-10-25 17:01:53,971 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 1 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.134:= 37444 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00098_flush_length
2016-10-25 17:01:54,092 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 2 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.133:= 47300 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00099_flush_length
2016-10-25 17:01:55,094 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 8 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.134:= 37540 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00012_flush_length
2016-10-25 17:02:11,269 INFO org.apache.hadoop.ipc.Server: IPC Server = handler 5 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol= .getBlockLocations from 192.168.120.133:= 47378 Call#4 Retry#0: java.io.FileNotFoundException: File does not exis= t: /user/hive/warehouse/mls_public_record_association_snapshot_or= c/delta_0000002_0000002_0000/bucket_00075_flush_length

i also search for find the flush_length files in the above mentioned l= ocation, but i only see buckets but no files ending with flush_length.

Any clue or help would be highly appreciated.

Regards,
Satyajit.


--001a1141b320f6e9d0053fccf4da--