Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 02EAF200C0A for ; Fri, 13 Jan 2017 10:07:31 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 01A97160B32; Fri, 13 Jan 2017 09:07:31 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2410E160B3F for ; Fri, 13 Jan 2017 10:07:29 +0100 (CET) Received: (qmail 7367 invoked by uid 500); 13 Jan 2017 09:07:29 -0000 Mailing-List: contact dev-help@sqoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@sqoop.apache.org Delivered-To: mailing list dev@sqoop.apache.org Received: (qmail 7356 invoked by uid 99); 13 Jan 2017 09:07:29 -0000 Received: from Unknown (HELO jira-lw-us.apache.org) (207.244.88.139) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Jan 2017 09:07:29 +0000 Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 7D3D02528A for ; Fri, 13 Jan 2017 09:07:26 +0000 (UTC) Date: Fri, 13 Jan 2017 09:07:26 +0000 (UTC) From: "Antonio Villar (JIRA)" To: dev@sqoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (SQOOP-3107) Error with append_values hint and large tables MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 13 Jan 2017 09:07:31 -0000 [ https://issues.apache.org/jira/browse/SQOOP-3107?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1582= 1450#comment-15821450 ]=20 Antonio Villar commented on SQOOP-3107: --------------------------------------- I duplicated the issue 3106 many times (3107 to 3122) because of the proble= m with JIRA yesterday. > Error with append_values hint and large tables > ---------------------------------------------- > > Key: SQOOP-3107 > URL: https://issues.apache.org/jira/browse/SQOOP-3107 > Project: Sqoop > Issue Type: Bug > Components: connectors/oracle > Affects Versions: 1.4.5 > Reporter: Antonio Villar > Labels: export, oracle > > I'm using Oracle Data Connector for Oracle and Hadoop to export data from= Hive to Oracle, when i use the append_values hint i get a Exception and ma= ppers don't end. > This only happens when i export more than 18 columns of the original tabl= e. > The command that i'm using is: > {code} > sqoop export \ > -D oraoop.partitioned=3Dtrue -D oraoop.template.table=3DECI_ORA.CARTERA= _XPAN_PROD_EXT_15 \ > -D oraoop.oracle.append.values.hint.usage=3DON \ > -connect jdbc:oracle:thin:ECI_ORA/XXXXXX@dpexr02-scan.lacaixa.es:1522/= s_dpedwr_datapool.lacaixa.es \ > --num-mappers 4 \ > --table ECI_ORA.AAA_1GIGA \ > --columns "TIINDI,CODINDI,EMPIMP,OFIIMP,EMPEMI,TICOGE,CONCSIG,CDCREP,E= MPDIV,CODDIV,CODCARTE,EMPGEST,CDSEGCLI,TIPCAR,NUESTCE,TIDADCE,EQUIPO,CDSEGB= AN,CODPROD,INDVALO101,INDVALO102,INDVALO103,INDVALO104,INDVALO105,INDVALO10= 6,INDVALO107,INDVALO108,INDVALO109,INDVALO110,INDVALO111,INDVALO112,INDVALO= 113,INDVALO114,INDVALO115,INDVALO116,INDVALO117,INDVALO118,INDVALO119,INDVA= LO201,INDVALO202,INDVALO203,INDVALO204,INDVALO205,INDVALO206,INDVALO207,IND= VALO208,INDVALO209,INDVALO210,INDVALO211,INDVALO212,INDVALO213,INDVALO214,I= NDVALO215,INDVALO216,INDVALO217,INDVALO218,INDVALO219,NIVEL" \ > --export-dir /apps/eciora/hive/l04_eciora.db/cartera_xpan_prod_ext_1 \ > --input-fields-terminated-by '\01' \ > --input-lines-terminated-by '\n' \ > --input-null-non-string '\\\\N' \ > --input-null-string '\\\\N' \ > --direct=20 > {code} > This is the log of the mapper: > {code} > 2016-12-29 11:39:36,886 INFO [main] org.apache.hadoop.metrics2.impl.Metri= csConfig: loaded properties from hadoop-metrics2.properties > 2016-12-29 11:39:36,987 INFO [main] org.apache.hadoop.metrics2.impl.Metri= csSystemImpl: Scheduled snapshot period at 10 second(s). > 2016-12-29 11:39:36,987 INFO [main] org.apache.hadoop.metrics2.impl.Metri= csSystemImpl: MapTask metrics system started > 2016-12-29 11:39:36,998 INFO [main] org.apache.hadoop.mapred.YarnChild: E= xecuting with tokens: > 2016-12-29 11:39:36,998 INFO [main] org.apache.hadoop.mapred.YarnChild: K= ind: mapreduce.job, Service: job_1481807148415_6344, Ident: (org.apache.had= oop.mapreduce.security.token.JobTokenIdentifier@6e171cd7) > 2016-12-29 11:39:37,079 INFO [main] org.apache.hadoop.mapred.YarnChild: S= leeping for 0ms before retrying again. Got null now. > 2016-12-29 11:39:37,364 INFO [main] org.apache.hadoop.mapred.YarnChild: m= apreduce.cluster.local.dir for child: /u12/hadoop/yarn/nm/usercache/Ieciora= 1/appcache/application_1481807148415_6344,/u11/hadoop/yarn/nm/usercache/Iec= iora1/appcache/application_1481807148415_6344,/u10/hadoop/yarn/nm/usercache= /Ieciora1/appcache/application_1481807148415_6344,/u09/hadoop/yarn/nm/userc= ache/Ieciora1/appcache/application_1481807148415_6344,/u08/hadoop/yarn/nm/u= sercache/Ieciora1/appcache/application_1481807148415_6344,/u07/hadoop/yarn/= nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u06/hadoop/y= arn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u05/hado= op/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/u04/= hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_6344,/= u03/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_1481807148415_63= 44,/u02/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_148180714841= 5_6344,/u01/hadoop/yarn/nm/usercache/Ieciora1/appcache/application_14818071= 48415_6344 > 2016-12-29 11:39:37,645 INFO [main] org.apache.hadoop.conf.Configuration.= deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id > 2016-12-29 11:39:38,182 INFO [main] org.apache.hadoop.mapred.Task: Using= ResourceCalculatorProcessTree : [ ] > 2016-12-29 11:39:38,528 INFO [main] org.apache.hadoop.mapred.MapTask: Pro= cessing split: org.apache.sqoop.mapreduce.hcat.SqoopHCatInputSplit@62f68dff > 2016-12-29 11:39:38,572 WARN [main] org.apache.sqoop.manager.oracle.OraOo= pUtilities: System property java.security.egd is not set to file:///dev/ura= ndom - Oracle connections may time out. > 2016-12-29 11:39:38,575 INFO [main] org.apache.sqoop.manager.oracle.OraOo= pOutputFormatBase: The number of batch-inserts to perform per commit has be= en changed from 0 to 1. This is in response to the Oracle APPEND_VALUES hin= t being used. > 2016-12-29 11:39:38,576 INFO [main] org.apache.sqoop.manager.oracle.OraOo= pOutputFormatBase: The number of rows per batch-insert has been changed fro= m 0 to 5000. This is in response to the Oracle APPEND_VALUES hint being use= d. > 2016-12-29 11:39:39,142 INFO [main] org.apache.sqoop.manager.oracle.OraOo= pOutputFormatBase: The number of rows per batch is: 5000 > 2016-12-29 11:39:39,142 INFO [main] org.apache.sqoop.manager.oracle.OraOo= pOutputFormatBase: The number of batches per commit is: 1 > 2016-12-29 11:39:39,341 INFO [main] org.apache.sqoop.manager.oracle.OraOo= pOutputFormatBase: This record writer is connected to Oracle via the JDBC U= RL:=20 > =09"oracle.jdbc.driver.T4CConnection@72a85671" > =09to the Oracle instance: "dpdb1rb1" > 2016-12-29 11:39:39,348 INFO [main] org.apache.sqoop.manager.oracle.OraOo= pOracleQueries: Session Time Zone set to GMT > 2016-12-29 11:39:39,350 INFO [main] org.apache.sqoop.manager.oracle.Oracl= eConnectionFactory: Initializing Oracle session with SQL : > begin=20 > dbms_application_info.set_module(module_name =3D> 'Data Connector for O= racle and Hadoop', action_name =3D> 'export 20161229113914CET');=20 > end; > 2016-12-29 11:39:39,351 INFO [main] org.apache.sqoop.manager.oracle.Oracl= eConnectionFactory: Initializing Oracle session with SQL : alter session fo= rce parallel dml > 2016-12-29 11:39:39,351 INFO [main] org.apache.sqoop.manager.oracle.Oracl= eConnectionFactory: Initializing Oracle session with SQL : alter session di= sable parallel query > 2016-12-29 11:39:39,500 INFO [main] org.apache.hive.hcatalog.mapreduce.In= ternalUtil: Initializing org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe= with properties {name=3Dl04_eciora.cartera_xpan_prod_ext_10000_rows, numFi= les=3D1, columns.types=3Dint,int,int,int,int,int,int,int,int,int,int,int,in= t,int,int,int,int,int,int,decimal(15,2),decimal(15,2),decimal(15,2),decimal= (15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2= ),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),dec= imal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(= 15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2)= ,decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),deci= mal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(15,2),decimal(1= 5,2),decimal(15,2),decimal(15,2),int, serialization.format=3D1, columns=3Dt= iindi,codindi,empimp,ofiimp,empemi,ticoge,concsig,cdcrep,empdiv,coddiv,codc= arte,empgest,cdsegcli,tipcar,nuestce,tidadce,equipo,cdsegban,codprod,indval= o101,indvalo102,indvalo103,indvalo104,indvalo105,indvalo106,indvalo107,indv= alo108,indvalo109,indvalo110,indvalo111,indvalo112,indvalo113,indvalo114,in= dvalo115,indvalo116,indvalo117,indvalo118,indvalo119,indvalo201,indvalo202,= indvalo203,indvalo204,indvalo205,indvalo206,indvalo207,indvalo208,indvalo20= 9,indvalo210,indvalo211,indvalo212,indvalo213,indvalo214,indvalo215,indvalo= 216,indvalo217,indvalo218,indvalo219,nivel, rawDataSize=3D1571259, numRows= =3D10000, serialization.lib=3Dorg.apache.hadoop.hive.serde2.lazy.LazySimple= SerDe, COLUMN_STATS_ACCURATE=3Dtrue, totalSize=3D1581259, serialization.nul= l.format=3D\N, transient_lastDdlTime=3D1482491985} > 2016-12-29 11:39:40,299 INFO [main] org.apache.sqoop.manager.oracle.OraOo= pOutputFormatBase: Batch-Mode insert statement: > insert /*+APPEND_VALUES*/ into "ECI_ORA"."AAA_1GIGA" > (TIINDI > ... > ,NIVEL) > values > (:TIINDI > ... > ,:NIVEL) > 2016-12-29 11:39:40,299 DEBUG [main] org.apache.sqoop.manager.oracle.OraO= opOutputFormatBase: Prepared Statement SQL: > insert /*+APPEND_VALUES*/ into "ECI_ORA"."AAA_1GIGA" > (TIINDI > =E2=80=A6. > ,:NIVEL) > 2016-12-29 11:39:41,279 DEBUG [main] org.apache.sqoop.manager.oracle.OraO= opOutputFormatBase: Prepared Statement SQL: > insert /*+APPEND_VALUES*/ into "ECI_ORA"."AAA_1GIGA" > (TIINDI > =E2=80=A6. > ,:INDVALO219 > ,:NIVEL) > 2016-12-29 11:39:41,679 ERROR [Thread-11] org.apache.hadoop.yarn.YarnUnca= ughtExceptionHandler: Thread Thread[Thread-11,5,main] threw an Exception. > java.lang.ArrayIndexOutOfBoundsException > =09at java.lang.System.arraycopy(Native Method) > =09at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePrepa= redStatement.java:12208) > =09at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatem= entWrapper.java:246) > =09at org.apache.sqoop.mapreduce.AsyncSqlOutputFormat$AsyncSqlExecThread.= run(AsyncSqlOutputFormat.java:231) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)