Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 42681200C22 for ; Tue, 21 Feb 2017 12:34:52 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 40EAC160B68; Tue, 21 Feb 2017 11:34:52 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3A632160B3E for ; Tue, 21 Feb 2017 12:34:51 +0100 (CET) Received: (qmail 96490 invoked by uid 500); 21 Feb 2017 11:34:50 -0000 Mailing-List: contact dev-help@atlas.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@atlas.incubator.apache.org Delivered-To: mailing list dev@atlas.incubator.apache.org Received: (qmail 96479 invoked by uid 99); 21 Feb 2017 11:34:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Feb 2017 11:34:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id E8C9A1A7ABA for ; Tue, 21 Feb 2017 11:34:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.199 X-Spam-Level: X-Spam-Status: No, score=-1.199 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id mP-e2pXFNcjV for ; Tue, 21 Feb 2017 11:34:47 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 56DF25F1B8 for ; Tue, 21 Feb 2017 11:34:46 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 5C508E02F1 for ; Tue, 21 Feb 2017 11:34:44 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 18A542411F for ; Tue, 21 Feb 2017 11:34:44 +0000 (UTC) Date: Tue, 21 Feb 2017 11:34:44 +0000 (UTC) From: "Vimal Sharma (JIRA)" To: dev@atlas.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (ATLAS-1572) sqoop metadata ingest fails with "org.apache.atlas.exception.AtlasBaseException: expected type List or Set; found org.apache.atlas.typesystem.Referenceable" MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 21 Feb 2017 11:34:52 -0000 [ https://issues.apache.org/jira/browse/ATLAS-1572?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:all-tabpanel ] Vimal Sharma updated ATLAS-1572: -------------------------------- Attachment: ATLAS-1572.patch Attaching the patch. Sqoop IT tests are passing > sqoop metadata ingest fails with "org.apache.atlas.exception.AtlasBaseExc= eption: expected type List or Set; found org.apache.atlas.typesystem.Refere= nceable" > -------------------------------------------------------------------------= ---------------------------------------------------------------------------= -------- > > Key: ATLAS-1572 > URL: https://issues.apache.org/jira/browse/ATLAS-1572 > Project: Atlas > Issue Type: Bug > Components: atlas-intg > Affects Versions: trunk, 0.8-incubating > Reporter: Ayub Khan > Priority: Blocker > Fix For: trunk, 0.8-incubating > > Attachments: ATLAS-1572.patch > > > Sqoop import succeeds but the metadata ingest fails with below error > {noformat} > 2017-02-20 11:20:43,179 WARN - [NotificationHookConsumer thread-0:] ~ Ma= x retries exceeded for message [{Id=3D'(type: sqoop_dbdatastore, id: )', traits=3D[], values=3D{owner=3Dhrt_qa, storeUri=3Djdbc:mysql://ct= r-e129-1487033772569-11862-01-000002.hwx.site/sqoopTests?zeroDateTimeBehavi= or=3DconvertToNull, dbStoreType=3Dmysql, qualifiedName=3Dmysql --url jdbc:m= ysql://ctr-e129-1487033772569-11862-01-000002.hwx.site/sqoopTests?zeroDateT= imeBehavior=3DconvertToNull --table createSqoopTablejmnwwf8rji, name=3Dmysq= l --url jdbc:mysql://ctr-e129-1487033772569-11862-01-000002.hwx.site/sqoopT= ests?zeroDateTimeBehavior=3DconvertToNull --table createSqoopTablejmnwwf8rj= i, description=3D, source=3DcreateSqoopTablejmnwwf8rji, storeUse=3DTABLE}},= {Id=3D'(type: hive_db, id: )', traits=3D[], values=3D{name=3Dd= efault, qualifiedName=3Ddefault@cl1, clusterName=3Dcl1}}, {Id=3D'(type: hiv= e_table, id: )', traits=3D[], values=3D{name=3Dcreatesqooptable= jmnwwf8rji, qualifiedName=3Ddefault.createsqooptablejmnwwf8rji@cl1, db=3D{I= d=3D'(type: hive_db, id: )', traits=3D[], values=3D{name=3Ddefa= ult, qualifiedName=3Ddefault@cl1, clusterName=3Dcl1}}}}, {Id=3D'(type: sqoo= p_process, id: )', traits=3D[], values=3D{outputs=3D{Id=3D'(typ= e: hive_table, id: )', traits=3D[], values=3D{name=3Dcreatesqoo= ptablejmnwwf8rji, qualifiedName=3Ddefault.createsqooptablejmnwwf8rji@cl1, d= b=3D{Id=3D'(type: hive_db, id: )', traits=3D[], values=3D{name= =3Ddefault, qualifiedName=3Ddefault@cl1, clusterName=3Dcl1}}}}, commandline= Opts=3D{db.clear.staging.table=3Dfalse, hive.import=3Dtrue, db.require.pass= word=3Dtrue, codegen.output.delimiters.enclose=3D0, codegen.input.delimiter= s.field=3D0, customtool.options.jsonmap=3D{}, hive.compute.stats.table=3Dfa= lse, db.connect.string=3Djdbc:mysql://ctr-e129-1487033772569-11862-01-00000= 2.hwx.site/sqoopTests?zeroDateTimeBehavior=3DconvertToNull, incremental.mod= e=3DNone, db.table=3DcreateSqoopTablejmnwwf8rji, verbose=3Dfalse, codegen.o= utput.delimiters.enclose.required=3Dfalse, mapreduce.num.mappers=3D4, hdfs.= append.dir=3Dfalse, direct.import=3Dfalse, hive.drop.delims=3Dfalse, hive.o= verwrite.table=3Dfalse, hbase.bulk.load.enabled=3Dfalse, hive.fail.table.ex= ists=3Dfalse, relaxed.isolation=3Dfalse, hdfs.delete-target.dir=3Dfalse, sp= lit.limit=3Dnull, db.username=3Dsqoop, codegen.input.delimiters.enclose.req= uired=3Dfalse, codegen.output.dir=3D., import.direct.split.size=3D0, reset.= onemapper=3Dfalse, codegen.output.delimiters.record=3D10, temporary.dirRoot= =3D_sqoop, hcatalog.create.table=3Dfalse, db.batch=3Dfalse, import.fetch.si= ze=3D-2147483648, accumulo.max.latency=3D5000, hdfs.file.format=3DTextFile,= codegen.output.delimiters.field=3D1, mainframe.input.dataset.type=3Dp, cod= egen.output.delimiters.escape=3D0, hcatalog.drop.and.create.table=3Dfalse, = import.max.inline.lob.size=3D16777216, hbase.create.table=3Dfalse, codegen.= auto.compile.dir=3Dtrue, codegen.compile.dir=3D/tmp/sqoop-hrt_qa/compile/f7= c5181710812973a20a3e4776c5a6ef, codegen.input.delimiters.enclose=3D0, expor= t.new.update=3DUpdateOnly, enable.compression=3Dfalse, skip.dist.cache=3Dfa= lse, accumulo.batch.size=3D10240000, hdfs.warehouse.dir=3D/user/hrt_qa, cod= egen.input.delimiters.record=3D0, codegen.input.delimiters.escape=3D0, accu= mulo.create.table=3Dfalse}, inputs=3D{Id=3D'(type: sqoop_dbdatastore, id: <= unassigned>)', traits=3D[], values=3D{owner=3Dhrt_qa, storeUri=3Djdbc:mysql= ://ctr-e129-1487033772569-11862-01-000002.hwx.site/sqoopTests?zeroDateTimeB= ehavior=3DconvertToNull, dbStoreType=3Dmysql, qualifiedName=3Dmysql --url j= dbc:mysql://ctr-e129-1487033772569-11862-01-000002.hwx.site/sqoopTests?zero= DateTimeBehavior=3DconvertToNull --table createSqoopTablejmnwwf8rji, name= =3Dmysql --url jdbc:mysql://ctr-e129-1487033772569-11862-01-000002.hwx.site= /sqoopTests?zeroDateTimeBehavior=3DconvertToNull --table createSqoopTablejm= nwwf8rji, description=3D, source=3DcreateSqoopTablejmnwwf8rji, storeUse=3DT= ABLE}}, qualifiedName=3Dsqoop import --connect jdbc:mysql://ctr-e129-148703= 3772569-11862-01-000002.hwx.site/sqoopTests?zeroDateTimeBehavior=3DconvertT= oNull --table createSqoopTablejmnwwf8rji --hive-import --hive-database defa= ult --hive-table createsqooptablejmnwwf8rji --hive-cluster cl1, name=3Dsqoo= p import --connect jdbc:mysql://ctr-e129-1487033772569-11862-01-000002.hwx.= site/sqoopTests?zeroDateTimeBehavior=3DconvertToNull --table createSqoopTab= lejmnwwf8rji --hive-import --hive-database default --hive-table createsqoop= tablejmnwwf8rji --hive-cluster cl1, startTime=3D2017-02-20T11:20:04.901Z, e= ndTime=3D2017-02-20T11:20:40.055Z, userName=3Dhrt_qa, operation=3Dimport}}]= (NotificationHookConsumer:324) > org.apache.atlas.exception.AtlasBaseException: expected type List or Set;= found org.apache.atlas.typesystem.Referenceable > at org.apache.atlas.repository.converters.AtlasArrayFormatConvert= er.fromV1ToV2(AtlasArrayFormatConverter.java:50) > at org.apache.atlas.repository.converters.AtlasArrayFormatConvert= er.fromV1ToV2(AtlasArrayFormatConverter.java:34) > at org.apache.atlas.repository.converters.AtlasStructFormatConver= ter.fromV1ToV2(AtlasStructFormatConverter.java:169) > at org.apache.atlas.repository.converters.AtlasEntityFormatConver= ter.fromV1ToV2(AtlasEntityFormatConverter.java:74) > at org.apache.atlas.repository.converters.AtlasInstanceConverter.= fromV1toV2Entity(AtlasInstanceConverter.java:222) > at org.apache.atlas.repository.converters.AtlasInstanceConverter.= toAtlasEntities(AtlasInstanceConverter.java:204) > at org.apache.atlas.notification.NotificationHookConsumer$HookCon= sumer.handleMessage(NotificationHookConsumer.java:256) > at org.apache.atlas.notification.NotificationHookConsumer$HookCon= sumer.run(NotificationHookConsumer.java:230) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 2017-02-20 11:21:06,094 ERROR - [NotificationHookConsumer thread-0:] ~ [D= ROPPED_NOTIFICATION] {"version":{"version":"1.0.0"},"message":{"entities":[= {"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Refer= ence","id":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerializa= tion$_Id","id":"-16210205233888062","version":0,"typeName":"sqoop_dbdatasto= re","state":"ACTIVE"},"typeName":"sqoop_dbdatastore","values":{"name":"mysq= l --url jdbc:mysql://ctr-e129-1487033772569-11862-01-000002.hwx.site/sqoopT= ests?zeroDateTimeBehavior\u003dconvertToNull --table createSqoopTablejmnwwf= 8rji","source":"createSqoopTablejmnwwf8rji","storeUse":"TABLE","description= ":"","storeUri":"jdbc:mysql://ctr-e129-1487033772569-11862-01-000002.hwx.si= te/sqoopTests?zeroDateTimeBehavior\u003dconvertToNull","qualifiedName":"mys= ql --url jdbc:mysql://ctr-e129-1487033772569-11862-01-000002.hwx.site/sqoop= Tests?zeroDateTimeBehavior\u003dconvertToNull --table createSqoopTablejmnww= f8rji","owner":"hrt_qa","dbStoreType":"mysql"},"traitNames":[],"traits":{},= "systemAttributes":{}},{"jsonClass":"org.apache.atlas.typesystem.json.Insta= nceSerialization$_Reference","id":{"jsonClass":"org.apache.atlas.typesystem= .json.InstanceSerialization$_Id","id":"-16210205233888061","version":0,"typ= eName":"hive_db","state":"ACTIVE"},"typeName":"hive_db","values":{"qualifie= dName":"default@cl1","clusterName":"cl1","name":"default"},"traitNames":[],= "traits":{},"systemAttributes":{}},{"jsonClass":"org.apache.atlas.typesyste= m.json.InstanceSerialization$_Reference","id":{"jsonClass":"org.apache.atla= s.typesystem.json.InstanceSerialization$_Id","id":"-16210205233888060","ver= sion":0,"typeName":"hive_table","state":"ACTIVE"},"typeName":"hive_table","= values":{"qualifiedName":"default.createsqooptablejmnwwf8rji@cl1","db":{"js= onClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference= ","id":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization= $_Id","id":"-16210205233888061","version":0,"typeName":"hive_db","state":"A= CTIVE"},"typeName":"hive_db","values":{"qualifiedName":"default@cl1","clust= erName":"cl1","name":"default"},"traitNames":[],"traits":{},"systemAttribut= es":{}},"name":"createsqooptablejmnwwf8rji"},"traitNames":[],"traits":{},"s= ystemAttributes":{}},{"jsonClass":"org.apache.atlas.typesystem.json.Instanc= eSerialization$_Reference","id":{"jsonClass":"org.apache.atlas.typesystem.j= son.InstanceSerialization$_Id","id":"-16210205233888059","version":0,"typeN= ame":"sqoop_process","state":"ACTIVE"},"typeName":"sqoop_process","values":= {"name":"sqoop import --connect jdbc:mysql://ctr-e129-1487033772569-11862-0= 1-000002.hwx.site/sqoopTests?zeroDateTimeBehavior\u003dconvertToNull --tabl= e createSqoopTablejmnwwf8rji --hive-import --hive-database default --hive-t= able createsqooptablejmnwwf8rji --hive-cluster cl1","startTime":"2017-02-20= T11:20:04.901Z","outputs":{"jsonClass":"org.apache.atlas.typesystem.json.In= stanceSerialization$_Reference","id":{"jsonClass":"org.apache.atlas.typesys= tem.json.InstanceSerialization$_Id","id":"-16210205233888060","version":0,"= typeName":"hive_table","state":"ACTIVE"},"typeName":"hive_table","values":{= "qualifiedName":"default.createsqooptablejmnwwf8rji@cl1","db":{"jsonClass":= "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference","id":{"= jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id","id= ":"-16210205233888061","version":0,"typeName":"hive_db","state":"ACTIVE"},"= typeName":"hive_db","values":{"qualifiedName":"default@cl1","clusterName":"= cl1","name":"default"},"traitNames":[],"traits":{},"systemAttributes":{}},"= name":"createsqooptablejmnwwf8rji"},"traitNames":[],"traits":{},"systemAttr= ibutes":{}},"commandlineOpts":{"db.clear.staging.table":"false","hive.impor= t":"true","db.require.password":"true","codegen.output.delimiters.enclose":= "0","codegen.input.delimiters.field":"0","customtool.options.jsonmap":"{}",= "hive.compute.stats.table":"false","db.connect.string":"jdbc:mysql://ctr-e1= 29-1487033772569-11862-01-000002.hwx.site/sqoopTests?zeroDateTimeBehavior\u= 003dconvertToNull","incremental.mode":"None","db.table":"createSqoopTablejm= nwwf8rji","verbose":"false","codegen.output.delimiters.enclose.required":"f= alse","mapreduce.num.mappers":"4","hdfs.append.dir":"false","direct.import"= :"false","hive.drop.delims":"false","hive.overwrite.table":"false","hbase.b= ulk.load.enabled":"false","hive.fail.table.exists":"false","relaxed.isolati= on":"false","hdfs.delete-target.dir":"false","split.limit":"null","db.usern= ame":"sqoop","codegen.input.delimiters.enclose.required":"false","codegen.o= utput.dir":".","import.direct.split.size":"0","reset.onemapper":"false","co= degen.output.delimiters.record":"10","temporary.dirRoot":"_sqoop","hcatalog= .create.table":"false","db.batch":"false","import.fetch.size":"-2147483648"= ,"accumulo.max.latency":"5000","hdfs.file.format":"TextFile","codegen.outpu= t.delimiters.field":"1","mainframe.input.dataset.type":"p","codegen.output.= delimiters.escape":"0","hcatalog.drop.and.create.table":"false","import.max= .inline.lob.size":"16777216","hbase.create.table":"false","codegen.auto.com= pile.dir":"true","codegen.compile.dir":"/tmp/sqoop-hrt_qa/compile/f7c518171= 0812973a20a3e4776c5a6ef","codegen.input.delimiters.enclose":"0","export.new= .update":"UpdateOnly","enable.compression":"false","skip.dist.cache":"false= ","accumulo.batch.size":"10240000","hdfs.warehouse.dir":"/user/hrt_qa","cod= egen.input.delimiters.record":"0","codegen.input.delimiters.escape":"0","ac= cumulo.create.table":"false"},"endTime":"2017-02-20T11:20:40.055Z","inputs"= :{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Refe= rence","id":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerializ= ation$_Id","id":"-16210205233888062","version":0,"typeName":"sqoop_dbdatast= ore","state":"ACTIVE"},"typeName":"sqoop_dbdatastore","values":{"name":"mys= ql --url jdbc:mysql://ctr-e129-1487033772569-11862-01-000002.hwx.site/sqoop= Tests?zeroDateTimeBehavior\u003dconvertToNull --table createSqoopTablejmnww= f8rji","source":"createSqoopTablejmnwwf8rji","storeUse":"TABLE","descriptio= n":"","storeUri":"jdbc:mysql://ctr-e129-1487033772569-11862-01-000002.hwx.s= ite/sqoopTests?zeroDateTimeBehavior\u003dconvertToNull","qualifiedName":"my= sql --url jdbc:mysql://ctr-e129-1487033772569-11862-01-000002.hwx.site/sqoo= pTests?zeroDateTimeBehavior\u003dconvertToNull --table createSqoopTablejmnw= wf8rji","owner":"hrt_qa","dbStoreType":"mysql"},"traitNames":[],"traits":{}= ,"systemAttributes":{}},"operation":"import","qualifiedName":"sqoop import = --connect jdbc:mysql://ctr-e129-1487033772569-11862-01-000002.hwx.site/sqoo= pTests?zeroDateTimeBehavior\u003dconvertToNull --table createSqoopTablejmnw= wf8rji --hive-import --hive-database default --hive-table createsqooptablej= mnwwf8rji --hive-cluster cl1","userName":"hrt_qa"},"traitNames":[],"traits"= :{},"systemAttributes":{}}],"type":"ENTITY_CREATE","user":"hrt_qa"}} (FAILE= D:339) > {noformat} > Steps to repro: > Run the below sqoop import command > {noformat} > /usr/hdp/current/sqoop-client/bin/sqoop import --connect jdbc:mysql://= ctr-e129-1487033772569-11862-01-000002.hwx.site/sqoopTests --username sqoop= --password sqoop --hive-import --warehouse-dir /user/hrt_qa --table creat= eSqoopTablejmnwwf8rji > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)