atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shwetha G S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ATLAS-710) Soft deletes: GET response on a deleted hive tables should have references to hive_db and hive_storagedesc
Date Sat, 30 Apr 2016 17:02:12 GMT

    [ https://issues.apache.org/jira/browse/ATLAS-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265371#comment-15265371
] 

Shwetha G S commented on ATLAS-710:
-----------------------------------

The issue is because schema tab in UI uses schema API which lists all columns for a given
table name. It doesn't filter the deleted entities. Instead, schema API should give the schema
for a given guid. 

> Soft deletes: GET response on a deleted hive tables should have references to hive_db
and hive_storagedesc
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: ATLAS-710
>                 URL: https://issues.apache.org/jira/browse/ATLAS-710
>             Project: Atlas
>          Issue Type: Bug
>    Affects Versions: 0.7-incubating
>            Reporter: Ayub Khan
>            Assignee: Shwetha G S
>
> Soft deletes: GET response on a deleted hive tables does not have references to hive_db
and hive_storagedesc.
> After creating the hive_table, get response shows references to db, sd and columns
> {noformat}
> curl 'http://localhost:21000/api/atlas/entities/1b11b234-7360-4679-93f9-d1c01f839f79'
-H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112
Safari/537.36' -H 'Accept: application/json, text/plain, */*' -H 'Referer: http://localhost:21000/index.html'
-H 'Cookie: JSESSIONID=3frbae3qwzi2xkdjyvz8nyko' -H 'Connection: keep-alive' --compressed
| python -m json.tool
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100  3271    0  3271    0     0   2088      0 --:--:--  0:00:01 --:--:--  2088
> {
>     "GUID": "1b11b234-7360-4679-93f9-d1c01f839f79",
>     "definition": {
>         "id": {
>             "id": "1b11b234-7360-4679-93f9-d1c01f839f79",
>             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>             "state": "ACTIVE",
>             "typeName": "hive_table",
>             "version": 0
>         },
>         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>         "traitNames": [],
>         "traits": {},
>         "typeName": "hive_table",
>         "values": {
>             "columns": [
>                 {
>                     "id": {
>                         "id": "2c50426e-7db3-4912-9bbc-13150e962ed5",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "name",
>                         "qualifiedName": "default.after_changes_13.name@primary",
>                         "type": "string"
>                     }
>                 },
>                 {
>                     "id": {
>                         "id": "2af81d25-bbcd-44bd-95b8-4fee915c9dba",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "age",
>                         "qualifiedName": "default.after_changes_13.age@primary",
>                         "type": "int"
>                     }
>                 }
>             ],
>             "comment": null,
>             "createTime": "2016-04-26T09:27:29.000Z",
>             "db": {
>                 "id": "b23cc76a-32e4-4137-860e-8cac2689ba0e",
>                 "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                 "state": "ACTIVE",
>                 "typeName": "hive_db",
>                 "version": 0
>             },
>             "description": null,
>             "lastAccessTime": "2016-04-26T09:27:29.000Z",
>             "name": "default.after_changes_13@primary",
>             "owner": "apathan",
>             "parameters": {
>                 "transient_lastDdlTime": "1461662849"
>             },
>             "partitionKeys": [
>                 {
>                     "id": {
>                         "id": "84db2789-82b3-460e-a827-ee8f94c786cc",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "ds",
>                         "qualifiedName": "default.after_changes_13.ds@primary",
>                         "type": "string"
>                     }
>                 }
>             ],
>             "retention": 0,
>             "sd": {
>                 "id": {
>                     "id": "5ebc8823-ecab-4c2d-8f0d-17a861984070",
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                     "state": "ACTIVE",
>                     "typeName": "hive_storagedesc",
>                     "version": 0
>                 },
>                 "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                 "traitNames": [],
>                 "traits": {},
>                 "typeName": "hive_storagedesc",
>                 "values": {
>                     "bucketCols": null,
>                     "compressed": false,
>                     "inputFormat": "org.apache.hadoop.mapred.TextInputFormat",
>                     "location": "hdfs://localhost:9000/user/hive/warehouse/after_changes_13",
>                     "numBuckets": -1,
>                     "outputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
>                     "parameters": null,
>                     "qualifiedName": "default.after_changes_13@primary_storage",
>                     "serdeInfo": {
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
>                         "typeName": "hive_serde",
>                         "values": {
>                             "name": null,
>                             "parameters": {
>                                 "serialization.format": "1"
>                             },
>                             "serializationLib": "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"
>                         }
>                     },
>                     "sortCols": null,
>                     "storedAsSubDirectories": false
>                 }
>             },
>             "tableName": "after_changes_13",
>             "tableType": "MANAGED_TABLE",
>             "temporary": false,
>             "viewExpandedText": null,
>             "viewOriginalText": null
>         }
>     },
>     "requestId": "qtp835142742-2359 - ccef18a2-916c-48b1-b629-a330c5689a69"
> }
> {noformat}
> After deleting the table, db and storage descriptor references are missing from the table
entity response.
> {noformat}
> apathan@Ayub-Pathans-Macbook-Pro:~ > curl 'http://localhost:21000/api/atlas/entities/1b11b234-7360-4679-93f9-d1c01f839f79'
-H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112
Safari/537.36' -H 'Accept: application/json, text/plain, */*' -H 'Referer: http://localhost:21000/index.html'
-H 'Cookie: JSESSIONID=3frbae3qwzi2xkdjyvz8nyko' -H 'Connection: keep-alive' --compressed
| python -m json.tool
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100  2142    0  2142    0     0  51088      0 --:--:-- --:--:-- --:--:-- 52243
> {
>     "GUID": "1b11b234-7360-4679-93f9-d1c01f839f79",
>     "definition": {
>         "id": {
>             "id": "1b11b234-7360-4679-93f9-d1c01f839f79",
>             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>             "state": "DELETED",
>             "typeName": "hive_table",
>             "version": 0
>         },
>         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>         "traitNames": [],
>         "traits": {},
>         "typeName": "hive_table",
>         "values": {
>             "columns": [
>                 {
>                     "id": {
>                         "id": "2c50426e-7db3-4912-9bbc-13150e962ed5",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "DELETED",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "name",
>                         "qualifiedName": "default.after_changes_13.name@primary",
>                         "type": "string"
>                     }
>                 },
>                 {
>                     "id": {
>                         "id": "2af81d25-bbcd-44bd-95b8-4fee915c9dba",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "DELETED",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "age",
>                         "qualifiedName": "default.after_changes_13.age@primary",
>                         "type": "int"
>                     }
>                 }
>             ],
>             "comment": null,
>             "createTime": "2016-04-26T09:27:29.000Z",
>             "db": null,
>             "description": null,
>             "lastAccessTime": "2016-04-26T09:27:29.000Z",
>             "name": "default.after_changes_13@primary",
>             "owner": "apathan",
>             "parameters": {
>                 "transient_lastDdlTime": "1461662849"
>             },
>             "partitionKeys": [
>                 {
>                     "id": {
>                         "id": "84db2789-82b3-460e-a827-ee8f94c786cc",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "DELETED",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "ds",
>                         "qualifiedName": "default.after_changes_13.ds@primary",
>                         "type": "string"
>                     }
>                 }
>             ],
>             "retention": 0,
>             "sd": null,
>             "tableName": "after_changes_13",
>             "tableType": "MANAGED_TABLE",
>             "temporary": false,
>             "viewExpandedText": null,
>             "viewOriginalText": null
>         }
>     },
>     "requestId": "qtp835142742-2432 - efe83211-ba90-4193-a644-a9d50dc9d410"
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message