atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shwetha G S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ATLAS-877) CreateTime of an entity(for example hive table) changes for every update
Date Fri, 17 Jun 2016 17:53:05 GMT

    [ https://issues.apache.org/jira/browse/ATLAS-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336555#comment-15336555
] 

Shwetha G S commented on ATLAS-877:
-----------------------------------

+1

> CreateTime of an entity(for example hive table) changes for every update
> ------------------------------------------------------------------------
>
>                 Key: ATLAS-877
>                 URL: https://issues.apache.org/jira/browse/ATLAS-877
>             Project: Atlas
>          Issue Type: Bug
>    Affects Versions: 0.7-incubating
>            Reporter: Ayub Khan
>            Assignee: Suma Shivaprasad
>            Priority: Critical
>             Fix For: 0.7-incubating
>
>         Attachments: ATLAS-877.patch
>
>
> Each time a hive table is updated(example column renamed), create time of table is also
updated.
> UI snapshot: https://monosnap.com/file/6NoT86CNpHgQmlyfBQTSUa1y3Ncc2Y
> hive table metadata snapshot from hive cli
> {noformat}
> 0: jdbc:hive2://localhost:10000/default> describe formatted column_changepgwcbgq1bd;
> OK
> +-------------------------------+-------------------------------------------------------------------------------------+-----------------------+--+
> |           col_name            |                                      data_type    
                                 |        comment        |
> +-------------------------------+-------------------------------------------------------------------------------------+-----------------------+--+
> | # col_name                    | data_type                                         
                                 | comment               |
> |                               | NULL                                              
                                 | NULL                  |
> | a3                            | int                                               
                                 |                       |
> | b                             | int                                               
                                 |                       |
> | c                             | int                                               
                                 |                       |
> | d                             | int                                               
                                 |                       |
> | e                             | int                                               
                                 |                       |
> | f                             | int                                               
                                 |                       |
> |                               | NULL                                              
                                 | NULL                  |
> | # Detailed Table Information  | NULL                                              
                                 | NULL                  |
> | Database:                     | db2pgwcbgq1bd                                     
                                 | NULL                  |
> | Owner:                        | apathan                                           
                                 | NULL                  |
> | CreateTime:                   | Wed Jun 08 12:41:28 IST 2016                      
                                 | NULL                  |
> | LastAccessTime:               | UNKNOWN                                           
                                 | NULL                  |
> | Protect Mode:                 | None                                              
                                 | NULL                  |
> | Retention:                    | 0                                                 
                                 | NULL                  |
> | Location:                     | hdfs://localhost:9000/user/hive/warehouse/db2pgwcbgq1bd.db/column_changepgwcbgq1bd
 | NULL                  |
> | Table Type:                   | MANAGED_TABLE                                     
                                 | NULL                  |
> | Table Parameters:             | NULL                                              
                                 | NULL                  |
> |                               | COLUMN_STATS_ACCURATE                             
                                 | false                 |
> |                               | last_modified_by                                  
                                 | apathan               |
> |                               | last_modified_time                                
                                 | 1465370834            |
> |                               | numFiles                                          
                                 | 0                     |
> |                               | numRows                                           
                                 | -1                    |
> |                               | rawDataSize                                       
                                 | -1                    |
> |                               | totalSize                                         
                                 | 0                     |
> |                               | transient_lastDdlTime                             
                                 | 1465370834            |
> |                               | NULL                                              
                                 | NULL                  |
> | # Storage Information         | NULL                                              
                                 | NULL                  |
> | SerDe Library:                | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                                 | NULL                  |
> | InputFormat:                  | org.apache.hadoop.mapred.TextInputFormat          
                                 | NULL                  |
> | OutputFormat:                 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                         | NULL                  |
> | Compressed:                   | No                                                
                                 | NULL                  |
> | Num Buckets:                  | -1                                                
                                 | NULL                  |
> | Bucket Columns:               | []                                                
                                 | NULL                  |
> | Sort Columns:                 | []                                                
                                 | NULL                  |
> | Storage Desc Params:          | NULL                                              
                                 | NULL                  |
> |                               | serialization.format                              
                                 | 1                     |
> +-------------------------------+-------------------------------------------------------------------------------------+-----------------------+--+
> 38 rows selected (0.111 seconds)
> {noformat}
> ATlas respones for entityquery 
> {noformat}
> curl 'http://localhost:21000/api/atlas/entities/6a99143d-e3dd-444c-b89c-efbfc5097569'
-H 'Cookie: JSESSIONID=12rn8aybkawa5ewiqdb2hv663' -H 'Accept-Encoding: gzip, deflate, sdch'
-H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS
X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36' -H 'Accept:
application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:21000/index.html'
-H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' --compressed | python -m
json.tool
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100  6820    0  6820    0     0   4293      0 --:--:--  0:00:01 --:--:--  4292
> {
>     "definition": {
>         "id": {
>             "id": "6a99143d-e3dd-444c-b89c-efbfc5097569",
>             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>             "state": "ACTIVE",
>             "typeName": "hive_table",
>             "version": 0
>         },
>         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>         "traitNames": [
>             "addTraitTrait63ozq0yxh3"
>         ],
>         "traits": {
>             "addTraitTrait63ozq0yxh3": {
>                 "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
>                 "typeName": "addTraitTrait63ozq0yxh3",
>                 "values": {}
>             }
>         },
>         "typeName": "hive_table",
>         "values": {
>             "aliases": null,
>             "columns": [
>                 {
>                     "id": {
>                         "id": "c12d4d8f-fc6d-41eb-9413-e09c12445078",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "a3",
>                         "qualifiedName": "db2pgwcbgq1bd.column_changepgwcbgq1bd.a3@primary",
>                         "table": {
>                             "id": "6a99143d-e3dd-444c-b89c-efbfc5097569",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "int"
>                     }
>                 },
>                 {
>                     "id": {
>                         "id": "a3e6dc35-d6f4-4429-9342-065206f879f9",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "b",
>                         "qualifiedName": "db2pgwcbgq1bd.column_changepgwcbgq1bd.b@primary",
>                         "table": {
>                             "id": "6a99143d-e3dd-444c-b89c-efbfc5097569",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "int"
>                     }
>                 },
>                 {
>                     "id": {
>                         "id": "2b73f828-6660-4caf-a632-cd53a78d9afb",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "c",
>                         "qualifiedName": "db2pgwcbgq1bd.column_changepgwcbgq1bd.c@primary",
>                         "table": {
>                             "id": "6a99143d-e3dd-444c-b89c-efbfc5097569",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "int"
>                     }
>                 },
>                 {
>                     "id": {
>                         "id": "6cd10281-5ca1-4c38-a58d-0ee09d04e7d4",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "d",
>                         "qualifiedName": "db2pgwcbgq1bd.column_changepgwcbgq1bd.d@primary",
>                         "table": {
>                             "id": "6a99143d-e3dd-444c-b89c-efbfc5097569",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "int"
>                     }
>                 },
>                 {
>                     "id": {
>                         "id": "91001553-dbb3-4401-aec9-8f9163a98536",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "e",
>                         "qualifiedName": "db2pgwcbgq1bd.column_changepgwcbgq1bd.e@primary",
>                         "table": {
>                             "id": "6a99143d-e3dd-444c-b89c-efbfc5097569",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "int"
>                     }
>                 },
>                 {
>                     "id": {
>                         "id": "4c231875-8a8d-491a-9126-0d6acbf0955f",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "f",
>                         "qualifiedName": "db2pgwcbgq1bd.column_changepgwcbgq1bd.f@primary",
>                         "table": {
>                             "id": "6a99143d-e3dd-444c-b89c-efbfc5097569",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "int"
>                     }
>                 },
>                 {
>                     "id": {
>                         "id": "b6d4b76e-2c1f-46d9-aecb-a086a750ae30",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "DELETED",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "a",
>                         "qualifiedName": "db2pgwcbgq1bd.column_changepgwcbgq1bd.a@primary",
>                         "table": {
>                             "id": "6a99143d-e3dd-444c-b89c-efbfc5097569",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "int"
>                     }
>                 }
>             ],
>             "comment": null,
>             "createTime": "2016-06-08T07:27:14.000Z",
>             "db": {
>                 "id": "d49df889-2c36-4346-9adc-6234a0045bfe",
>                 "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                 "state": "ACTIVE",
>                 "typeName": "hive_db",
>                 "version": 0
>             },
>             "description": null,
>             "lastAccessTime": "2016-06-08T07:27:14.000Z",
>             "name": "db2pgwcbgq1bd.column_changepgwcbgq1bd@primary",
>             "owner": "apathan",
>             "parameters": {
>                 "COLUMN_STATS_ACCURATE": "false",
>                 "last_modified_by": "apathan",
>                 "last_modified_time": "1465370834",
>                 "numFiles": "0",
>                 "numRows": "-1",
>                 "rawDataSize": "-1",
>                 "totalSize": "0",
>                 "transient_lastDdlTime": "1465370834"
>             },
>             "partitionKeys": null,
>             "retention": 0,
>             "sd": {
>                 "id": {
>                     "id": "55257985-2d63-4517-a612-2aa0b6edf36f",
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                     "state": "ACTIVE",
>                     "typeName": "hive_storagedesc",
>                     "version": 0
>                 },
>                 "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                 "traitNames": [],
>                 "traits": {},
>                 "typeName": "hive_storagedesc",
>                 "values": {
>                     "bucketCols": null,
>                     "compressed": false,
>                     "inputFormat": "org.apache.hadoop.mapred.TextInputFormat",
>                     "location": "hdfs://localhost:9000/user/hive/warehouse/db2pgwcbgq1bd.db/column_changepgwcbgq1bd",
>                     "numBuckets": -1,
>                     "outputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
>                     "parameters": null,
>                     "qualifiedName": "db2pgwcbgq1bd.column_changepgwcbgq1bd@primary_storage",
>                     "serdeInfo": {
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
>                         "typeName": "hive_serde",
>                         "values": {
>                             "name": null,
>                             "parameters": {
>                                 "serialization.format": "1"
>                             },
>                             "serializationLib": "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"
>                         }
>                     },
>                     "sortCols": null,
>                     "storedAsSubDirectories": false,
>                     "table": {
>                         "id": "6a99143d-e3dd-444c-b89c-efbfc5097569",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_table",
>                         "version": 0
>                     }
>                 }
>             },
>             "tableName": "column_changepgwcbgq1bd",
>             "tableType": "MANAGED_TABLE",
>             "temporary": false,
>             "viewExpandedText": null,
>             "viewOriginalText": null
>         }
>     },
>     "requestId": "qtp1211076369-4597 - 923515ee-bb5c-41ac-b395-0d9f37e6ece8"
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message