atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vimal Sharma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ATLAS-760) Missing table parameters(last_modified_by and last_modified_time)
Date Mon, 30 May 2016 06:25:12 GMT

    [ https://issues.apache.org/jira/browse/ATLAS-760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306284#comment-15306284
] 

Vimal Sharma commented on ATLAS-760:
------------------------------------

I have submit a patch with suggested additions to table attributes i.e last_modified_by and
last_modified_time.
[~suma.shivaprasad] Can you please review the patch

> Missing table parameters(last_modified_by and last_modified_time)
> -----------------------------------------------------------------
>
>                 Key: ATLAS-760
>                 URL: https://issues.apache.org/jira/browse/ATLAS-760
>             Project: Atlas
>          Issue Type: Bug
>    Affects Versions: 0.7-incubating
>            Reporter: Ayub Khan
>            Assignee: Vimal Sharma
>            Priority: Critical
>             Fix For: 0.7-incubating
>
>         Attachments: ATLAS-760.patch
>
>
> Missing table parameters(last_modified_by and last_modified_time)
> Hive table metadata showing vboth params..
> {noformat}
> 0: jdbc:hive2://localhost:10000/default> describe formatted table3_s1vxgkt0bh;
> OK
> +-------------------------------+--------------------------------------------------------------+--------------------------------+--+
> |           col_name            |                          data_type                
          |            comment             |
> +-------------------------------+--------------------------------------------------------------+--------------------------------+--+
> | # col_name                    | data_type                                         
          | comment                        |
> |                               | NULL                                              
          | NULL                           |
> | viewtime                      | int                                               
          |                                |
> | userid                        | bigint                                            
          |                                |
> | page_url                      | string                                            
          |                                |
> | referrer_url                  | string                                            
          |                                |
> | ip                            | string                                            
          |                                |
> |                               | NULL                                              
          | NULL                           |
> | # Partition Information       | NULL                                              
          | NULL                           |
> | # col_name                    | data_type                                         
          | comment                        |
> |                               | NULL                                              
          | NULL                           |
> | dt                            | string                                            
          |                                |
> | country                       | string                                            
          | partitioned columns comments.  |
> |                               | NULL                                              
          | NULL                           |
> | # Detailed Table Information  | NULL                                              
          | NULL                           |
> | Database:                     | default                                           
          | NULL                           |
> | Owner:                        | apathan                                           
          | NULL                           |
> | CreateTime:                   | Mon May 09 14:42:34 IST 2016                      
          | NULL                           |
> | LastAccessTime:               | UNKNOWN                                           
          | NULL                           |
> | Protect Mode:                 | None                                              
          | NULL                           |
> | Retention:                    | 0                                                 
          | NULL                           |
> | Location:                     | hdfs://localhost:9000/user/hive/warehouse/table3_s1vxgkt0bh
 | NULL                           |
> | Table Type:                   | MANAGED_TABLE                                     
          | NULL                           |
> | Table Parameters:             | NULL                                              
          | NULL                           |
> |                               | last_modified_by                                  
          | apathan                        |
> |                               | last_modified_time                                
          | 1462785155                     |
> |                               | transient_lastDdlTime                             
          | 1462785155                     |
> |                               | NULL                                              
          | NULL                           |
> | # Storage Information         | NULL                                              
          | NULL                           |
> | SerDe Library:                | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
          | NULL                           |
> | InputFormat:                  | org.apache.hadoop.mapred.TextInputFormat          
          | NULL                           |
> | OutputFormat:                 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
  | NULL                           |
> | Compressed:                   | No                                                
          | NULL                           |
> | Num Buckets:                  | -1                                                
          | NULL                           |
> | Bucket Columns:               | []                                                
          | NULL                           |
> | Sort Columns:                 | []                                                
          | NULL                           |
> | Storage Desc Params:          | NULL                                              
          | NULL                           |
> |                               | serialization.format                              
          | 1                              |
> +-------------------------------+--------------------------------------------------------------+--------------------------------+--+
> 38 rows selected (0.101 seconds)
> {noformat}
> Entity api response showing missing parameters
> {noformat}
> curl http://localhost:21000/api/atlas/entities/b17b2c93-dbf7-4592-b59e-4614c6e3d37e |
python -m json.tool
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100  6508    0  6508    0     0   4152      0 --:--:--  0:00:01 --:--:--  4150
> {
>     "GUID": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
>     "definition": {
>         "id": {
>             "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
>             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>             "state": "ACTIVE",
>             "typeName": "hive_table",
>             "version": 0
>         },
>         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>         "traitNames": [],
>         "traits": {},
>         "typeName": "hive_table",
>         "values": {
>             "columns": [
>                 {
>                     "id": {
>                         "id": "f118ccf4-bd8b-4a0e-af35-17a5c5018e06",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "viewtime",
>                         "qualifiedName": "default.table3_s1vxgkt0bh.viewtime@primary",
>                         "table": {
>                             "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "int"
>                     }
>                 },
>                 {
>                     "id": {
>                         "id": "4b0d47e1-16ba-4cdd-85a5-6f35a968c0a3",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "userid",
>                         "qualifiedName": "default.table3_s1vxgkt0bh.userid@primary",
>                         "table": {
>                             "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "bigint"
>                     }
>                 },
>                 {
>                     "id": {
>                         "id": "ca652a37-a5b8-4563-be27-4c6adb20f2d9",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "page_url",
>                         "qualifiedName": "default.table3_s1vxgkt0bh.page_url@primary",
>                         "table": {
>                             "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "string"
>                     }
>                 },
>                 {
>                     "id": {
>                         "id": "050d14cb-6434-4230-896e-063c6df04de9",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "referrer_url",
>                         "qualifiedName": "default.table3_s1vxgkt0bh.referrer_url@primary",
>                         "table": {
>                             "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "string"
>                     }
>                 },
>                 {
>                     "id": {
>                         "id": "39d7eede-e6e1-4a82-ab4a-0dbbe211cd42",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "ip",
>                         "qualifiedName": "default.table3_s1vxgkt0bh.ip@primary",
>                         "table": {
>                             "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "string"
>                     }
>                 }
>             ],
>             "comment": null,
>             "createTime": "2016-05-09T09:12:34.000Z",
>             "db": {
>                 "id": "802c10fe-10b3-423e-a587-19c3d45a3ca5",
>                 "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                 "state": "ACTIVE",
>                 "typeName": "hive_db",
>                 "version": 0
>             },
>             "description": null,
>             "lastAccessTime": "2016-05-09T09:12:34.000Z",
>             "name": "default.table3_s1vxgkt0bh@primary",
>             "owner": "apathan",
>             "parameters": {
>                 "transient_lastDdlTime": "1462785154"
>             },
>             "partitionKeys": [
>                 {
>                     "id": {
>                         "id": "ba216f52-6ea0-4b00-831f-87edd9b3d6ab",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "dt",
>                         "qualifiedName": "default.table3_s1vxgkt0bh.dt@primary",
>                         "table": {
>                             "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "string"
>                     }
>                 },
>                 {
>                     "id": {
>                         "id": "fe1bcce2-5ad9-4506-9cd5-2db1e61d4de5",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": "partitioned columns comments.",
>                         "name": "country",
>                         "qualifiedName": "default.table3_s1vxgkt0bh.country@primary",
>                         "table": {
>                             "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "string"
>                     }
>                 }
>             ],
>             "retention": 0,
>             "sd": {
>                 "id": {
>                     "id": "5499b90c-570f-4911-abe4-cca0cc749db3",
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                     "state": "ACTIVE",
>                     "typeName": "hive_storagedesc",
>                     "version": 0
>                 },
>                 "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                 "traitNames": [],
>                 "traits": {},
>                 "typeName": "hive_storagedesc",
>                 "values": {
>                     "bucketCols": null,
>                     "compressed": false,
>                     "inputFormat": "org.apache.hadoop.mapred.TextInputFormat",
>                     "location": "hdfs://localhost:9000/user/hive/warehouse/db2s1vxgkt0bh.db/table2_s1vxgkt0bh",
>                     "numBuckets": -1,
>                     "outputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
>                     "parameters": null,
>                     "qualifiedName": "default.table3_s1vxgkt0bh@primary_storage",
>                     "serdeInfo": {
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
>                         "typeName": "hive_serde",
>                         "values": {
>                             "name": null,
>                             "parameters": {
>                                 "serialization.format": "1"
>                             },
>                             "serializationLib": "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"
>                         }
>                     },
>                     "sortCols": null,
>                     "storedAsSubDirectories": false,
>                     "table": {
>                         "id": "b17b2c93-dbf7-4592-b59e-4614c6e3d37e",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_table",
>                         "version": 0
>                     }
>                 }
>             },
>             "tableName": "table3_s1vxgkt0bh",
>             "tableType": "MANAGED_TABLE",
>             "temporary": false,
>             "viewExpandedText": null,
>             "viewOriginalText": null
>         }
>     },
>     "requestId": "qtp1307096070-2007 - afa719f8-ad88-4293-a4d6-16c6a72190ed"
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message