atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suma Shivaprasad (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (ATLAS-758) hdfs location of hive table is pointing to old location even after rename.
Date Tue, 10 May 2016 17:56:12 GMT

     [ https://issues.apache.org/jira/browse/ATLAS-758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Suma Shivaprasad updated ATLAS-758:
-----------------------------------
    Attachment: ATLAS-758.patch

> hdfs location of hive table is pointing to old location even after rename.
> --------------------------------------------------------------------------
>
>                 Key: ATLAS-758
>                 URL: https://issues.apache.org/jira/browse/ATLAS-758
>             Project: Atlas
>          Issue Type: Bug
>    Affects Versions: 0.7-incubating
>            Reporter: Ayub Khan
>            Assignee: Suma Shivaprasad
>            Priority: Critical
>             Fix For: 0.7-incubating
>
>         Attachments: ATLAS-758.patch, bug.log
>
>
> hdfs location of hive table is pointing to old location even after rename.
> Steps to repro:
> {noformat}
> 0: jdbc:hive2://localhost:10000/default> create database db_temporary;
> OK
> No rows affected (0.281 seconds)
> 0: jdbc:hive2://localhost:10000/default> use db_temporary;
> OK
> No rows affected (0.035 seconds)
> 0: jdbc:hive2://localhost:10000/default> create table temp_table (name String);
> OK
> No rows affected (0.204 seconds)
> 0: jdbc:hive2://localhost:10000/default> describe formatted temp_table;
> OK
> +-------------------------------+-----------------------------------------------------------------------+-----------------------+--+
> |           col_name            |                               data_type           
                   |        comment        |
> +-------------------------------+-----------------------------------------------------------------------+-----------------------+--+
> | # col_name                    | data_type                                         
                   | comment               |
> |                               | NULL                                              
                   | NULL                  |
> | name                          | string                                            
                   |                       |
> |                               | NULL                                              
                   | NULL                  |
> | # Detailed Table Information  | NULL                                              
                   | NULL                  |
> | Database:                     | db_temporary                                      
                   | NULL                  |
> | Owner:                        | apathan                                           
                   | NULL                  |
> | CreateTime:                   | Sat May 07 18:02:47 IST 2016                      
                   | NULL                  |
> | LastAccessTime:               | UNKNOWN                                           
                   | NULL                  |
> | Protect Mode:                 | None                                              
                   | NULL                  |
> | Retention:                    | 0                                                 
                   | NULL                  |
> | Location:                     | hdfs://localhost:9000/user/hive/warehouse/db_temporary.db/temp_table
 | NULL                  |
> | Table Type:                   | MANAGED_TABLE                                     
                   | NULL                  |
> | Table Parameters:             | NULL                                              
                   | NULL                  |
> |                               | transient_lastDdlTime                             
                   | 1462624367            |
> |                               | NULL                                              
                   | NULL                  |
> | # Storage Information         | NULL                                              
                   | NULL                  |
> | SerDe Library:                | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                   | NULL                  |
> | InputFormat:                  | org.apache.hadoop.mapred.TextInputFormat          
                   | NULL                  |
> | OutputFormat:                 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
           | NULL                  |
> | Compressed:                   | No                                                
                   | NULL                  |
> | Num Buckets:                  | -1                                                
                   | NULL                  |
> | Bucket Columns:               | []                                                
                   | NULL                  |
> | Sort Columns:                 | []                                                
                   | NULL                  |
> | Storage Desc Params:          | NULL                                              
                   | NULL                  |
> |                               | serialization.format                              
                   | 1                     |
> +-------------------------------+-----------------------------------------------------------------------+-----------------------+--+
> 26 rows selected (0.119 seconds)
> 0: jdbc:hive2://localhost:10000/default> alter table db_temporary.temp_table rename
to default.temp_table;
> OK
> No rows affected (0.189 seconds)
> 0: jdbc:hive2://localhost:10000/default> describe formatted default.temp_table;
> OK
> +-------------------------------+-------------------------------------------------------------+-----------------------+--+
> |           col_name            |                          data_type                
         |        comment        |
> +-------------------------------+-------------------------------------------------------------+-----------------------+--+
> | # col_name                    | data_type                                         
         | comment               |
> |                               | NULL                                              
         | NULL                  |
> | name                          | string                                            
         |                       |
> |                               | NULL                                              
         | NULL                  |
> | # Detailed Table Information  | NULL                                              
         | NULL                  |
> | Database:                     | default                                           
         | NULL                  |
> | Owner:                        | apathan                                           
         | NULL                  |
> | CreateTime:                   | Sat May 07 18:02:47 IST 2016                      
         | NULL                  |
> | LastAccessTime:               | UNKNOWN                                           
         | NULL                  |
> | Protect Mode:                 | None                                              
         | NULL                  |
> | Retention:                    | 0                                                 
         | NULL                  |
> | Location:                     | hdfs://localhost:9000/user/hive/warehouse/temp_table
       | NULL                  |
> | Table Type:                   | MANAGED_TABLE                                     
         | NULL                  |
> | Table Parameters:             | NULL                                              
         | NULL                  |
> |                               | last_modified_by                                  
         | apathan               |
> |                               | last_modified_time                                
         | 1462624451            |
> |                               | transient_lastDdlTime                             
         | 1462624451            |
> |                               | NULL                                              
         | NULL                  |
> | # Storage Information         | NULL                                              
         | NULL                  |
> | SerDe Library:                | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
         | NULL                  |
> | InputFormat:                  | org.apache.hadoop.mapred.TextInputFormat          
         | NULL                  |
> | OutputFormat:                 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
 | NULL                  |
> | Compressed:                   | No                                                
         | NULL                  |
> | Num Buckets:                  | -1                                                
         | NULL                  |
> | Bucket Columns:               | []                                                
         | NULL                  |
> | Sort Columns:                 | []                                                
         | NULL                  |
> | Storage Desc Params:          | NULL                                              
         | NULL                  |
> |                               | serialization.format                              
         | 1                     |
> +-------------------------------+-------------------------------------------------------------+-----------------------+--+
> 28 rows selected (0.095 seconds)
> 0: jdbc:hive2://localhost:10000/default>
> {noformat}
> Entity response still pointing to the old location.
> {noformat}
> curl http://localhost:21000/api/atlas/entities/e19c1d7a-d16a-4129-8cb4-e7916ac5d284?user.name=apathan?user.name=apathan
| python -m json.tool
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100  2781    0  2781    0     0   1793      0 --:--:--  0:00:01 --:--:--  1793
> {
>     "GUID": "e19c1d7a-d16a-4129-8cb4-e7916ac5d284",
>     "definition": {
>         "id": {
>             "id": "e19c1d7a-d16a-4129-8cb4-e7916ac5d284",
>             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>             "state": "ACTIVE",
>             "typeName": "hive_table",
>             "version": 0
>         },
>         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>         "traitNames": [],
>         "traits": {},
>         "typeName": "hive_table",
>         "values": {
>             "columns": [
>                 {
>                     "id": {
>                         "id": "15db117f-d4fb-4b6a-b7f3-308f96107728",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_column",
>                         "version": 0
>                     },
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                     "traitNames": [],
>                     "traits": {},
>                     "typeName": "hive_column",
>                     "values": {
>                         "comment": null,
>                         "name": "name",
>                         "qualifiedName": "default.temp_table.name@primary",
>                         "table": {
>                             "id": "e19c1d7a-d16a-4129-8cb4-e7916ac5d284",
>                             "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                             "state": "ACTIVE",
>                             "typeName": "hive_table",
>                             "version": 0
>                         },
>                         "type": "string"
>                     }
>                 }
>             ],
>             "comment": null,
>             "createTime": "2016-05-07T12:32:47.000Z",
>             "db": {
>                 "id": "802c10fe-10b3-423e-a587-19c3d45a3ca5",
>                 "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                 "state": "ACTIVE",
>                 "typeName": "hive_db",
>                 "version": 0
>             },
>             "description": null,
>             "lastAccessTime": "2016-05-07T12:32:47.000Z",
>             "name": "default.temp_table@primary",
>             "owner": "apathan",
>             "parameters": {
>                 "transient_lastDdlTime": "1462624367"
>             },
>             "partitionKeys": null,
>             "retention": 0,
>             "sd": {
>                 "id": {
>                     "id": "443d9aef-22e0-429d-9117-8d775442bb70",
>                     "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                     "state": "ACTIVE",
>                     "typeName": "hive_storagedesc",
>                     "version": 0
>                 },
>                 "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
>                 "traitNames": [],
>                 "traits": {},
>                 "typeName": "hive_storagedesc",
>                 "values": {
>                     "bucketCols": null,
>                     "compressed": false,
>                     "inputFormat": "org.apache.hadoop.mapred.TextInputFormat",
>                     "location": "hdfs://localhost:9000/user/hive/warehouse/db_temporary.db/temp_table",
>                     "numBuckets": -1,
>                     "outputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
>                     "parameters": null,
>                     "qualifiedName": "default.temp_table@primary_storage",
>                     "serdeInfo": {
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
>                         "typeName": "hive_serde",
>                         "values": {
>                             "name": null,
>                             "parameters": {
>                                 "serialization.format": "1"
>                             },
>                             "serializationLib": "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"
>                         }
>                     },
>                     "sortCols": null,
>                     "storedAsSubDirectories": false,
>                     "table": {
>                         "id": "e19c1d7a-d16a-4129-8cb4-e7916ac5d284",
>                         "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
>                         "state": "ACTIVE",
>                         "typeName": "hive_table",
>                         "version": 0
>                     }
>                 }
>             },
>             "tableName": "temp_table",
>             "tableType": "MANAGED_TABLE",
>             "temporary": false,
>             "viewExpandedText": null,
>             "viewOriginalText": null
>         }
>     },
>     "requestId": "qtp1307096070-6343 - d1ee28d0-f533-464e-b075-cd9a5116dbc8"
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message