atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ayub Khan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ATLAS-758) hdfs location of hive table is pointing to old location even after rename.
Date Sat, 07 May 2016 12:44:12 GMT
Ayub Khan created ATLAS-758:
-------------------------------

             Summary: hdfs location of hive table is pointing to old location even after rename.
                 Key: ATLAS-758
                 URL: https://issues.apache.org/jira/browse/ATLAS-758
             Project: Atlas
          Issue Type: Bug
    Affects Versions: 0.7-incubating
            Reporter: Ayub Khan
            Assignee: Suma Shivaprasad
            Priority: Critical
             Fix For: 0.7-incubating


hdfs location of hive table is pointing to old location even after rename.

Steps to repro:
{noformat}
0: jdbc:hive2://localhost:10000/default> create database db_temporary;
OK
No rows affected (0.281 seconds)
0: jdbc:hive2://localhost:10000/default> use db_temporary;
OK
No rows affected (0.035 seconds)
0: jdbc:hive2://localhost:10000/default> create table temp_table (name String);
OK
No rows affected (0.204 seconds)
0: jdbc:hive2://localhost:10000/default> describe formatted temp_table;
OK
+-------------------------------+-----------------------------------------------------------------------+-----------------------+--+
|           col_name            |                               data_type                
              |        comment        |
+-------------------------------+-----------------------------------------------------------------------+-----------------------+--+
| # col_name                    | data_type                                              
              | comment               |
|                               | NULL                                                   
              | NULL                  |
| name                          | string                                                 
              |                       |
|                               | NULL                                                   
              | NULL                  |
| # Detailed Table Information  | NULL                                                   
              | NULL                  |
| Database:                     | db_temporary                                           
              | NULL                  |
| Owner:                        | apathan                                                
              | NULL                  |
| CreateTime:                   | Sat May 07 18:02:47 IST 2016                           
              | NULL                  |
| LastAccessTime:               | UNKNOWN                                                
              | NULL                  |
| Protect Mode:                 | None                                                   
              | NULL                  |
| Retention:                    | 0                                                      
              | NULL                  |
| Location:                     | hdfs://localhost:9000/user/hive/warehouse/db_temporary.db/temp_table
 | NULL                  |
| Table Type:                   | MANAGED_TABLE                                          
              | NULL                  |
| Table Parameters:             | NULL                                                   
              | NULL                  |
|                               | transient_lastDdlTime                                  
              | 1462624367            |
|                               | NULL                                                   
              | NULL                  |
| # Storage Information         | NULL                                                   
              | NULL                  |
| SerDe Library:                | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe     
              | NULL                  |
| InputFormat:                  | org.apache.hadoop.mapred.TextInputFormat               
              | NULL                  |
| OutputFormat:                 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
           | NULL                  |
| Compressed:                   | No                                                     
              | NULL                  |
| Num Buckets:                  | -1                                                     
              | NULL                  |
| Bucket Columns:               | []                                                     
              | NULL                  |
| Sort Columns:                 | []                                                     
              | NULL                  |
| Storage Desc Params:          | NULL                                                   
              | NULL                  |
|                               | serialization.format                                   
              | 1                     |
+-------------------------------+-----------------------------------------------------------------------+-----------------------+--+
26 rows selected (0.119 seconds)
0: jdbc:hive2://localhost:10000/default> alter table db_temporary.temp_table rename to
default.temp_table;
OK
No rows affected (0.189 seconds)
0: jdbc:hive2://localhost:10000/default> describe formatted default.temp_table;
OK
+-------------------------------+-------------------------------------------------------------+-----------------------+--+
|           col_name            |                          data_type                     
    |        comment        |
+-------------------------------+-------------------------------------------------------------+-----------------------+--+
| # col_name                    | data_type                                              
    | comment               |
|                               | NULL                                                   
    | NULL                  |
| name                          | string                                                 
    |                       |
|                               | NULL                                                   
    | NULL                  |
| # Detailed Table Information  | NULL                                                   
    | NULL                  |
| Database:                     | default                                                
    | NULL                  |
| Owner:                        | apathan                                                
    | NULL                  |
| CreateTime:                   | Sat May 07 18:02:47 IST 2016                           
    | NULL                  |
| LastAccessTime:               | UNKNOWN                                                
    | NULL                  |
| Protect Mode:                 | None                                                   
    | NULL                  |
| Retention:                    | 0                                                      
    | NULL                  |
| Location:                     | hdfs://localhost:9000/user/hive/warehouse/temp_table   
    | NULL                  |
| Table Type:                   | MANAGED_TABLE                                          
    | NULL                  |
| Table Parameters:             | NULL                                                   
    | NULL                  |
|                               | last_modified_by                                       
    | apathan               |
|                               | last_modified_time                                     
    | 1462624451            |
|                               | transient_lastDdlTime                                  
    | 1462624451            |
|                               | NULL                                                   
    | NULL                  |
| # Storage Information         | NULL                                                   
    | NULL                  |
| SerDe Library:                | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe     
    | NULL                  |
| InputFormat:                  | org.apache.hadoop.mapred.TextInputFormat               
    | NULL                  |
| OutputFormat:                 | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
 | NULL                  |
| Compressed:                   | No                                                     
    | NULL                  |
| Num Buckets:                  | -1                                                     
    | NULL                  |
| Bucket Columns:               | []                                                     
    | NULL                  |
| Sort Columns:                 | []                                                     
    | NULL                  |
| Storage Desc Params:          | NULL                                                   
    | NULL                  |
|                               | serialization.format                                   
    | 1                     |
+-------------------------------+-------------------------------------------------------------+-----------------------+--+
28 rows selected (0.095 seconds)
0: jdbc:hive2://localhost:10000/default>
{noformat}

Entity response still pointing to the old location.
{noformat}
curl http://localhost:21000/api/atlas/entities/e19c1d7a-d16a-4129-8cb4-e7916ac5d284?user.name=apathan?user.name=apathan
| python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2781    0  2781    0     0   1793      0 --:--:--  0:00:01 --:--:--  1793
{
    "GUID": "e19c1d7a-d16a-4129-8cb4-e7916ac5d284",
    "definition": {
        "id": {
            "id": "e19c1d7a-d16a-4129-8cb4-e7916ac5d284",
            "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
            "state": "ACTIVE",
            "typeName": "hive_table",
            "version": 0
        },
        "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
        "traitNames": [],
        "traits": {},
        "typeName": "hive_table",
        "values": {
            "columns": [
                {
                    "id": {
                        "id": "15db117f-d4fb-4b6a-b7f3-308f96107728",
                        "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
                        "state": "ACTIVE",
                        "typeName": "hive_column",
                        "version": 0
                    },
                    "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
                    "traitNames": [],
                    "traits": {},
                    "typeName": "hive_column",
                    "values": {
                        "comment": null,
                        "name": "name",
                        "qualifiedName": "default.temp_table.name@primary",
                        "table": {
                            "id": "e19c1d7a-d16a-4129-8cb4-e7916ac5d284",
                            "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
                            "state": "ACTIVE",
                            "typeName": "hive_table",
                            "version": 0
                        },
                        "type": "string"
                    }
                }
            ],
            "comment": null,
            "createTime": "2016-05-07T12:32:47.000Z",
            "db": {
                "id": "802c10fe-10b3-423e-a587-19c3d45a3ca5",
                "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
                "state": "ACTIVE",
                "typeName": "hive_db",
                "version": 0
            },
            "description": null,
            "lastAccessTime": "2016-05-07T12:32:47.000Z",
            "name": "default.temp_table@primary",
            "owner": "apathan",
            "parameters": {
                "transient_lastDdlTime": "1462624367"
            },
            "partitionKeys": null,
            "retention": 0,
            "sd": {
                "id": {
                    "id": "443d9aef-22e0-429d-9117-8d775442bb70",
                    "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
                    "state": "ACTIVE",
                    "typeName": "hive_storagedesc",
                    "version": 0
                },
                "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
                "traitNames": [],
                "traits": {},
                "typeName": "hive_storagedesc",
                "values": {
                    "bucketCols": null,
                    "compressed": false,
                    "inputFormat": "org.apache.hadoop.mapred.TextInputFormat",
                    "location": "hdfs://localhost:9000/user/hive/warehouse/db_temporary.db/temp_table",
                    "numBuckets": -1,
                    "outputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
                    "parameters": null,
                    "qualifiedName": "default.temp_table@primary_storage",
                    "serdeInfo": {
                        "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Struct",
                        "typeName": "hive_serde",
                        "values": {
                            "name": null,
                            "parameters": {
                                "serialization.format": "1"
                            },
                            "serializationLib": "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"
                        }
                    },
                    "sortCols": null,
                    "storedAsSubDirectories": false,
                    "table": {
                        "id": "e19c1d7a-d16a-4129-8cb4-e7916ac5d284",
                        "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
                        "state": "ACTIVE",
                        "typeName": "hive_table",
                        "version": 0
                    }
                }
            },
            "tableName": "temp_table",
            "tableType": "MANAGED_TABLE",
            "temporary": false,
            "viewExpandedText": null,
            "viewOriginalText": null
        }
    },
    "requestId": "qtp1307096070-6343 - d1ee28d0-f533-464e-b075-cd9a5116dbc8"
}
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message