spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gatorsmile <...@git.apache.org>
Subject [GitHub] spark issue #17394: [SPARK-20067] [SQL] Unify and Clean Up Desc Commands Usi...
Date Tue, 28 Mar 2017 04:31:13 GMT
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/17394
  
    After refactoring, I think `DESC FORMATTED` might be more useful for users.
    
    `DESC FORMATTED` outputs the metadata in a more formatted way. Users can query it using
the key value (stored in `col_name`) to get the result. They do not need to parse the string
to extract the data value, unlike `DESC EXTENDED`.
    
    Below is the output of `DESC FORMATTED `
    
    ```
    +----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+-------+
    |col_name                    |data_type                                              
                                                                                         
|comment|
    +----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+-------+
    |key                         |int                                                    
                                                                                         
|test   |
    |value                       |string                                                 
                                                                                         
|null   |
    |ds                          |string                                                 
                                                                                         
|null   |
    |hr                          |string                                                 
                                                                                         
|null   |
    |# Partition Information     |                                                       
                                                                                         
|       |
    |# col_name                  |data_type                                              
                                                                                         
|comment|
    |ds                          |string                                                 
                                                                                         
|null   |
    |hr                          |string                                                 
                                                                                         
|null   |
    |                            |                                                       
                                                                                         
|       |
    |# Detailed Table Information|                                                       
                                                                                         
|       |
    |Table                       |`default`.`tab1`                                       
                                                                                         
|       |
    |Owner                       |xiao                                                   
                                                                                         
|       |
    |Created                     |Mon Mar 27 21:25:07 PDT 2017                           
                                                                                         
|       |
    |Last Access                 |Wed Dec 31 16:00:00 PST 1969                           
                                                                                         
|       |
    |Type                        |EXTERNAL                                               
                                                                                         
|       |
    |Provider                    |hive                                                   
                                                                                         
|       |
    |Comment                     |Apache Spark                                           
                                                                                         
|       |
    |Properties                  |[serialization.format=1]                               
                                                                                         
|       |
    |Location                    |file:/private/var/folders/6r/15tqm8hn3ldb3rmbfqm1gf4c0000gn/T/spark-8d941400-8f59-4080-8094-611fa517f11e
                                         |       |
    |Serde Library               |org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe     
                                                                                         
|       |
    |InputFormat                 |org.apache.hadoop.mapred.TextInputFormat               
                                                                                         
|       |
    |OutputFormat                |org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                                                                                       | 
     |
    |Partition Provider          |Catalog                                                
                                                                                         
|       |
    |Partition Columns           |[`ds`, `hr`]                                           
                                                                                         
|       |
    |Schema                      |root
     |-- key: integer (nullable = true)
     |-- value: string (nullable = true)
     |-- ds: string (nullable = true)
     |-- hr: string (nullable = true)
    |       |
    +----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+-------+
    ```
    
    Below is the output of `DESC EXTENDED`
    
    ```
    +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-------+
    |col_name                                                                            
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
            |data_type|comment|
    +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-------+
    |key                                                                                 
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
            |int      |test   |
    |value                                                                               
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
            |string   |null   |
    |ds                                                                                  
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
            |string   |null   |
    |hr                                                                                  
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
            |string   |null   |
    |# Partition Information                                                             
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
            |         |       |
    |# col_name                                                                          
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
            |data_type|comment|
    |ds                                                                                  
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
            |string   |null   |
    |hr                                                                                  
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
            |string   |null   |
    |                                                                                    
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
            |         |       |
    |# Detailed Table Information                                                        
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
            |         |       |
    |Table: `default`.`tab1`
    Owner: xiao
    Created: Mon Mar 27 21:25:07 PDT 2017
    Last Access: Wed Dec 31 16:00:00 PST 1969
    Type: EXTERNAL
    Provider: hive
    Comment: Apache Spark
    Properties: [serialization.format=1]
    Location: file:/private/var/folders/6r/15tqm8hn3ldb3rmbfqm1gf4c0000gn/T/spark-8d941400-8f59-4080-8094-611fa517f11e
    Serde Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
    InputFormat: org.apache.hadoop.mapred.TextInputFormat
    OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
    Partition Provider: Catalog
    Partition Columns: [`ds`, `hr`]
    Schema: root
     |-- key: integer (nullable = true)
     |-- value: string (nullable = true)
     |-- ds: string (nullable = true)
     |-- hr: string (nullable = true)
    |         |       |
    +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+-------+
    
    
    ```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message