hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleksandr Diachenko <odiache...@pivotal.io>
Subject Re: [DISCUSS] Hive Complex types in \d
Date Mon, 25 Apr 2016 22:13:34 GMT
Using comments column to displaying Hive type might be an option as well,
but this column seems to me more suitable to display Hive's comments on
columns(Hive columns could have comments -
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL).
Comments are separate entities, so they have nothing common with types in
general.

Regards, Alex.



On Mon, Apr 25, 2016 at 2:16 PM, Shivram Mani <shivram.mani@gmail.com>
wrote:

> Option 2 is definitely a no for me as it breaks uniformity. We can't have
> different cases display different results for widely used commands such as
> \d
>
> Option 1 is preferable, but even here I wouldn't recommend introducing
> another column. Can't we instead leverage the description column
> (pg_catalog.col_description)
> Refer to http://www.postgresql.org/docs/9.1/static/sql-comment.html about
> this attribute. Built in function col_description exists to return this
> info.
>
> On Mon, Apr 25, 2016 at 1:35 PM, Noa Horn <noapjgr@gmail.com> wrote:
>
> > I vote for option 1.
> > I think \d should match the information displayed for other tables.
> > \d+ can be more flexible and contain more information that doesn't apply
> to
> > all objects.
> > For example, for a view the view statement will only be displayed with
> \d+.
> >
> >
> >
> > On 22 April 2016 at 16:19, Ting(Goden) Yao <tyao@pivotal.io> wrote:
> >
> > > i prefer option2 for both \d and \d+ we should tell users source types
> > and
> > > HAWQ types.
> > > This is more meaningful to them.
> > >
> > >
> > > On Fri, Apr 22, 2016 at 2:18 PM Oleksandr Diachenko <
> > odiachenko@pivotal.io
> > > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am working on https://issues.apache.org/jira/browse/HAWQ-703 and
> > > wanted
> > > > to get some opinions.
> > > >
> > > > As for now HAWQ supports \d for Hive tables with primitive types
> > > only(int,
> > > > boolean, date etc).
> > > > If user wants to describe Hive table which has some complex
> > types(array,
> > > > map, struct, uniontype) error occurs.
> > > >
> > > > *Example:*
> > > >
> > > > User have Hive table:
> > > >
> > > > *hive> describe hive_small_data;*
> > > >
> > > > *OK*
> > > >
> > > > *s1                  string                                  *
> > > >
> > > > *s2                  string                                  *
> > > >
> > > > *n1                  int                                     *
> > > >
> > > > *d1                  double*
> > > > User is able to describe this table in psql:
> > > >
> > > > *# \d hcatalog.default.hive_small_data*
> > > >
> > > > *PXF Hive Table "default.hive_small_data"*
> > > >
> > > > * Column |  Type  *
> > > >
> > > > *--------+--------*
> > > >
> > > > * s1     | text*
> > > >
> > > > * s2     | text*
> > > >
> > > > * n1     | int4*
> > > >
> > > > * d1     | float8*
> > > > Thus psql shows column and types mapped to HAWQ's types(string ->
> text,
> > > int
> > > > -> int4).
> > > >
> > > > The goal in HAWQ-703 - to be able to query Hive tables with complex
> > > > columns, represent those complex types as TEXT. Long term goal will
> be
> > > > mapping HIve's complex types to HAWQ's types when applicable.
> > > >
> > > > *Disclaimer*:
> > > >
> > > > Proposed changes will affect only Hive tables, behavior for all other
> > > > objects will remain the same.
> > > >
> > > > *The problems:*
> > > >
> > > >    1. Not all Hive complex types could be mapped to HAWQ types. For
> > > example
> > > >    - uniontype, there is no corresponding HAWQ type.
> > > >    2. Assuming fact that we do mapping Hive types to HAWQ types,
> > > displaying
> > > >    only HAWQ type in \d might be not enough for user and even
> > confusing.
> > > >
> > > >
> > > > *Possible options:*
> > > >
> > > >    1. \d behavior for Hive tables remains the same, column Type shows
> > > HAWQ
> > > >    type, \d+ shows additional column, Source Type, where Type=HAWQ
> > type,
> > > >    Source Type=Hive type.
> > > >       -
> > > >
> > > >       *# \d hcatalog.default.reg_collections;*
> > > >
> > > >       *PXF Hive Table "default.reg_collections"*
> > > >
> > > >       * Column |  Type  *
> > > >
> > > >       *------------+--------*
> > > >
> > > >       * s1          | text*
> > > >
> > > >       * f1           | float4*
> > > >
> > > >       * a1          | text*
> > > >
> > > >       * m1         | text*
> > > >
> > > >       * sr1         | text*
> > > >       -
> > > >
> > > >       *# \d+ hcatalog.default.reg_collections;*
> > > >
> > > >       *PXF Hive Table "default.reg_collections"*
> > > >
> > > >       * Column |  Type  | Source Type                          |*
> > > >
> > > >
>  *------------+---------+--------------------------------------+*
> > > >
> > > >       * s1          | text     |  string
> > > >        |*
> > > >
> > > >       * f1           | float4  |  float
> > > >        |*
> > > >
> > > >       * a1          | text     |  array<string>
> > |*
> > > >
> > > >       * m1         | text     |  map<strung, float>            
   |*
> > > >
> > > >       * sr1         | text     |  struct<a:string,b:string,c:int>
|*
> > > >       2. \d shows three columns (Column, Type, Source Type), where
> > > >    Type=HAWQ type, Source Type=Hive type.
> > > >       -
> > > >
> > > >       *# \d+ hcatalog.default.reg_collections;*
> > > >
> > > >       *PXF Hive Table "default.reg_collections"*
> > > >
> > > >       * Column |  Type  | Source Type                          |*
> > > >
> > > >
>  *------------+---------+--------------------------------------+*
> > > >
> > > >       * s1          | text     |  string
> > > >         |*
> > > >
> > > >       * f1           | float4  |  float
> > > >         |*
> > > >
> > > >       * a1          | text     |  array<string>
> > >  |*
> > > >
> > > >       * m1         | text     |  map<strung, float>            
   |*
> > > >
> > > >       * sr1         | text     |  struct<a:string,b:string,c:int>
|*
> > > >       3. \d shows Column, Type, where Type=Hive type.
> > > >
> > > >
> > > >    -
> > > >
> > > >       *# \d hcatalog.default.reg_collections;*
> > > >
> > > >       *PXF Hive Table "default.reg_collections"*
> > > >
> > > >       * Column |    Type                                   |*
> > > >
> > > >       *------------+--------------------------------------+*
> > > >
> > > >       * s1          | string                                     |*
> > > >
> > > >       * f1           | float                                       |*
> > > >
> > > >       * a1          | array<string>                         |*
> > > >
> > > >       * m1         | map<strung, float>                 |*
> > > >
> > > >       * sr1         |  struct<a:string,b:string,c:int> |*
> > > >
> > > > I would prefer option 1.
> > > > Any thoughts/opinions?
> > > >
> > > >
> > > > Regards, Alex.
> > > >
> > >
> >
> >
> >
> > --
> > אין דנים אדם בשעת קצת צערו
> > http://www.nrg.co.il/online/1/ART1/765/515.html
> >
>
>
>
> --
> shivram mani
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message