hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sushanth Sowmyan (JIRA)" <>
Subject [jira] [Updated] (HIVE-4816) Followup from HIVE-4551 : Reworking HCat Record Transformation
Date Wed, 03 Jul 2013 23:10:20 GMT


Sushanth Sowmyan updated HIVE-4816:

    Attachment: 4816.patch

Attaching the original exploratory patch I had up on HIVE-4551 that introduces notion of a
DataTransformer and a TransformedHCatRecord.

This patch still has one important bug that needs to be ironed out - I attach it mostly to
show what I intended by my previous comment.

The bug is that it still doesn't do conversion of shorts/bytes inside a nested structure,
and so even if all the tests currently pass with this patch, this patch is broken(an indication
that we need more tests ). Essentially, a lot more of PigHCatUtil.getJavaObj needs to move
into the DataTransformer, and out of PigHCatUtil.
> Followup from HIVE-4551 : Reworking HCat Record Transformation
> --------------------------------------------------------------
>                 Key: HIVE-4816
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Sushanth Sowmyan
>            Assignee: Sushanth Sowmyan
>         Attachments: 4816.patch
> This is a post-bug followup from HIVE-4551, to refactor some HCatalog code to be cleaner/neater.
Reposting contents from my previous jira, this jira needs to track the following:
> a) HCatRecord and HCatSchema reflect underlying raw data and do no promotions.
>  b) Introduce a ConversionImpl, which defines various datatype conversion functions,
which all default to returning the input, and having a config that allows a user which conversions
are implemented.
>  c) Introduce a PromotedHCatRecord & PromotedHCatSchema that wrap HCatRecord/HCatSchema
and use a ConversionImpl.
>  d) Implement a PigLoaderConversionImpl/PigStorerConversionImpl in hcat-pig-adapter,
which implements the following: Short->Int promotion, Short->Int promotion, Boolean->Int
>  e) Have HCatLoader/HCatStorer use the promoted versions of HCatRecord/HCatSchema which
use the PigConversionImpl.
>  f) Remove the current HCatContext promotion parameters and make them be HCatLoader/HCatStorer

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message