hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Min Zhou (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-537) Hive TypeInfo/ObjectInspector to support union (besides struct, array, and map)
Date Thu, 11 Jun 2009 09:37:12 GMT

    [ https://issues.apache.org/jira/browse/HIVE-537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718373#action_12718373

Min Zhou commented on HIVE-537:

first approach:
  O(mN/p) + O(m(N/p log (N/p))) + O(mN/r) + O(m)
I don't agree with you about this O(m).  It would be indeed very large cost.  and meanwhile,
 you should adding the cost in the end joining all results into one. 

 for the second approach, I think it should be  
  O(N/p) + O(mN/p log (mN/p)) + O(mN/r)  

> Hive TypeInfo/ObjectInspector to support union (besides struct, array, and map)
> -------------------------------------------------------------------------------
>                 Key: HIVE-537
>                 URL: https://issues.apache.org/jira/browse/HIVE-537
>             Project: Hadoop Hive
>          Issue Type: New Feature
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
> There are already some cases inside the code that we use heterogeneous data: JoinOperator,
and UnionOperator (in the sense that different parents can pass in records with different
> We currently use Operator's parentID to distinguish that. However that approach does
not extend to more complex plans that might be needed in the future.
> We will support the union type like this:
> {code}
> TypeDefinition:
>   type: primitivetype | structtype | arraytype | maptype | uniontype
>   uniontype: "union" "<" tag ":" type ("," tag ":" type)* ">"
> Example:
>   union<0:int,1:double,2:array<string>,3:struct<a:int,b:string>>
> Example of serialized data format:
>   We will first store the tag byte before we serialize the object. On deserialization,
we will first read out the tag byte, then we know what is the current type of the following
object, so we can deserialize it successfully.
> Interface for ObjectInspector:
> interface UnionObjectInspector {
>   /** Returns the array of OIs that are for each of the tags
>    */
>   ObjectInspector[] getObjectInspectors();
>   /** Return the tag of the object.
>    */
>   byte getTag(Object o);
>   /** Return the field based on the tag value associated with the Object.
>    */
>   Object getField(Object o);
> };
> {code}

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message