hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-4322) SkewedInfo in Metastore Thrift API cannot be deserialized in Python
Date Sun, 14 Apr 2013 01:08:15 GMT

    [ https://issues.apache.org/jira/browse/HIVE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631241#comment-13631241
] 

Hudson commented on HIVE-4322:
------------------------------

Integrated in Hive-trunk-hadoop2 #157 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/157/])
    HIVE-4322: SkewedInfo in Metastore Thrift API cannot be deserialized in Python (Samuel
Yuan via Gang Tim Liu) (Revision 1467491)

     Result = FAILURE
gangtimliu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467491
Files : 
* /hive/trunk/metastore/if/hive_metastore.thrift
* /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
* /hive/trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h
* /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
* /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java
* /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java
* /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java
* /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java
* /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java
* /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java
* /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
* /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedValueList.java
* /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java
* /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java
* /hive/trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
* /hive/trunk/metastore/src/gen/thrift/gen-php/metastore/Types.php
* /hive/trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py
* /hive/trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb
* /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
* /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/listbucketingpruner/ListBucketingPruner.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java

                
> SkewedInfo in Metastore Thrift API cannot be deserialized in Python
> -------------------------------------------------------------------
>
>                 Key: HIVE-4322
>                 URL: https://issues.apache.org/jira/browse/HIVE-4322
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore, Thrift API
>    Affects Versions: 0.11.0
>            Reporter: Samuel Yuan
>            Assignee: Samuel Yuan
>            Priority: Minor
>             Fix For: 0.11.0
>
>         Attachments: HIVE-4322.HIVE-4322.HIVE-4322.HIVE-4322.D10203.1.patch
>
>
> The Thrift-generated Python code that deserializes Thrift objects fails whenever a complex
type is used as a map key, because by default mutable Python objects such as lists do not
have a hash function. See https://issues.apache.org/jira/browse/THRIFT-162 for related discussion.
> The SkewedInfo struct contains a map which uses a list as a key, breaking the Python
Thrift interface. It is not possible to specify the mapping from Thrift types to Python types,
or otherwise we could map Thrift lists to Python tuples. Instead, the proposed workaround
wraps the list inside a new struct. This alone does not accomplish anything, but allows Python
clients to define a hash function for the struct class, e.g.:
> def f(object):
>     return hash(tuple(object.skewedValueList))
> SkewedValueList.__hash__ = f
> In practice a more efficient hash might be defined that does not involve copying the
list. The advantage of wrapping the list inside a struct is that the client does not have
to define the hash on the list itself, which would change the behaviour of lists everywhere
else in the code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message