hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7286) Parameterize HCatMapReduceTest for testing against all Hive storage formats
Date Thu, 10 Jul 2014 20:29:05 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14057911#comment-14057911
] 

David Chen commented on HIVE-7286:
----------------------------------

Currently, the patch will only test against the entires in the table that specify a TestStorageFormat
class since there is currently no way to infer the mapping of SerDe to InputFormat/OutputFormat.
Once HIVE-5976 is in, then the TestStorageFormat classes will be removed and the StorageFormatDescriptor
classes will be used. However, this will require additional code changes to properly handle
storage formats with configurable SerDes, such as RCFile. The test code for Avro is in this
patch, but the tests (along with the tests for Parquet) will currently fail due to HIVE-4329.

To clarify, once HIVE-5976 is in and the corresponding changes to this fixture are made, then
SerDe devs would only need to add an entry to the table if they _do not_ want their SerDe
tested against HCatalog.

> Parameterize HCatMapReduceTest for testing against all Hive storage formats
> ---------------------------------------------------------------------------
>
>                 Key: HIVE-7286
>                 URL: https://issues.apache.org/jira/browse/HIVE-7286
>             Project: Hive
>          Issue Type: Test
>          Components: HCatalog
>            Reporter: David Chen
>            Assignee: David Chen
>         Attachments: HIVE-7286.1.patch
>
>
> Currently, HCatMapReduceTest, which is extended by the following test suites:
>  * TestHCatDynamicPartitioned
>  * TestHCatNonPartitioned
>  * TestHCatPartitioned
>  * TestHCatExternalDynamicPartitioned
>  * TestHCatExternalNonPartitioned
>  * TestHCatExternalPartitioned
>  * TestHCatMutableDynamicPartitioned
>  * TestHCatMutableNonPartitioned
>  * TestHCatMutablePartitioned
> These tests run against RCFile. Currently, only TestHCatDynamicPartitioned is run against
any other storage format (ORC).
> Ideally, HCatalog should be tested against all storage formats supported by Hive. The
easiest way to accomplish this is to turn HCatMapReduceTest into a parameterized test fixture
that enumerates all Hive storage formats. Until HIVE-5976 is implemented, we would need to
manually create the mapping of SerDe to InputFormat and OutputFormat. This way, we can explicitly
keep track of which storage formats currently work with HCatalog or which ones are untested
or have test failures. The test fixture should also use Reflection to find all classes in
the classpath that implements the SerDe interface and raise a failure if any of them are not
enumerated.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message