hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC
Date Fri, 08 Apr 2016 21:20:25 GMT

    [ https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15232951#comment-15232951
] 

Hive QA commented on HIVE-9660:
-------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12797641/HIVE-9660.07.patch

{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 172 failed/errored test(s), 9771 tests executed
*Failed tests:*
{noformat}
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join30.q-vector_data_types.q-scriptfile1.q-and-12-more - did not produce
a TEST-*.xml file
TestSparkCliDriver-auto_join9.q-bucketmapjoin11.q-smb_mapjoin_2.q-and-12-more - did not produce
a TEST-*.xml file
TestSparkCliDriver-date_udf.q-join23.q-auto_join4.q-and-12-more - did not produce a TEST-*.xml
file
TestSparkCliDriver-groupby4.q-timestamp_null.q-auto_join23.q-and-12-more - did not produce
a TEST-*.xml file
TestSparkCliDriver-ppd_gby_join.q-groupby_rollup1.q-auto_sortmerge_join_4.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join3.q-union26.q-load_dyn_part15.q-and-12-more - did not produce a
TEST-*.xml file
TestSparkCliDriver-stats13.q-groupby6_map.q-join_casesensitive.q-and-12-more - did not produce
a TEST-*.xml file
TestSparkCliDriver-vector_distinct_2.q-input17.q-load_dyn_part2.q-and-12-more - did not produce
a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_orig_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_file_dump
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_int_type_promotion
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_lengths
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_llap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_min_max
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_fast_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_all_types
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_orig_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_aggregate_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_binary_join_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_char_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_data_types
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_distinct_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_groupby_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_interval_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_number_compare_projection
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_orderby_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_reduce1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_reduce2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_reduce3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_string_concat
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_varchar_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_part_varchar
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_orig_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_values_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge8
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union_fast_stats
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_types
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_orig_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_aggregate_9
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_binary_join_groupby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_char_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_data_types
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_distinct_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_interval_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_orderby_5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_reduce1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_reduce2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_reduce3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_string_concat
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_varchar_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_part_varchar
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorizationWithAcid
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.columnProjection[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.columnProjection[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.metaData[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.metaData[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.test1[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.test1[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testBitPack64Large[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testBitPack64Large[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testDate1900[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testDate2038[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testHiveDecimalAllNulls[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testHiveDecimalAllNulls[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testHiveDecimalIsNullReset[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testPredicatePushdown[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testPredicatePushdown[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testSeek[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testSeek[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testSnappy[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testSnappy[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testStringAndBinaryStatistics[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testStringAndBinaryStatistics[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testStripeLevelStats[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testStripeLevelStats[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testTimestamp[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testTimestamp[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testUnionAndTimestamp[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testUnionAndTimestamp[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testWithoutIndex[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testWithoutIndex[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testZeroCopySeek[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testZeroCopySeek[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcNullOptimization.testColumnsWithNullAndCompression
org.apache.hadoop.hive.ql.io.orc.TestOrcNullOptimization.testMultiStripeWithNull
org.apache.hadoop.hive.ql.io.orc.TestOrcNullOptimization.testMultiStripeWithoutNull
org.apache.hadoop.hive.ql.io.orc.TestOrcRawRecordMerger.testRecordReaderOldBaseAndDelta
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testOrcSerDeStatsComplex
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testOrcSerDeStatsComplexOldFormat
org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.testExternalFooterCache
org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.testExternalFooterCachePpd
org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.testSplitEliminationComplexExpr
org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.testSplitEliminationLargeMaxSplit
org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.testSplitEliminationSmallMaxSplit
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.metaData
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.test1
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.testNonDictionaryRepeatingString
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.testRepeating
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.testSeek
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.testSnappy
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.testStringAndBinaryStatistics
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.testStringPadding
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.testStripeLevelStats
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.testStructs
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.testTimestamp
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.testUnionAndTimestamp
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.testUnions
org.apache.hadoop.hive.ql.io.orc.TestVectorOrcFile.testWithoutIndex
org.apache.hadoop.hive.ql.io.orc.TestVectorizedORCReader.createFile
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccessWithReadOnly
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase
org.apache.hive.hcatalog.pig.TestE2EScenarios.testReadOrcAndRCFromPig
org.apache.hive.hcatalog.pig.TestHCatStorer.testWriteTinyint[3]
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7513/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7513/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7513/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 172 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12797641 - PreCommit-HIVE-TRUNK-Build

> store end offset of compressed data for RG in RowIndex in ORC
> -------------------------------------------------------------
>
>                 Key: HIVE-9660
>                 URL: https://issues.apache.org/jira/browse/HIVE-9660
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: HIVE-9660.01.patch, HIVE-9660.02.patch, HIVE-9660.03.patch, HIVE-9660.04.patch,
HIVE-9660.05.patch, HIVE-9660.06.patch, HIVE-9660.07.patch, HIVE-9660.07.patch, HIVE-9660.patch,
HIVE-9660.patch
>
>
> Right now the end offset is estimated, which in some cases results in tons of extra data
being read.
> We can add a separate array to RowIndex (positions_v2?) that stores number of compressed
buffers for each RG, or end offset, or something, to remove this estimation magic



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message