hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11317) ACID: Improve transaction Abort logic due to timeout
Date Sat, 01 Aug 2015 00:21:04 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14650059#comment-14650059
] 

Hive QA commented on HIVE-11317:
--------------------------------



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12748236/HIVE-11317.patch

{color:red}ERROR:{color} -1 due to 146 failed/errored test(s), 9280 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_case_with_row_sequence
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_serde_regex
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_drop_partition
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_drop_table
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_move_tbl
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_add_partition_with_whitelist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_addpart1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_partition_change_col_dup_col
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_partition_change_col_nonexist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_partition_with_whitelist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_rename_partition_failure
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_rename_partition_failure2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_table_wrong_regex
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_altern1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_archive1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_archive2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_archive_corrupt
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_archive_multi1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_archive_multi2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_archive_multi3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_archive_multi4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_archive_multi5
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_archive_multi6
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_cannot_create_default_role
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_caseinsensitivity
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_create_role_no_admin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_drop_admin_role
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_drop_role_no_admin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_fail_1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_fail_8
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_grant_group
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_grant_table_allpriv
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_grant_table_dup
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_grant_table_fail1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_grant_table_fail_nogrant
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_invalid_priv_v2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_priv_current_role_neg
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_public_create
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_public_drop
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_revoke_table_fail1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_revoke_table_fail2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_role_case
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_role_cycles1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_role_cycles2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_role_grant
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_role_grant2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_role_grant_nosuchrole
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_role_grant_otherrole
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_role_grant_otheruser
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_set_role_neg1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_set_role_neg2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_show_grant_otherrole
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_show_grant_otheruser_all
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_show_grant_otheruser_alltabs
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_show_grant_otheruser_wtab
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_show_role_principals_no_admin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_show_roles_no_admin
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_table_grant_nosuchrole
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorize_grant_public
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorize_revoke_public
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_bad_exec_hooks
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_cachingprintstream
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_column_rename1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_column_rename2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_column_rename4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_compute_stats_long
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_create_function_nonexistent_class
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_create_function_nonudf_class
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_create_table_wrong_regex
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_create_unknown_genericudf
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_create_unknown_udf_udaf
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_create_view_failure1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_create_view_failure2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_create_view_failure4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_database_create_already_exists
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_database_create_invalid_name
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_database_drop_not_empty
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_database_drop_not_empty_restrict
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dbtxnmgr_nodblock
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dbtxnmgr_nodbunlock
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dbtxnmgr_notablelock
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dbtxnmgr_notableunlock
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_deletejar
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_describe_xpath1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_describe_xpath2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_describe_xpath3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_describe_xpath4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_disallow_incompatible_type_change_on1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_disallow_incompatible_type_change_on2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_drop_table_failure2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_drop_view_failure1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part_max
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dyn_part_max_per_node
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exchange_partition
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_external1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_external2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_index_compact_entry_limit
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_index_compact_size_limit
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_insert_into4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_limit_partition_stats
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_load_wrong_fileformat
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_load_wrong_fileformat_rc_seq
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_load_wrong_fileformat_txt_seq
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lockneg1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lockneg2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lockneg3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lockneg4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lockneg5
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lockneg_query_tbl_in_locked_db
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lockneg_try_db_lock_conflict
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lockneg_try_drop_locked_db
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_lockneg_try_lock_db_in_use
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nested_complex_neg
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_error
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_serde_regex
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_serde_regex2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_serde_regex3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_aggregator_error_1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_aggregator_error_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_publisher_error_1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_publisher_error_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_temp_table_index
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_temp_table_rename
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_touch1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_assert_true2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_function_does_not_implement_udf
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_nonexistent_resource
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_reflect_neg
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_test_error
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_test_error_reduce
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_cluster_tasklog_retrieval
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_local_mapred_error_cache
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_turnoff
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_table_nonprintable_negative
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_udf_local_resource
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4780/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4780/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4780/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 146 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12748236 - PreCommit-HIVE-TRUNK-Build

> ACID: Improve transaction Abort logic due to timeout
> ----------------------------------------------------
>
>                 Key: HIVE-11317
>                 URL: https://issues.apache.org/jira/browse/HIVE-11317
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore, Transactions
>    Affects Versions: 1.0.0
>            Reporter: Eugene Koifman
>            Assignee: Eugene Koifman
>              Labels: triage
>         Attachments: HIVE-11317.patch
>
>
> the logic to Abort transactions that have stopped heartbeating is in
> TxnHandler.timeOutTxns()
> This is only called when DbTxnManger.getValidTxns() is called.
> So if there is a lot of txns that need to be timed out and the there are not SQL clients
talking to the system, there is nothing to abort dead transactions, and thus compaction can't
clean them up so garbage accumulates in the system.
> Also, streaming api doesn't call DbTxnManager at all.
> Need to move this logic into Initiator (or some other metastore side thread).
> Also, make sure it is broken up into multiple small(er) transactions against metastore
DB.
> Also more timeOutLocks() locks there as well.
> see about adding TXNS.COMMENT field which can be used for "Auto aborted due to timeout"
for example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message