cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Shuler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11197) upgrade bootstrap tests flap when migration tasks fail
Date Mon, 22 Feb 2016 15:32:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157139#comment-15157139
] 

Michael Shuler commented on CASSANDRA-11197:
--------------------------------------------

Test logs are available as job artifacts. (hint: from the artifact link, it may be easier
to use the link at the bottom to grab a zip file of *all* the artifacts for local searching)

Starting here, we look for where the test we are interested in was run by looking for the
test_bucket files:
* http://cassci.datastax.com/job/snazy-11064-3.0-dtest/1/artifact/

I found {{incremental_repair_test.py:TestIncRepair.multiple_repair_test}} in {{cassandra-dtest/test_bucket.011}},
so this particular test ran on node 11.
* http://cassci.datastax.com/job/snazy-11064-3.0-dtest/1/artifact/cassandra-dtest/
* http://cassci.datastax.com/job/snazy-11064-3.0-dtest/1/artifact/cassandra-dtest/test_bucket.011/*view*/

Now that we know where it ran, we can grab the node 11 ccm node log tarball for investigation
- the ccm node logs are are in timestamped/test-named directories:
* http://cassci.datastax.com/job/snazy-11064-3.0-dtest/1/artifact/jenkins-snazy-11064-3.0-dtest-1_logs.011.tar.gz

{noformat}
$ tar xzvf jenkins-snazy-11064-3.0-dtest-1_logs.011.tar.gz 
logs/                                                                                    
                                              
logs/last                                                                                
                                              
logs/1455969761739_mixed_version_test.TestSchemaChanges.test_friendly_unrecognized_table_handling/
                                     
logs/1455969761739_mixed_version_test.TestSchemaChanges.test_friendly_unrecognized_table_handling/node2_debug.log
                      
logs/1455969761739_mixed_version_test.TestSchemaChanges.test_friendly_unrecognized_table_handling/node1.log
                            
logs/1455969761739_mixed_version_test.TestSchemaChanges.test_friendly_unrecognized_table_handling/node2.log
                            
logs/1455969761739_mixed_version_test.TestSchemaChanges.test_friendly_unrecognized_table_handling/node1_debug.log
logs/snapshot_test_pre-restore
logs/1455969234161_snapshot_test.TestArchiveCommitlog.test_archive_commitlog_point_in_time_with_active_commitlog_ln/
logs/1455969234161_snapshot_test.TestArchiveCommitlog.test_archive_commitlog_point_in_time_with_active_commitlog_ln/node1.log
logs/1455969234161_snapshot_test.TestArchiveCommitlog.test_archive_commitlog_point_in_time_with_active_commitlog_ln/node1_debug.log
logs/1455969534820_repair_tests.incremental_repair_test.TestIncRepair.multiple_repair_test/
logs/1455969534820_repair_tests.incremental_repair_test.TestIncRepair.multiple_repair_test/node5.log
logs/1455969534820_repair_tests.incremental_repair_test.TestIncRepair.multiple_repair_test/node2_debug.log
logs/1455969534820_repair_tests.incremental_repair_test.TestIncRepair.multiple_repair_test/node3_debug.log
logs/1455969534820_repair_tests.incremental_repair_test.TestIncRepair.multiple_repair_test/node5_debug.log
logs/1455969534820_repair_tests.incremental_repair_test.TestIncRepair.multiple_repair_test/node3.log
logs/1455969534820_repair_tests.incremental_repair_test.TestIncRepair.multiple_repair_test/node1.log
logs/1455969534820_repair_tests.incremental_repair_test.TestIncRepair.multiple_repair_test/node2.log
logs/1455969534820_repair_tests.incremental_repair_test.TestIncRepair.multiple_repair_test/node1_debug.log
{noformat}

> upgrade bootstrap tests flap when migration tasks fail
> ------------------------------------------------------
>
>                 Key: CASSANDRA-11197
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11197
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Jim Witschey
>            Assignee: DS Test Eng
>              Labels: dtest
>
> I've seen these tests flap:
> {code}
> upgrade_tests/upgrade_through_versions_test.py:ProtoV4Upgrade_3_1_UpTo_3_2_HEAD.bootstrap_test
> upgrade_tests/upgrade_through_versions_test.py:ProtoV4Upgrade_3_3_UpTo_Trunk_HEAD.bootstrap_test
> upgrade_tests/upgrade_through_versions_test.py:ProtoV3Upgrade_3_0_UpTo_3_1_HEAD.bootstrap_multidc_test
> upgrade_tests/upgrade_through_versions_test.py:ProtoV3Upgrade_3_2_UpTo_3_3_HEAD.bootstrap_multidc_test
> upgrade_tests/upgrade_through_versions_test.py:ProtoV3Upgrade_3_3_UpTo_Trunk_HEAD.bootstrap_multidc_test
> upgrade_tests/upgrade_through_versions_test.py:ProtoV4Upgrade_3_3_UpTo_Trunk_HEAD.bootstrap_multidc_test
> {code}
> There may be more upgrade paths that flap, I'm not sure. All the failures I've seen look
like this:
> {code}
> Unexpected error in node5 node log: ['ERROR [main] 2016-02-18 20:05:13,012 MigrationManager.java:164
- Migration task failed to complete\nERROR [main] 2016-02-18 20:05:14,012 MigrationManager.java:164
- Migration task failed to complete']
> {code}
> [~rhatch] Do these look familiar at all?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message