Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8D54DF123 for ; Thu, 28 Mar 2013 05:17:19 +0000 (UTC) Received: (qmail 3306 invoked by uid 500); 28 Mar 2013 05:17:19 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 3097 invoked by uid 500); 28 Mar 2013 05:17:16 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 3023 invoked by uid 99); 28 Mar 2013 05:17:15 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Mar 2013 05:17:15 +0000 Date: Thu, 28 Mar 2013 05:17:15 +0000 (UTC) From: "Jeffrey Zhong (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-8207) Replication could have data loss when machine name contains hyphen "-" MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-8207?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1361= 6060#comment-13616060 ]=20 Jeffrey Zhong commented on HBASE-8207: -------------------------------------- [~tedyu@apache.org] It depends if log splitting can finish in time before r= eplication gives up. In our test case setup, log splitting normally complet= es within 1-2 secs while replication takes about 5 secs to give up. So in m= ost cases, the test runs fine. In local dev env, we don't have machine name= containing "-" so we can't even reproduce it.=20 Since we just changed Jenkins, I can't find more build history. From my "fl= aky test detector" tool(HBASE-8018) result in trunk which I ran on March 5.= We can see replication in trunk is flaky at that time: _HBase-TRUNK=C2=A0(from last 10 runs)_ Failed Test Cases =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03908 3909= 3910 3912 3913 3914 3915 3916 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailo= ver =C2=A0 =C2=A01 =C2=A0 =C2=A01 =C2=A0 =C2=A01 =C2=A0 -1 =C2=A0 =C2=A00 = =C2=A0 -1 =C2=A0 =C2=A00 =C2=A0 =C2=A01 org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.= queuefailover =C2=A0 =C2=A01 =C2=A0 =C2=A01 =C2=A0 =C2=A01 =C2=A0 -1 =C2=A0= =C2=A00 =C2=A0 -1 =C2=A0 =C2=A00 =C2=A0 =C2=A01 _HBase-0.95 (from last 10 runs configurable)_ Failed Test Cases =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A021= =C2=A0 22 =C2=A0 23 =C2=A0 24 =C2=A0 25 =C2=A0 27 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailo= ver =C2=A0 =C2=A01 =C2=A0 -1 =C2=A0 =C2=A00 =C2=A0 =C2=A01 =C2=A0 -1 =C2=A0= =C2=A00 org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.= queuefailover =C2=A0 =C2=A00 =C2=A0 =C2=A01 =C2=A0 -1 =C2=A0 =C2=A00 =C2=A0= -1 =C2=A0 =C2=A00 =20 > Replication could have data loss when machine name contains hyphen "-" > ---------------------------------------------------------------------- > > Key: HBASE-8207 > URL: https://issues.apache.org/jira/browse/HBASE-8207 > Project: HBase > Issue Type: Bug > Components: Replication > Affects Versions: 0.95.0, 0.94.6 > Reporter: Jeffrey Zhong > Assignee: Jeffrey Zhong > Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: failed.txt > > > In the recent test case TestReplication* failures, I'm finally able to fi= nd the cause(or one of causes) for its intermittent failures. > When a machine name contains "-", it breaks the function ReplicationSourc= e.checkIfQueueRecovered. It causes the following issue: > deadRegionServers list is way off so that replication doesn't wait for lo= g splitting finish for a wal file and move on to the next one(data loss) > You can see that replication use those weird paths constructed from deadR= egionServers to check a file existence > {code} > 2013-03-26 21:26:51,385 INFO [ReplicationExecutor-0.replicationSource,2-= ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserve= r.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2= -user/hbase/.logs/1.compute.internal,52170,1364333181125/ip-10-197-0-156.us= -west-1.compute.internal%252C52170%252C1364333181125.1364333199540 > 2013-03-26 21:26:51,386 INFO [ReplicationExecutor-0.replicationSource,2-= ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserve= r.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2= -user/hbase/.logs/1.compute.internal,52170,1364333181125-splitting/ip-10-19= 7-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.136433319954= 0 > 2013-03-26 21:26:51,387 INFO [ReplicationExecutor-0.replicationSource,2-= ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserve= r.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2= -user/hbase/.logs/west/ip-10-197-0-156.us-west-1.compute.internal%252C52170= %252C1364333181125.1364333199540 > 2013-03-26 21:26:51,389 INFO [ReplicationExecutor-0.replicationSource,2-= ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserve= r.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2= -user/hbase/.logs/west-splitting/ip-10-197-0-156.us-west-1.compute.internal= %252C52170%252C1364333181125.1364333199540 > 2013-03-26 21:26:51,391 INFO [ReplicationExecutor-0.replicationSource,2-= ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserve= r.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2= -user/hbase/.logs/156.us/ip-10-197-0-156.us-west-1.compute.internal%252C521= 70%252C1364333181125.1364333199540 > 2013-03-26 21:26:51,394 INFO [ReplicationExecutor-0.replicationSource,2-= ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserve= r.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2= -user/hbase/.logs/156.us-splitting/ip-10-197-0-156.us-west-1.compute.intern= al%252C52170%252C1364333181125.1364333199540 > 2013-03-26 21:26:51,396 INFO [ReplicationExecutor-0.replicationSource,2-= ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserve= r.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2= -user/hbase/.logs/0/ip-10-197-0-156.us-west-1.compute.internal%252C52170%25= 2C1364333181125.1364333199540 > 2013-03-26 21:26:51,398 INFO [ReplicationExecutor-0.replicationSource,2-= ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserve= r.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2= -user/hbase/.logs/0-splitting/ip-10-197-0-156.us-west-1.compute.internal%25= 2C52170%252C1364333181125.1364333199540 > {code} > This happened in the recent test failure in http://54.241.6.143/job/HBase= -0.94/org.apache.hbase$hbase/21/testReport/junit/org.apache.hadoop.hbase.re= plication/TestReplicationQueueFailover/queueFailover/?auto_refresh=3Dfalse > Search for=20 > {code} > File does not exist: hdfs://localhost:52882/user/ec2-user/hbase/.oldlogs/= ip-10-197-0-156.us-west-1.compute.internal%2C52170%2C1364333181125.13643331= 99540 > {code} > After 10 times retries, replication source gave up and move on to the nex= t file. Data loss happens.=20 > Since lots of EC2 machine names contain "-" including our Jenkin servers,= this is a high impact issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrato= rs For more information on JIRA, see: http://www.atlassian.com/software/jira