Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 28 Mar 2013 20:21:18 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12639494.1364429873688.71653.1364502078490@arcas>
In-Reply-To: <JIRA.12639494.1364429873688@arcas>
References: <JIRA.12639494.1364429873688@arcas>
Subject: [jira] [Comment Edited] (HBASE-8207) Replication could have data
 loss when machine name contains hyphen "-"
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13616617#comment-13616617 ] 

Ted Yu edited comment on HBASE-8207 at 3/28/13 8:20 PM:
--------------------------------------------------------

@Jeff:
Patch is great: we avoid incompatibilities for the next 0.94 release.
Actually we can use your patch for 0.94 and switch to new delimiter in 0.95 / 0.98 so that code maintenance work is less.
{code}
+  // Since servername can contains "-" like "ip-10-46-221-101.ec2.internal", so we need skip some
{code}
'can contains "-" like' => 'can contain "-", such as'
'we need skip' -> 'we need to skip'
{code}
+  // 2-ip-10-46-221-101.ec2.internal,52170,1364333181125-ip-10-46-221-101.ec2.internal,52171,
+  // 1364333181127-...
{code}
Normally we don't want long line. But for the above sample, we'd better keep it on the same line.
{code}
+    // extract dead servers
+    if (parts.length > 1) {
{code}
Since there is no else block, you can revert the condition and bail out early.
{code}
+      // valid server name delimiter "-" has to be after ","
{code}
'after ","' -> 'after "," in ServerName'
{code}
+                LOG.error("Found invalid server name:" + serverName);
{code}
The above log sentence is the same for first and second server name candidates. It would be nice to use slightly different terms.
{code}
+      if(startIndex < len - 1){
{code}
I think we should add a log statement for the else block of the above check - basically there is no second server name in that case.

One general minor comment is that you should insert space between if and the following left paren.
                
      was (Author: yuzhihong@gmail.com):
    @Jeff:
Patch is great: we avoid incompatibilities for the next 0.94 release.
Actually we can use your patch for 0.94 and switch to new delimiter in 0.95 / 0.98 so that code maintenance work is less.
{code}
+  // Since servername can contains "-" like "ip-10-46-221-101.ec2.internal", so we need skip some
{code}
'can contains "-" like' -> 'can contain "-", such as'
'we need skip' -> 'we need to skip'
{code}
+  // 2-ip-10-46-221-101.ec2.internal,52170,1364333181125-ip-10-46-221-101.ec2.internal,52171,
+  // 1364333181127-...
{code}
Normally we don't want long line. But for the above sample, we'd better keep it on the same line.
{code}
+    // extract dead servers
+    if (parts.length > 1) {
{code}
Since there is no else block, you can revert the condition and bail out early.
{code}
+      // valid server name delimiter "-" has to be after ","
{code}
'after ","' -> 'after "," in ServerName'
{code}
+                LOG.error("Found invalid server name:" + serverName);
{code}
The above log sentence is the same for first and second server name candidates. It would be nice to use slightly different terms.
{code}
+      if(startIndex < len - 1){
{code}
I think we should add a log statement for the else block of the above check - basically there is no second server name in that case.

One general minor comment is that you should insert space between if and the following left paren.
                  
> Replication could have data loss when machine name contains hyphen "-"
> ----------------------------------------------------------------------
>
>                 Key: HBASE-8207
>                 URL: https://issues.apache.org/jira/browse/HBASE-8207
>             Project: HBase
>          Issue Type: Bug
>          Components: Replication
>    Affects Versions: 0.95.0, 0.94.6
>            Reporter: Jeffrey Zhong
>            Assignee: Jeffrey Zhong
>            Priority: Critical
>             Fix For: 0.95.0, 0.98.0, 0.94.7
>
>         Attachments: failed.txt, hbase-8207.patch, HBASE-8212-94.patch
>
>
> In the recent test case TestReplication* failures, I'm finally able to find the cause(or one of causes) for its intermittent failures.
> When a machine name contains "-", it breaks the function ReplicationSource.checkIfQueueRecovered. It causes the following issue:
> deadRegionServers list is way off so that replication doesn't wait for log splitting finish for a wal file and move on to the next one(data loss)
> You can see that replication use those weird paths constructed from deadRegionServers to check a file existence
> {code}
> 2013-03-26 21:26:51,385 INFO  [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserver.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2-user/hbase/.logs/1.compute.internal,52170,1364333181125/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,386 INFO  [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserver.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2-user/hbase/.logs/1.compute.internal,52170,1364333181125-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,387 INFO  [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserver.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2-user/hbase/.logs/west/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,389 INFO  [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserver.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2-user/hbase/.logs/west-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,391 INFO  [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserver.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2-user/hbase/.logs/156.us/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,394 INFO  [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserver.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2-user/hbase/.logs/156.us-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,396 INFO  [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserver.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2-user/hbase/.logs/0/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,398 INFO  [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125] regionserver.ReplicationSource(524): Possible location hdfs://localhost:52882/user/ec2-user/hbase/.logs/0-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> {code}
> This happened in the recent test failure in http://54.241.6.143/job/HBase-0.94/org.apache.hbase$hbase/21/testReport/junit/org.apache.hadoop.hbase.replication/TestReplicationQueueFailover/queueFailover/?auto_refresh=false
> Search for 
> {code}
> File does not exist: hdfs://localhost:52882/user/ec2-user/hbase/.oldlogs/ip-10-197-0-156.us-west-1.compute.internal%2C52170%2C1364333181125.1364333199540
> {code}
> After 10 times retries, replication source gave up and move on to the next file. Data loss happens. 
> Since lots of EC2 machine names contain "-" including our Jenkin servers, this is a high impact issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira