hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13588) ExportSnapshot should checksum data transferred
Date Sun, 03 May 2015 04:13:06 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525638#comment-14525638
] 

Ted Yu commented on HBASE-13588:
--------------------------------

TestExportSnapshot#testExportRetry fails with patch locally.
In test output, I saw:
{code}
2015-05-02 21:06:37,106 WARN  [ContainersLauncher #0] nodemanager.DefaultContainerExecutor(207):
Exit code from container container_1430625969815_0001_01_000003 is : 143
Error: java.io.IOException: checksums don't match for input: hdfs://localhost:60254/user/tyu/test-data/98603dee-5efc-43d1-a51e-cd70354870a9/data/default/testtb-1430625973413/
           5373881af77da1f7f8463f160a317c15/cf/381595c4c327402fab3f3780b59f121b and output:
file:/Users/tyu/1-hbase/hbase-server/target/test-data/7a27b9b5-48e4-4b27-b29d-16ecd441778e/local-export-
1430625977202/archive/data/default/testtb-1430625973413/5373881af77da1f7f8463f160a317c15/cf/381595c4c327402fab3f3780b59f121b
  at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:299)
  at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:193)
  at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:119)
  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
{code}

> ExportSnapshot should checksum data transferred
> -----------------------------------------------
>
>                 Key: HBASE-13588
>                 URL: https://issues.apache.org/jira/browse/HBASE-13588
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 2.0.0, 1.0.1, 0.98.13
>            Reporter: Mubeen Afzal
>            Assignee: Mubeen Afzal
>         Attachments: HBASE-13588.patch
>
>
> We sent quite a bit of data using ExportSnapshot and noticed some corruption.  We use
checksums to verify if we have transferred a particular file but never do any verification
after the file has been transferred.  I propose that we do a checksum upon completing file
transfer and if the checksums do not match up, I propose we fail by throwing an IOException.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message