hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Noguchi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1491) After successful distcp, couple of checksum error files
Date Fri, 15 Jun 2007 18:21:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12505336
] 

Koji Noguchi commented on HADOOP-1491:
--------------------------------------

To confirm Dhruba and Raghu's analysis, 
I inserted one debug print statement inside DFSClient.newBackupFile to print out the "result"
and "src".

On one node, two mappers started (almost) at the same time by the distcp.
There were difinitely clashing on the temporary file names.  
Attaching the two userlogs.


Picked files from the clashing and dfs -get from source and target cluster. ls -l showed 

-rw-r--r--  1 knoguchi users 133142 Jun 15 10:46 part-270-source
-rw-r--r--  1 knoguchi users 133848 Jun 15 10:47 part-270-target
-rw-r--r--  1 knoguchi users 133848 Jun 15 10:48 part-277-source
-rw-r--r--  1 knoguchi users 133848 Jun 15 10:47 part-277-target

After the copy, part-270 file was corrupted.




> After successful distcp, couple of checksum error files
> -------------------------------------------------------
>
>                 Key: HADOOP-1491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1491
>             Project: Hadoop
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.12.3
>            Reporter: Koji Noguchi
>
> Tried copying 700,000 files  with distcp. 8 mappers per node.  Single dfs.client.buffer.dir.
> Distcp ran on 25 nodes mapreduce.
> Couple of tasks failed, but job was successful. 
> When checked, 12  files were corrupted. (Checksum error)
> This is repeatable.
> I'll add more information as we find.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message