hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Kunz (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-732) HDFS files are ending up truncated
Date Sun, 25 Oct 2009 16:53:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769833#action_12769833
] 

Christian Kunz commented on HDFS-732:
-------------------------------------

Concerning the file in the first comment I found logs of 2 datanodes showing that indeed the
block size shrunk from 18153472 to 17825792. 18153472 is not the correct size, but it is larger
than 17825792, and I would argue, that a block should never be recovered by a block of smaller
size.

Logs from datanode receiving original block:

2009-10-23 19:46:47,934 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_6703874482275767879_76799972 src: /xxx.yyy.zzz.43:34608 dest: /xxx.yyy.zzz.43:uuu10
2009-10-23 21:15:59,694 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(xxx.yyy.zzz.43:uuu10,
storageID=DS-243564233-xxx.yyy.zzz.43-uuu10-1254870555871, infoPort=50075, ipcPort=8020):Exception
writing block blk_6703874482275767879_76799972 to mirror xxx.yyy.zzz.56:uuu10
2009-10-23 21:15:59,694 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in
receiveBlock for block blk_6703874482275767879_76799972 java.io.IOException: Connection reset
by peer
2009-10-23 21:15:59,711 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
blk_6703874482275767879_76799972 2 Exception java.io.EOFException
2009-10-23 21:15:59,713 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
blk_6703874482275767879_76799972 2 : Thread is interrupted.
2009-10-23 21:15:59,713 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
2 for block blk_6703874482275767879_76799972 terminating
2009-10-23 21:15:59,713 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_6703874482275767879_76799972
received exception java.io.IOException: Connection reset by peer

Logs from datanode next in the pipeline:

2009-10-23 19:46:48,174 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_6703874482275767879_76799972 src: /xxx.yyy.zzz.43:34609 dest: /xxx.yyy.zzz.56:uuu10
2009-10-23 21:15:59,661 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(xxx.yyy.zzz.56:uuu10,
storageID=DS-807595239-72.30.217.12-50010-1203107050520, infoPort=50075, ipcPort=8020):Exception
writing block blk_6703874482275767879_76799972 to mirror xxx.yyy.zzz.44:uuu10
2009-10-23 21:15:59,661 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in
receiveBlock for block blk_6703874482275767879_76799972 java.io.IOException: Connection reset
by peer
2009-10-23 21:15:59,680 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
blk_6703874482275767879_76799972 1 Exception java.io.EOFException
2009-10-23 21:15:59,681 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
blk_6703874482275767879_76799972 1 : Thread is interrupted.
2009-10-23 21:15:59,681 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
1 for block blk_6703874482275767879_76799972 terminating
2009-10-23 21:15:59,681 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_6703874482275767879_76799972
received exception java.io.IOException: Connection reset by peer
2009-10-23 21:16:00,069 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: oldblock=blk_6703874482275767879_76799972(length=18153472),
newblock=blk_6703874482275767879_76840998(length=17825792), datanode=xxx.yyy.zzz.56:uuu10
2009-10-23 21:16:00,134 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_6703874482275767879_76840998 src: /xxx.yyy.zzz.43:36067 dest: /xxx.yyy.zzz.56:uuu10
2009-10-23 21:16:00,134 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Reopen already-open
Block for append blk_6703874482275767879_76840998
2009-10-23 21:16:00,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Changing block
file offset of block blk_6703874482275767879_76840998 from 0 to 17825792 meta file offset
to 17415
2009-10-23 21:16:00,171 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(xxx.yyy.zzz.56:uuu10,
storageID=DS-807595239-72.30.217.12-50010-1203107050520, infoPort=50075, ipcPort=8020):Exception
writing block blk_6703874482275767879_76840998 to mirror xxx.yyy.zzz.44:uuu10
2009-10-23 21:16:00,171 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in
receiveBlock for block blk_6703874482275767879_76840998 java.io.IOException: Connection reset
by peer
2009-10-23 21:16:00,400 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
blk_6703874482275767879_76840998 1 Exception java.nio.channels.ClosedByInterruptException
2009-10-23 21:16:00,417 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
blk_6703874482275767879_76840998 1 : Thread is interrupted.
2009-10-23 21:16:00,417 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
1 for block blk_6703874482275767879_76840998 terminating
2009-10-23 21:16:00,417 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_6703874482275767879_76840998
received exception java.io.IOException: Connection reset by peer


> HDFS files are ending up truncated
> ----------------------------------
>
>                 Key: HDFS-732
>                 URL: https://issues.apache.org/jira/browse/HDFS-732
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.20.1
>            Reporter: Christian Kunz
>
> We recently started to use hadoop-0.20.1 in our production environment (less than 2 weeks
ago) and already had 3 instances of truncated files, more than we had for months using hadoop-0.18.3.
> Writing is done using libhdfs, although it rather seems to be a problem on the server
side.
> I will post some relevant logs (they are too large to be put into the description)
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message