hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Kunz (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-732) HDFS files are ending up truncated
Date Sun, 25 Oct 2009 01:30:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769746#action_12769746
] 

Christian Kunz commented on HDFS-732:
-------------------------------------

Here I post a message I sent to common-dev on September 4, about an incidence where
commitBlockSynchronization replaced a block with a block of smaller size.
This happened on hadoop-0.18.3, but it seemed to be very rare. Question is whether this is
the same problem as the one in the previous comment, and if so, whether something changed
in hadoop-0.20.1 to increase the rate of such incidences.

Anyhow, are there situations where block sizes can shrink? If not, this should <b>absolutely</b>
not be allowed.

Here are the relevant log messages of the namenode, showing the block size went from 273592
to 262144.

log.2009-09-01:2009-09-01 20:52:45,451 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.allocateBlock:
fileName.blk_2101888387690035515_63606967

log.2009-09-01:2009-09-01 20:57:43,535 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: xxx.yyy.zzz.234:54610 is added to blk_2101888387690035515_63606967
size 273592

log.2009-09-01:2009-09-01 20:57:47,847 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: xxx.yyy.zzz.250:54610 is added to blk_2101888387690035515_63606967
size 273592

log.2009-09-01:2009-09-01 20:58:01,480 INFO org.apache.hadoop.fs.FSNamesystem:
commitBlockSynchronization(lastblock=blk_2101888387690035515_63606967, newgenerationstamp=63637360,
newlength=262144,
newtargets=[xxx.yyy.zzz.234:54610])

log.2009-09-01:2009-09-01 20:58:01,482 INFO org.apache.hadoop.fs.FSNamesystem:
commitBlockSynchronization(blk_2101888387690035515_63637360) successful

log.2009-09-01:2009-09-01 21:01:07,508 INFO org.apache.hadoop.fs.FSNamesystem:
commitBlockSynchronization(lastblock=blk_2101888387690035515_63637360, newgenerationstamp=63639267,
newlength=262144,
newtargets=[xxx.yyy.zzz.234:54610])

log.2009-09-01:2009-09-01 21:01:07,512 INFO org.apache.hadoop.fs.FSNamesystem:
commitBlockSynchronization(blk_2101888387690035515_63639267) successful

log.2009-09-01:2009-09-01 21:01:08,946 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
ask xxx.yyy.zzz.234:54610 to replicate blk_2101888387690035515_63639267 to datanode(s) xxx.yyy.zzz.230:54610
xxx.yyy.zzz.115:54610

log.2009-09-01:2009-09-01 21:01:12,061 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: xxx.yyy.zzz.230:54610 is added to blk_2101888387690035515_63639267
size 262144

log.2009-09-01:2009-09-01 21:01:12,097 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: xxx.yyy.zzz.115:54610 is added to blk_2101888387690035515_63639267
size 262144

log.2009-09-01:2009-09-01 21:35:03,362 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.processReport: block blk_2101888387690035515_63637360 on xxx.yyy.zzz.250:54610
size 273592 does not belong to any file.

log.2009-09-01:2009-09-01 21:35:06,344 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
ask xxx.yyy.zzz.250:54610 to delete blk_2101888387690035515_63637360

> HDFS files are ending up truncated
> ----------------------------------
>
>                 Key: HDFS-732
>                 URL: https://issues.apache.org/jira/browse/HDFS-732
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.20.1
>            Reporter: Christian Kunz
>
> We recently started to use hadoop-0.20.1 in our production environment (less than 2 weeks
ago) and already had 3 instances of truncated files, more than we had for months using hadoop-0.18.3.
> Writing is done using libhdfs, although it rather seems to be a problem on the server
side.
> I will post some relevant logs (they are too large to be put into the description)
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message