hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Kunz (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-732) HDFS files are ending up truncated
Date Wed, 28 Oct 2009 19:51:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771082#action_12771082
] 

Christian Kunz commented on HDFS-732:
-------------------------------------

>From what I see in the code, 'closed = true' is set in processDatanodeError after any
exception, and in the finally clause of closeInternal.

Seemingly multiple tries of close must have been introduced in the past for some early version
of hadoop (maybe hadoop-0.15 or earlier) when it reduced the number of task failures, and
since then we did not have problems with it (although without any beneficial effects) till
hadoop-0.20.1.

Okay, we will change our applications to not retry a close, assuming that the output stream
should not be used after a call to close, whether successful or not.

Maybe, hdfsCloseFile in libhdfs should be changed as well to deallocate resources not only
when it is successful.

> HDFS files are ending up truncated
> ----------------------------------
>
>                 Key: HDFS-732
>                 URL: https://issues.apache.org/jira/browse/HDFS-732
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.20.1
>            Reporter: Christian Kunz
>
> We recently started to use hadoop-0.20.1 in our production environment (less than 2 weeks
ago) and already had 3 instances of truncated files, more than we had for months using hadoop-0.18.3.
> Writing is done using libhdfs, although it rather seems to be a problem on the server
side.
> I will post some relevant logs (they are too large to be put into the description)
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message