hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-89) files are not visible until they are closed
Date Thu, 09 Aug 2007 21:59:42 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-89?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12518853

Konstantin Shvachko commented on HADOOP-89:

I see 2 problems with this patch:
# It does not close any of the 3 issues raised in the description above.
You can see files in the ls, but you cannot see file progress and you still loose all accumulated
data if the file is not closed.
# Implementation-wise I would expect that when files are made visible we will get rid of the
pending create structure, 
but it is still there, so now you will have to synchronize pending create data with the corresponding

I'd propose to make files both visible and readable while they are still being created. Namely,
we should let
clients read portions of the file that have been written and replicated on data-nodes.

> files are not visible until they are closed
> -------------------------------------------
>                 Key: HADOOP-89
>                 URL: https://issues.apache.org/jira/browse/HADOOP-89
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.1.0
>            Reporter: Yoram Arnon
>            Assignee: dhruba borthakur
>            Priority: Critical
>         Attachments: atomicCreation.patch
> the current behaviour, whereby a file is not visible until it is closed has several flaws,including:
> 1. no practical way to know if a file/job is progressing
> 2. no way to implement files that never close, such as log files
> 3. failure to close a file results in loss of the file
> The part of the file that's written should be visible.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message