Return-Path: 
 <hadoop-dev-return-13569-apmail-lucene-hadoop-dev-archive=lucene.apache.org@lucene.apache.org>
Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org
Received: (qmail 86800 invoked from network); 20 Jun 2007 00:40:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 20 Jun 2007 00:40:47 -0000
Received: (qmail 88001 invoked by uid 500); 20 Jun 2007 00:40:50 -0000
Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org
Received: (qmail 87710 invoked by uid 500); 20 Jun 2007 00:40:50 -0000
Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hadoop-dev-help@lucene.apache.org>
List-Unsubscribe: <mailto:hadoop-dev-unsubscribe@lucene.apache.org>
List-Post: <mailto:hadoop-dev@lucene.apache.org>
List-Id: <hadoop-dev.lucene.apache.org>
Reply-To: hadoop-dev@lucene.apache.org
Delivered-To: mailing list hadoop-dev@lucene.apache.org
Received: (qmail 87700 invoked by uid 99); 20 Jun 2007 00:40:50 -0000
Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jun 2007 17:40:50 -0700
X-ASF-Spam-Status: No, hits=-100.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.4] (HELO brutus.apache.org) (140.211.11.4)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jun 2007 17:40:46 -0700
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 21E44714191
	for <hadoop-dev@lucene.apache.org>; Tue, 19 Jun 2007 17:40:26 -0700 (PDT)
Message-ID: <6213003.1182300026135.JavaMail.jira@brutus>
Date: Tue, 19 Jun 2007 17:40:26 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Updated: (HADOOP-1292) dfs -copyToLocal should guarantee
 file is complete
In-Reply-To: <3642053.1177435635281.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-1292:
-------------------------------------------

    Attachment:     (was: HADOOP-1292_20070619.patch)

> dfs -copyToLocal should guarantee file is complete
> --------------------------------------------------
>
>                 Key: HADOOP-1292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1292
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>            Reporter: eric baldeschwieler
>
> We should copy to a temporary file, maybe _tmp.<realname>, and then rename the file when the copy is complete.  Restarting a copy should reuse the _tmp file, just checksumming it.  Then ^Cing a copy will do the right thing.
> Original suggestion:
> On Apr 23, 2007, at 2:38 AM, Richard Kasperski wrote:
> I'd like to have a guarantee that a file copy is both completed and that the file is whole. In the past I've done this  by copying the file to a temporary name tmp.<realname> and then moving it to <realname> once I have the file copy is complete. This has the following very nice properties; If the <realname> exists then the file copy is complete and I'm not looking at a partial copy of the file. I believe that the copy to the cluster has both of these properties in that the file doesn't appear in a DFS directory until the whole file has been copied. The copy from the cluster to a local file system does not have these guarantees and it would be very nice if it did. There are two scenarios under what I wish to use this. First is that if I ctrl-c the 'hadoop dfs -copyToLocal' I know what parts are complete and what parts aren't. Second I can run a background compressor to compress the files as they are copied.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.