Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7F12EDBE9 for ; Mon, 1 Oct 2012 08:35:24 +0000 (UTC) Received: (qmail 94549 invoked by uid 500); 1 Oct 2012 08:35:19 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 94375 invoked by uid 500); 1 Oct 2012 08:35:14 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 94364 invoked by uid 99); 1 Oct 2012 08:35:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Oct 2012 08:35:14 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of liulei412@gmail.com designates 209.85.223.176 as permitted sender) Received: from [209.85.223.176] (HELO mail-ie0-f176.google.com) (209.85.223.176) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Oct 2012 08:35:07 +0000 Received: by ieak11 with SMTP id k11so13864125iea.35 for ; Mon, 01 Oct 2012 01:34:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=eM5KtOMHrgjuFitT8tF6flmdfirPz7LgRvIUx6K+gls=; b=b6BCBi471pcFdF71oyYRvwe9JpY4PueYiUMe1gV+WJ4oOOhgIyOCuGGtmW91uimw6t FKCk6+T7ybTQd5ormII4JL8Pq3ez1vregaih2iRuphHLrMbriqBLJHUydR4njomA9z/m JjGIvQS5Qy/VlHMYWp6oWahkPEBOoNP8di+W4Y1z4m9IA4ahuNUQMp2bqiv93YLFlILs 0SeGSMCJ47XBWFFK98mAUyBNrdVwnmJlTC+6cfCF+KzjXYwhc+6Hjrh8nSjfvPxrff5q wBlfmrPhDT9WOw2afRCTMVlx10oEw+/0WVXq9mxzuLAvrpc4GhPi+FXaIvs3BovmIpzm oCNw== MIME-Version: 1.0 Received: by 10.42.18.8 with SMTP id v8mr10589383ica.16.1349080486788; Mon, 01 Oct 2012 01:34:46 -0700 (PDT) Received: by 10.64.168.72 with HTTP; Mon, 1 Oct 2012 01:34:46 -0700 (PDT) In-Reply-To: References: Date: Mon, 1 Oct 2012 16:34:46 +0800 Message-ID: Subject: Re: DFSClient may read wrong data in local read From: jlei liu To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=20cf304271b6303d4004cafb471f --20cf304271b6303d4004cafb471f Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: quoted-printable Hi Colin, Thanks for your reply. What is mean that the patch will work on files that are in the process of being written? Thanks, LiuLei 2012/10/1 Colin McCabe > I'm going to post a patch to HDFS-347 shortly. From the user's point > of view, the important thing about the new implementation of local > reads is that it will work with security, and will work on files that > are in the process of being written. > > cheers, > Colin > > > On Thu, Sep 27, 2012 at 9:30 PM, jlei liu wrote: > > Hi Colin=A3=AC thanks for your reply. > > > > Where can I see the new design for BlockReaderLocal class? > > > > Thanks, > > > > LiuLei > > > > > > > > 2012/9/28 Colin McCabe > >> > >> We don't make very strong guarantees about what happens when clients > >> read from a deleted file. DFSClients definitely may read data from a > >> deleted file even if local reads are not enabled. > >> > >> Incidentally, BlockReaderLocal is being redesigned to pass file > >> descriptors rather than paths, which will be more secure and fix some > >> corner cases surrounding append and local reads. > >> > >> cheers, > >> Colin > >> > >> > >> On Wed, Sep 26, 2012 at 11:19 PM, jlei liu wrote= : > >> > In local read, BlockReaderLocal class use "static Map >> > LocalDatanodeInfo> localDatanodeInfoMap" property to store local blo= ck > >> > file > >> > path and local meta file path. When I stop HDFS cluster or I kill th= e > >> > local > >> > DataNode and delete file use "./hadoop dfs -rm path" command , the > >> > DFSClient > >> > still can read the data form local file. I think that may lead to > >> > DFSClient > >> > read wrong data. > >> > > >> > I think we should fix the problem. > >> > > >> > > >> > Thanks, > >> > > >> > LiuLei > > > > > --20cf304271b6303d4004cafb471f Content-Type: text/html; charset=GB2312 Content-Transfer-Encoding: quoted-printable
Hi Colin, Thanks for your reply.
 
What is me= an that the patch will work on files that are in the process of being writt= en?
 
Thanks,
LiuLei


2012/10/1 Colin McCabe <cmccabe@alumni.cmu.edu>
I'm going to post a patch to HDFS-347 shortly.  From the user'= s point
of view, the important thing about the new implementation of local
reads is that it will work with security, and will work on files that
are in the process of being written.

cheers,
Colin


On Thu, Sep 27, 2012 at 9:30 PM, jlei liu <liulei412@gmail.com> wrote:
> Hi Colin=A3=AC thanks for your reply.
>
> Where can I see the new design for BlockReaderLocal class?
>
> Thanks,
>
> LiuLei
>
>
>
> 2012/9/28 Colin McCabe <c= mccabe@alumni.cmu.edu>
>>
>> We don't make very strong guarantees about what happens when c= lients
>> read from a deleted file.  DFSClients definitely may read dat= a from a
>> deleted file even if local reads are not enabled.
>>
>> Incidentally, BlockReaderLocal is being redesigned to pass file >> descriptors rather than paths, which will be more secure and fix s= ome
>> corner cases surrounding append and local reads.
>>
>> cheers,
>> Colin
>>
>>
>> On Wed, Sep 26, 2012 at 11:19 PM, jlei liu <liulei412@gmail.com> wrote:
>> > In local read, BlockReaderLocal class use "static Map<= ;Integer,
>> > LocalDatanodeInfo> localDatanodeInfoMap" property to = store local block
>> > file
>> > path and local meta file path. When I stop HDFS cluster or I = kill the
>> > local
>> > DataNode and delete file use "./hadoop dfs -rm path"= ; command , the
>> > DFSClient
>> > still can read the data form local file. I think that may lea= d to
>> > DFSClient
>> > read wrong data.
>> >
>> > I think we should fix the problem.
>> >
>> >
>> > Thanks,
>> >
>> > LiuLei
>
>

--20cf304271b6303d4004cafb471f--