hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jlei liu <liulei...@gmail.com>
Subject Re: DFSClient may read wrong data in local read
Date Mon, 01 Oct 2012 09:02:26 GMT
Hi Colin,

We can use Unix Domian Socket to implement Local Read.  After the Local
DataNode  is dead, the client read data from Local DataNode will throw
Exception,  so I think use Unix Domain Sokcet to implement Local Read is
more reliable.
2012/10/1 lei liu <liulei412@gmail.com>

> Hi Colin, Thanks for your reply.
>
> What is mean that the patch will work on files that are in the process of
> being written?
>
> Thanks,
> LiuLei
>
>
> 2012/10/1 Colin McCabe <cmccabe@alumni.cmu.edu>
>
>> I'm going to post a patch to HDFS-347 shortly.  From the user's point
>> of view, the important thing about the new implementation of local
>> reads is that it will work with security, and will work on files that
>> are in the process of being written.
>>
>> cheers,
>> Colin
>>
>>
>> On Thu, Sep 27, 2012 at 9:30 PM, jlei liu <liulei412@gmail.com> wrote:
>> > Hi Colin´╝î thanks for your reply.
>> >
>> > Where can I see the new design for BlockReaderLocal class?
>> >
>> > Thanks,
>> >
>> > LiuLei
>> >
>> >
>> >
>> > 2012/9/28 Colin McCabe <cmccabe@alumni.cmu.edu>
>> >>
>> >> We don't make very strong guarantees about what happens when clients
>> >> read from a deleted file.  DFSClients definitely may read data from a
>> >> deleted file even if local reads are not enabled.
>> >>
>> >> Incidentally, BlockReaderLocal is being redesigned to pass file
>> >> descriptors rather than paths, which will be more secure and fix some
>> >> corner cases surrounding append and local reads.
>> >>
>> >> cheers,
>> >> Colin
>> >>
>> >>
>> >> On Wed, Sep 26, 2012 at 11:19 PM, jlei liu <liulei412@gmail.com>
>> wrote:
>> >> > In local read, BlockReaderLocal class use "static Map<Integer,
>> >> > LocalDatanodeInfo> localDatanodeInfoMap" property to store local
>> block
>> >> > file
>> >> > path and local meta file path. When I stop HDFS cluster or I kill the
>> >> > local
>> >> > DataNode and delete file use "./hadoop dfs -rm path" command , the
>> >> > DFSClient
>> >> > still can read the data form local file. I think that may lead to
>> >> > DFSClient
>> >> > read wrong data.
>> >> >
>> >> > I think we should fix the problem.
>> >> >
>> >> >
>> >> > Thanks,
>> >> >
>> >> > LiuLei
>> >
>> >
>>
>
>

Mime
View raw message