hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jack Levin <magn...@gmail.com>
Subject Re: Question about dead datanode
Date Wed, 26 Feb 2014 19:54:01 GMT
Submitted JIRA patch: https://issues.apache.org/jira/browse/HDFS-6022
(with test)

On Mon, Feb 24, 2014 at 12:16 PM, Jack Levin <magnito@gmail.com> wrote:
> I will do that.
>
> -Jack
>
> On Mon, Feb 24, 2014 at 6:23 AM, Steve Loughran <stevel@hortonworks.com> wrote:
>> that's a very old version of cloudera's branch you are working with there;
>> patching that is not a good way to go, as you are on the slippery slope of
>> having your own private branch and all the costs of it.
>>
>> It looks like dead node logic has -> DFSInputStream, where it is still
>> instance-specific:
>>
>>   /* XXX Use of CocurrentHashMap is temp fix. Need to fix
>>    * parallel accesses to DFSInputStream (through ptreads) properly */
>>   private final ConcurrentHashMap<DatanodeInfo, DatanodeInfo> deadNodes =
>>              new ConcurrentHashMap<DatanodeInfo, DatanodeInfo>();
>>
>> This implies the problem still exists -and the opportunity to fix it -but
>> you will need to modify your patch to apply to hadoop trunk, ideally think
>> of a test, then submit a patch to the HDFS project on JIRA.
>>
>>
>> On 19 February 2014 04:48, Stack <stack@duboce.net> wrote:
>>
>>> On Sat, Feb 15, 2014 at 8:01 PM, Jack Levin <magnito@gmail.com> wrote:
>>>
>>> > Looks like I patched it in DFSClient.java, here is the patch:
>>> > https://gist.github.com/anonymous/9028934
>>> >
>>> > ....
>>>
>>>
>>> > I moved 'deadNodes' list outside as global field that is accessible by
>>> > all running threads, so at any point datanode does go down, each
>>> > thread is basically informed that the datanode _is_ down.
>>> >
>>>
>>> We need to add something like this to current versions of DFSClient, a
>>> global status, so each stream does not have to discover bad DNs for itself.
>>> St.Ack
>>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.

Mime
View raw message