Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of pallavi.palleti@corp.aol.com
 designates 64.12.206.40 as permitted sender)
Message-ID: <4BB33232.8050400@corp.aol.com>
Date: Wed, 31 Mar 2010 16:59:54 +0530
From: Pallavi Palleti <pallavi.palleti@corp.aol.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
 rv:1.9.1.8) Gecko/20100301 Fedora/3.0.3-1.fc12 Lightning/1.0b1
 Thunderbird/3.0.3
MIME-Version: 1.0
To: common-user@hadoop.apache.org
Subject: Re: Query over DFSClient
References: <4BB1E10A.7060609@corp.aol.com>
In-Reply-To: <4BB1E10A.7060609@corp.aol.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

I am looking into hadoop-20 source code for below issue. From DFSClient, 
I could see that once the datanodes given by namenode are not reachable, 
it is setting "lastException" variable to error message saying "recovery 
from primary datanode is failed N times, aborting.."(line No:2546 in 
processDataNodeError).  However, I couldn't figure out where this 
exception is thrown. I could see the throw statement in isClosed() but 
not finding the exact sequence after Streamer exits with lastException 
set to isClosed() method call. It would be great if some one could shed 
some light on this. I am essentially looking whether DFSClient 
approaches namenode in the case of failure of all datanodes that 
namenode has given for a given data block previously.

Thanks
Pallavi


On 03/30/2010 05:01 PM, Pallavi Palleti wrote:
> Hi,
>
> Could some one kindly let me know if the DFSClient takes care of 
> datanode failures and attempt to write to another datanode if primary 
> datanode (and replicated datanodes) fail. I looked into the souce code 
> of DFSClient and figured out that it attempts to write to one of the 
> datanodes in pipeline and fails if it failed to write to at least one 
> of them. However, I am not sure as I haven't explored fully. If so, is 
> there a way of querying namenode to provide different datanodes in the 
> case of failure. I am sure the Mapper would be doing similar 
> thing(attempting to fetch different datanode from namenode)  if it 
> fails to write to datanodes. Kindly let me know.
>
> Thanks
> Pallavi
>