Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of edlinuxguru@gmail.com
 designates 209.85.217.228 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=Lf2S5rkdwo8hg190ecOQU7073RvFzuUa0gLqU5elisqaFSK7wYLQ6nTVoZMsg7HBTC
         il82+C0WklqnQmoKhqC79W5ErxGm5Y0hEG3SMdf7E/QE4vB43lqsjz23wCVV93trlTBa
         tAPg5dx0HC6HJMld4v7zZz7L0E9KryJzSg2EQ=
MIME-Version: 1.0
In-Reply-To: <C701FFC7.EC8%awittenauer@linkedin.com>
References: <4ADC6D81.6070204@galatea.com>
	 <C701FFC7.EC8%awittenauer@linkedin.com>
Date: Mon, 19 Oct 2009 14:46:33 -0400
Message-ID: <cbbf4b570910191146p32d490c0s668efc11c8e53d5d@mail.gmail.com>
Subject: Re: editing etc hosts files of a cluster
From: Edward Capriolo <edlinuxguru@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Oct 19, 2009 at 2:36 PM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
> A bit more specific:
>
> At Yahoo!, we had either every server as a DNS slave or a DNS caching
> server.
>
> In the case of LinkedIn, we're running Solaris so nscd is significantly
> better than its Linux counterpart. =A0However, we still seem to be blowin=
g out
> the cache too much. =A0So we'll likely switch to DNS caching servers here=
 as
> well.
>
> On 10/19/09 6:45 AM, "Last-chance Architect" <architect@galatea.com> wrot=
e:
>
>> DNS ;)
>>
>> Ramesh.Ramasamy wrote:
>>> Hi,
>>>
>>> I have a cluster setup with 3 nodes, and I'm adding hostname details (i=
n
>>> /etc/hosts) manually in each node. Seems it is not an effective approac=
h.
>>> How this scenario is handled in big clusters?
>>>
>>> Is there any simple of way to add the hostname details in all the nodes=
 by
>>> editing a single entry/file/script?
>>>
>>> Thanks and Regards,
>>> Ramesh
>>>
>>>
>
>

Allan,

I am interested in your post. What has caused you to run caching DNS
servers on each of your nodes? Is this a hadoop specific problem or a
problem  specific to your implementation?

My assumption here is that a hadoop cluster of say 1000 nodes would
repeatedly talk to the same 1000 nodes. Are you saying that nscd is
inadequacy to handle the size of the cache, or nscd is not very
efficient? What exactly is the reason you are running a caching DNS
server on each node?

Thank you,
Edward