hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: editing etc hosts files of a cluster
Date Mon, 19 Oct 2009 18:46:33 GMT
On Mon, Oct 19, 2009 at 2:36 PM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
> A bit more specific:
> At Yahoo!, we had either every server as a DNS slave or a DNS caching
> server.
> In the case of LinkedIn, we're running Solaris so nscd is significantly
> better than its Linux counterpart.  However, we still seem to be blowing out
> the cache too much.  So we'll likely switch to DNS caching servers here as
> well.
> On 10/19/09 6:45 AM, "Last-chance Architect" <architect@galatea.com> wrote:
>> DNS ;)
>> Ramesh.Ramasamy wrote:
>>> Hi,
>>> I have a cluster setup with 3 nodes, and I'm adding hostname details (in
>>> /etc/hosts) manually in each node. Seems it is not an effective approach.
>>> How this scenario is handled in big clusters?
>>> Is there any simple of way to add the hostname details in all the nodes by
>>> editing a single entry/file/script?
>>> Thanks and Regards,
>>> Ramesh


I am interested in your post. What has caused you to run caching DNS
servers on each of your nodes? Is this a hadoop specific problem or a
problem  specific to your implementation?

My assumption here is that a hadoop cluster of say 1000 nodes would
repeatedly talk to the same 1000 nodes. Are you saying that nscd is
inadequacy to handle the size of the cache, or nscd is not very
efficient? What exactly is the reason you are running a caching DNS
server on each node?

Thank you,

View raw message