Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 55CDCD2E4 for ; Thu, 14 Feb 2013 18:06:16 +0000 (UTC) Received: (qmail 95278 invoked by uid 500); 14 Feb 2013 18:06:11 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 95183 invoked by uid 500); 14 Feb 2013 18:06:11 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 95176 invoked by uid 99); 14 Feb 2013 18:06:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Feb 2013 18:06:11 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of uniquejeff@gmail.com designates 209.85.210.176 as permitted sender) Received: from [209.85.210.176] (HELO mail-ia0-f176.google.com) (209.85.210.176) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Feb 2013 18:06:04 +0000 Received: by mail-ia0-f176.google.com with SMTP id i18so2503845iac.21 for ; Thu, 14 Feb 2013 10:05:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=8w4xuteg1LQxXBn+57vfBg5gP8XEGKvyVbQVkjXDNxI=; b=ETLQ3ZXw2FtBXxsBqFx3AhddHzmosWDNVHLRVqC370bLOzf7TJ3B2j3JQP2FpNA/Ck jVNTKIvapMyqOPksnzGvvIighsnQmqOsZxAMNjixSgC/ZppAGyQskM+WWFmaYGch0g5C Ee3+GeDSQaXn/vKPFeqU1644y4BSg1sipw80BV06/VPNuIehPNvYlBGY/XV3SkmrHlAt dddLK8KfZWYLqq/6+LBIpOeJ8w+AKYVRV4IIws7fYi63MrNzFJLjs2eqf76NPKbvdtt6 NGVNaCtNbIcvWFyrPok8I3AH68J8GqJk0WnOKXZKgiC/JE29A7+scGEh3c86h4JH66PJ DePw== MIME-Version: 1.0 X-Received: by 10.50.12.137 with SMTP id y9mr381586igb.57.1360865125556; Thu, 14 Feb 2013 10:05:25 -0800 (PST) Received: by 10.64.18.49 with HTTP; Thu, 14 Feb 2013 10:05:25 -0800 (PST) In-Reply-To: <3e15a80e-2980-4eaf-b37e-ff7554582b38@email.android.com> References: <3e15a80e-2980-4eaf-b37e-ff7554582b38@email.android.com> Date: Thu, 14 Feb 2013 13:05:25 -0500 Message-ID: Subject: Re: Host NameNode, DataNode, JobTracker or TaskTracker on the same machine From: Jeff LI To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=14dae934095165628804d5b31a52 X-Virus-Checked: Checked by ClamAV on apache.org --14dae934095165628804d5b31a52 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks for your response. I'm running SNN on another machine. Could you explain a bit more on why I may run out of memory or disk? I understand that NameNode holds file system metadata in memory. I found through this post that ( http://developer.yahoo.com/blogs/hadoop/posts/2010/05/scalability_of_the_ha= doop_dist/ ) as a rule of thumb, 1 GB metadata =E2=89=88 1 PB physical storage Currently, my cluster has about 128TB of disk storage in total and 64GB memory on each machine. Does this suggests that I'm protected against running out of memory from metadata? Thanks Cheers Jeff On Thu, Feb 14, 2013 at 12:41 PM, Tariq wrote: > You may run out of memory,out of disk. If SNN is also running on the same > machine then you are totally screwed in case of any breakdown > > shashwat shriparv wrote: > > >If you are doing it for production all the process should be running on > >seperate machine as it will decrease the overload of the machine. > > > > > > > >=E2=88=9E > >Shashwat Shriparv > > > > > > > >On Thu, Feb 14, 2013 at 10:40 PM, Jeff LI wrote: > > > >> Hello, > >> > >> Is there a good reason that we should not host NameNode, DataNode, > >> JobTracker or TaskTracker services on the same machine? > >> > >> Not doing so is suggested here > >http://wiki.apache.org/hadoop/NameNode, > >> but I'd like to know the reasoning of this. > >> > >> Thanks > >> > >> Cheers > >> > >> Jeff > >> > >> > > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity. > --14dae934095165628804d5b31a52 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks for your response. =C2=A0I'm running SNN on another machine.
Could you explain a bit more on why I may run out of memory= or disk?

I understand that NameNode holds file sy= stem metadata in memory. =C2=A0I found through this post that (http://developer.yahoo.com/blogs/hadoop/posts/2010/05/scalabilit= y_of_the_hadoop_dist/)=C2=A0
as a rule of thumb,=C2=A0
1 GB metadata =E2=89=88 1 PB physi= cal storage

Currently, my cluster has about 128TB = of disk storage in total and 64GB memory on each machine. =C2=A0Does this s= uggests that I'm protected against running out of memory from metadata?= =C2=A0

Thanks

Cheers

Jeff


On Th= u, Feb 14, 2013 at 12:41 PM, Tariq <dontariq@gmail.com> wro= te:
You may run out of memory,out of disk. If SN= N is also running on the same machine then you are totally screwed in case = of any breakdown

shashwat shriparv <dwivedis= hashwat@gmail.com> wrote:

>If you are doing it for production all the process should be running on=
>seperate machine as it will decrease the overload of the machine.
>
>
>
>=E2=88=9E
>Shashwat Shriparv
>
>
>
>On Thu, Feb 14, 2013 at 10:40 PM, Jeff LI <uniquejeff@gmail.com> wrote:
>
>> Hello,
>>
>> Is there a good reason that we should not host NameNode, DataNode,=
>> JobTracker or TaskTracker services on the same machine?
>>
>> Not doing so is suggested here
>ht= tp://wiki.apache.org/hadoop/NameNode,
>> but I'd like to know the reasoning of this.
>>
>> Thanks
>>
>> Cheers
>>
>> Jeff
>>
>>

--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

--14dae934095165628804d5b31a52--