Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B8DAFD430 for ; Wed, 13 Feb 2013 20:07:31 +0000 (UTC) Received: (qmail 94982 invoked by uid 500); 13 Feb 2013 20:07:30 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 94871 invoked by uid 500); 13 Feb 2013 20:07:30 -0000 Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-user@hadoop.apache.org Delivered-To: mailing list hdfs-user@hadoop.apache.org Received: (qmail 94863 invoked by uid 99); 13 Feb 2013 20:07:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Feb 2013 20:07:30 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dontariq@gmail.com designates 209.85.212.43 as permitted sender) Received: from [209.85.212.43] (HELO mail-vb0-f43.google.com) (209.85.212.43) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Feb 2013 20:07:25 +0000 Received: by mail-vb0-f43.google.com with SMTP id fs19so1022474vbb.30 for ; Wed, 13 Feb 2013 12:07:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=FcDx6LC6rGtdiSjtRZFClCHgTRoodj2RgSualY3tUoM=; b=P0oMBcdtf/6qorWUjWUw0ngcSziymoIxce2yJITTEx/pwI0JpBhhs0bPVnRyEfm76C 5gEgCCNeNbGTx8pGx4hJ683CB1EWrwEwhU0NVq2QqDcD3d0nRYlPA+5LTEeT6vt3wcz2 uypPxeAbgit5nZAkJABVoEAKGiSKhn5HkGOlaidGpM2s4pNlrhbQhkjnVY+L3L0J0OG6 qotVdjvKgL1yvMMpF/o53gM2afz+Szmx/0oThGKj1fDpnOx0c2gYRs5w3mw1EOccAdgL etO9uMb49n2IfNmbzUpdhCaQdmdHmKMWDeBMP6XFIqMO1DFyRRjeBStaw9xkSCyuaWsW 5HHg== X-Received: by 10.52.175.66 with SMTP id by2mr27425743vdc.53.1360786025129; Wed, 13 Feb 2013 12:07:05 -0800 (PST) MIME-Version: 1.0 Received: by 10.59.8.227 with HTTP; Wed, 13 Feb 2013 12:06:25 -0800 (PST) In-Reply-To: References: From: Mohammad Tariq Date: Thu, 14 Feb 2013 01:36:25 +0530 Message-ID: Subject: Re: Managing space in Master Node To: "hdfs-user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=bcaec51a8260a4a91f04d5a0af3a X-Virus-Checked: Checked by ClamAV on apache.org --bcaec51a8260a4a91f04d5a0af3a Content-Type: text/plain; charset=ISO-8859-1 Just don't run the DN daemon on that machine. Dedicate that machine to NN. Remove the hostname of the NN machine from the 'slaves' file. Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot.com On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee < arkoprovomukherjee@gmail.com> wrote: > Hi, > > Thanks for the help! > > However, I am still unsure about how to "turn off" the datanode > feature on the NN. > > I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to > a directory. > > Should I just comment out that property? What would happen to the > current data that there is in the Master? Will it get removed > automatically? > > Thanks & regards > Arko > > > > On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq > wrote: > > You can specify the logging level as specified by Charles. But turning > logs > > off is never a good idea. Logs are really helpful in problem diagnosis, > > which are eventual. > > > > Warm Regards, > > Tariq > > https://mtariq.jux.com/ > > cloudfront.blogspot.com > > > > > > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee > > wrote: > >> > >> Hi, > >> > >> Yeah, my NameNode is also seconding as a DataNode. > >> > >> I would like to "turn off" this feature. > >> > >> Request help regarding the same. > >> > >> Thanks & regards > >> Arko > >> > >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker wrote: > >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode > which > >> > is > >> > not recommended practice. Normally, the only files the NN stores are > the > >> > image and edits files. It does not store any actual HDFS data. If you > >> > must > >> > run a DN on the NN, try turning down the logging in > >> > /conf/log4j.properties: > >> > > >> > #hadoop.root.logger=INFO,console > >> > #hadoop.root.logger=WARN,console > >> > hadoop.root.logger=ERROR,console > >> > > >> > Depending on the logging information you require, of course. > >> > > >> > -Chuck > >> > > >> > > >> > -----Original Message----- > >> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com] > >> > Sent: Wednesday, February 13, 2013 11:32 AM > >> > To: hdfs-user@hadoop.apache.org > >> > Subject: Managing space in Master Node > >> > > >> > Hello Gurus, > >> > > >> > I am managing a Hadoop Cluster to run some experiments. > >> > > >> > The issue I am continuously facing is that the Master Node runs out of > >> > disk > >> > space due to logs and data files. > >> > > >> > I can monitor and delete log files. However, I cannot delete the HDFS > >> > data. > >> > > >> > Thus, is there a way to force Hadoop not to save any HDFS data in the > >> > Master > >> > Node? > >> > > >> > Then I can use my master to handle the metadata only and store the > logs. > >> > > >> > Thanks & regards > >> > Arko > >> > SDL Enterprise Technologies, Inc. - all rights reserved. The > >> > information contained in this email may be confidential and/or legally > >> > privileged. It has been sent for the sole use of the intended > recipient(s). > >> > If you are not the intended recipient of this mail, you are hereby > notified > >> > that any unauthorized review, use, disclosure, dissemination, > distribution, > >> > or copying of this communication, or any of its contents, is strictly > >> > prohibited. If you have received this communication in error, please > reply > >> > to the sender and destroy all copies of the message. > >> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA > 01880, > >> > USA > >> > > > > > > --bcaec51a8260a4a91f04d5a0af3a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Just don't run the DN daemon on that machine. Dedicate= that machine to NN. Remove the hostname of the NN machine from the 'sl= aves' file.



On Thu, Feb 14, 2013 at 1:31 AM, Arko Pr= ovo Mukherjee <arkoprovomukherjee@gmail.com> wrot= e:
Hi,

Thanks for the help!

However, I am still unsure about how to "turn off" the datanode feature on the NN.

I checked the "hdfs-site.xml" file and the dfs.data.dir is pointe= d to
a directory.

Should I just comment out that property? What would happen to the
current data that there is in the Master? Will it get removed
automatically?

Thanks & regards
Arko



On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <dontariq@gmail.com> wrote:
> You can specify the logging level as specified by Charles. But turning= logs
> off is never a good idea. Logs are really helpful in problem diagnosis= ,
> which are eventual.
>
> Warm Regards,
> Tariq
> https://mtariq.j= ux.com/
> cloudfron= t.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> <arkoprovomukherjee= @gmail.com> wrote:
>>
>> Hi,
>>
>> Yeah, my NameNode is also seconding as a DataNode.
>>
>> I would like to "turn off" this feature.
>>
>> Request help regarding the same.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cbaker@sdl.com> wrote:
>> > Hi Arko. Sounds like you may be running a DataNode on the Nam= eNode which
>> > is
>> > not recommended practice. Normally, the only files the NN sto= res are the
>> > image and edits files. It does not store any actual HDFS data= . If you
>> > must
>> > run a DN on the NN, try turning down the logging in
>> > /conf/log4j.properties:
>> >
>> > #hadoop.root.logger=3DINFO,console
>> > #hadoop.root.logger=3DWARN,console
>> > hadoop.root.logger=3DERROR,console
>> >
>> > Depending on the logging information you require, of course.<= br> >> >
>> > -Chuck
>> >
>> >
>> > -----Original Message-----
>> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>> > Sent: Wednesday, February 13, 2013 11:32 AM
>> > To: hdfs-user@= hadoop.apache.org
>> > Subject: Managing space in Master Node
>> >
>> > Hello Gurus,
>> >
>> > I am managing a Hadoop Cluster to run some experiments.
>> >
>> > The issue I am continuously facing is that the Master Node ru= ns out of
>> > disk
>> > space due to logs and data files.
>> >
>> > I can monitor and delete log files. However, I cannot delete = the HDFS
>> > data.
>> >
>> > Thus, is there a way to force Hadoop not to save any HDFS dat= a in the
>> > Master
>> > Node?
>> >
>> > Then I can use my master to handle the metadata only and stor= e the logs.
>> >
>> > Thanks & regards
>> > Arko
>> > SDL Enterprise Technologies, Inc. - all rights reserved. =A0T= he
>> > information contained in this email may be confidential and/o= r legally
>> > privileged. It has been sent for the sole use of the intended= recipient(s).
>> > If you are not the intended recipient of this mail, you are h= ereby notified
>> > that any unauthorized review, use, disclosure, dissemination,= distribution,
>> > or copying of this communication, or any of its contents, is = strictly
>> > prohibited. If you have received this communication in error,= please reply
>> > to the sender and destroy all copies of the message.
>> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield= , MA 01880,
>> > USA
>> >
>
>

--bcaec51a8260a4a91f04d5a0af3a--