Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 06790D4A3 for ; Wed, 13 Feb 2013 20:21:44 +0000 (UTC) Received: (qmail 50037 invoked by uid 500); 13 Feb 2013 20:21:42 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 49920 invoked by uid 500); 13 Feb 2013 20:21:42 -0000 Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-user@hadoop.apache.org Delivered-To: mailing list hdfs-user@hadoop.apache.org Received: (qmail 49911 invoked by uid 99); 13 Feb 2013 20:21:42 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Feb 2013 20:21:42 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of prvs=749fb12fd=cbaker@sdl.com designates 65.100.157.33 as permitted sender) Received: from [65.100.157.33] (HELO mailus.sdl.com) (65.100.157.33) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Feb 2013 20:21:38 +0000 X-Disclaimer: None X-IronPort-AV: E=Sophos;i="4.84,659,1355097600"; d="scan'208";a="9362750" Received: from usmail0301.global.sdl.corp ([10.32.3.3]) by ushub0301.sdl.corp with Microsoft SMTPSVC(6.0.3790.4675); Wed, 13 Feb 2013 12:21:14 -0800 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: Managing space in Master Node Date: Wed, 13 Feb 2013 13:21:14 -0700 Message-ID: In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Managing space in Master Node Thread-Index: Ac4KJpm8IQ0lZFqsRNWEZyOAtpAfPgAAJClA References: From: "Charles Baker" To: X-OriginalArrivalTime: 13 Feb 2013 20:21:14.0159 (UTC) FILETIME=[ABBC9BF0:01CE0A27] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org The best way is to decommission the DN so the blocks get replicated to the other nodes in the cluster. Manually copying the data is not recommended. S= ee if this works for you: http://wiki.apache.org/hadoop/FAQ#I_want_to_make_a_large_cluster_smaller_by= _t aking_out_a_bunch_of_nodes_simultaneously._How_can_this_be_done.3F -----Original Message----- From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]=20 Sent: Wednesday, February 13, 2013 12:13 PM To: hdfs-user@hadoop.apache.org Subject: Re: Managing space in Master Node Thanks! However, in that case, how do I handle the current data that is already in NN? Do I have to reformat HDFS? Thanks & regards Arko On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq wrote: > Just don't run the DN daemon on that machine. Dedicate that machine to NN. > Remove the hostname of the NN machine from the 'slaves' file. > > Warm Regards, > Tariq > https://mtariq.jux.com/ > cloudfront.blogspot.com > > > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee=20 > wrote: >> >> Hi, >> >> Thanks for the help! >> >> However, I am still unsure about how to "turn off" the datanode=20 >> feature on the NN. >> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to=20 >> a directory. >> >> Should I just comment out that property? What would happen to the=20 >> current data that there is in the Master? Will it get removed=20 >> automatically? >> >> Thanks & regards >> Arko >> >> >> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq >> wrote: >> > You can specify the logging level as specified by Charles. But=20 >> > turning logs off is never a good idea. Logs are really helpful in=20 >> > problem diagnosis, which are eventual. >> > >> > Warm Regards, >> > Tariq >> > https://mtariq.jux.com/ >> > cloudfront.blogspot.com >> > >> > >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee=20 >> > wrote: >> >> >> >> Hi, >> >> >> >> Yeah, my NameNode is also seconding as a DataNode. >> >> >> >> I would like to "turn off" this feature. >> >> >> >> Request help regarding the same. >> >> >> >> Thanks & regards >> >> Arko >> >> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker wrote: >> >> > Hi Arko. Sounds like you may be running a DataNode on the=20 >> >> > NameNode which is not recommended practice. Normally, the only=20 >> >> > files the NN stores are the image and edits files. It does not=20 >> >> > store any actual HDFS data. If you must run a DN on the NN, try=20 >> >> > turning down the logging in >> >> > /conf/log4j.properties: >> >> > >> >> > #hadoop.root.logger=3DINFO,console=20 >> >> > #hadoop.root.logger=3DWARN,console=20 >> >> > hadoop.root.logger=3DERROR,console >> >> > >> >> > Depending on the logging information you require, of course. >> >> > >> >> > -Chuck >> >> > >> >> > >> >> > -----Original Message----- >> >> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com] >> >> > Sent: Wednesday, February 13, 2013 11:32 AM >> >> > To: hdfs-user@hadoop.apache.org >> >> > Subject: Managing space in Master Node >> >> > >> >> > Hello Gurus, >> >> > >> >> > I am managing a Hadoop Cluster to run some experiments. >> >> > >> >> > The issue I am continuously facing is that the Master Node runs=20 >> >> > out of disk space due to logs and data files. >> >> > >> >> > I can monitor and delete log files. However, I cannot delete the=20 >> >> > HDFS data. >> >> > >> >> > Thus, is there a way to force Hadoop not to save any HDFS data=20 >> >> > in the Master Node? >> >> > >> >> > Then I can use my master to handle the metadata only and store=20 >> >> > the logs. >> >> > >> >> > Thanks & regards >> >> > Arko >> >> > SDL Enterprise Technologies, Inc. - all rights reserved. The=20 >> >> > information contained in this email may be confidential and/or=20 >> >> > legally privileged. It has been sent for the sole use of the=20 >> >> > intended recipient(s). >> >> > If you are not the intended recipient of this mail, you are=20 >> >> > hereby notified that any unauthorized review, use, disclosure,=20 >> >> > dissemination, distribution, or copying of this communication,=20 >> >> > or any of its contents, is strictly prohibited. If you have=20 >> >> > received this communication in error, please reply to the sender=20 >> >> > and destroy all copies of the message. >> >> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield,=20 >> >> > MA 01880, USA >> >> > >> > >> > > > SDL Enterprise Technologies, Inc. - all rights reserved. The information c= ontained in this email may be confidential and/or legally privileged. It ha= s been sent for the sole use of the intended recipient(s). If you are not t= he intended recipient of this mail, you are hereby notified that any unauth= orized review, use, disclosure, dissemination, distribution, or copying of = this communication, or any of its contents, is strictly prohibited. If you = have received this communication in error, please reply to the sender and d= estroy all copies of the message. Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA