Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of prvs=751331ceb=cbaker@sdl.com
 designates 65.100.157.33 as permitted sender)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Subject: RE: Managing space in Master Node
Date: Fri, 15 Feb 2013 14:10:33 -0700
Message-ID: 
 <B34E9F024DB2F441879FDAE698E892A101B5F8D2@usmail0301.global.sdl.corp>
In-Reply-To: 
 <CAD+YOd8UbXWDZY=uOeuOn=SHe39qB0hFpbsYA06U7ZJdn4tkDg@mail.gmail.com>
Thread-Topic: Managing space in Master Node
Thread-Index: Ac4Lto9uXcyHYnjFQz2yaNxcNKNlbwACiqMA
References: 
 <CAD+YOd_8M0QO2YW-KfHeG+i2Di0v8WNd3aqOT9-qrxYDOt9HUw@mail.gmail.com>
 <3BC058F5B0CB894299B8102FC71CDA940CE9120B@048-CH1MPN1-152.048d.mgd.msft.net>
 <CAD+YOd9Md5EhWH4UR6mwi8tuy2ahEpFAFW-Onm7BCq9mQZ_4sA@mail.gmail.com>
 <CAD+YOd-7iRNHtrpzAOQjcPEk+z7-JfBr6zx7hdXZw82q8uHikQ@mail.gmail.com>
 <B34E9F024DB2F441879FDAE698E892A101B5F7A5@usmail0301.global.sdl.corp>
 <CAD+YOd8UbXWDZY=uOeuOn=SHe39qB0hFpbsYA06U7ZJdn4tkDg@mail.gmail.com>
From: "Charles Baker" <cbaker@sdl.com>
To: <hdfs-user@hadoop.apache.org>
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hey Arko. It should be safe to delete then.

-Chuck


-----Original Message-----
From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]=20
Sent: Friday, February 15, 2013 11:56 AM
To: hdfs-user@hadoop.apache.org
Subject: Re: Managing space in Master Node

Hello Charles,

Thanks a lot for your reply and help!

Yes, the NN data (image, edit files) is kept separate from the data files.
"dfs.name.dir=3D/hadoop/hdfs/name" whereas "dfs.data.dir=3D/hadoop/hdfs/dat=
a".

Their contents match with the description you specified

Can I safely go ahead and delete all contents from the data dir?

Thanks & regards
Arko

On Thu, Feb 14, 2013 at 8:16 PM, Charles Baker <cbaker@sdl.com> wrote:
> Hey Arko. Glad to hear it worked out for you. Just make sure that your=20
> NN's image and edit files aren't also (perhaps inadvertently) stored=20
> in that directory structure. The NameNode image/edit dir contains the
following:
>
> drwxr-xr-x 2 hdfs hdfs 4096 Feb 14 17:48 current drwxr-xr-x 2 hdfs=20
> hdfs 4096 Jul 17  2012 image
> -rw-r--r-- 1 hdfs hdfs    0 Sep 27 12:28 in_use.lock
> drwxr-xr-x 2 hdfs hdfs 4096 Sep 17 23:25 previous.checkpoint
>
> whereas, the DN data dir should contain:
>
> drwxr-xr-x  2 hdfs hdfs 57344 Feb 14 14:04 blocksBeingWritten=20
> drwxr-xr-x 66 hdfs hdfs  8192 Jan 22 02:44 current
> drwxr-xr-x  2 hdfs hdfs     6 Aug 10  2012 detach
> -rw-r--r--  1 hdfs hdfs     0 Sep 27 11:19 in_use.lock
> -rw-r--r--  1 hdfs hdfs   157 Aug 10  2012 storage
> drwxr-xr-x  2 hdfs hdfs     6 Feb  4 17:34 tmp
>
>
> It's a good practice to locate the JobTracker and SecondaryNameNode=20
> services on a different machine than the NN. If you have an=20
> opportunity to do so, I recommend getting a separate machine for those two
services as well.
>
> -Chuck
>
>
> -----Original Message-----
> From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
> Sent: Thursday, February 14, 2013 4:21 PM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: Managing space in Master Node
>
> Hello Everyone - I have decommissioned my NN so that only the=20
> JobTracker, NameNode and SecondaryNameNodes are running on it now.
> I would like to thank everyone who helped me to perform this task!
>
> I have just one more issue: The data files created in the Master Node=20
> seem to be still there. in /hdfs/data/current Now that this server is=20
> not being used as a DataNode, can I safely delete the files under this=20
> folder to clear up space?
>
> Thanks & regards
> Arko
>
> On Thu, Feb 14, 2013 at 11:48 AM, Arko Provo Mukherjee=20
> <arkoprovomukherjee@gmail.com> wrote:
>> Thanks! I have already requested for downtime. Will do the changes soon!
>> Warm regards
>> Arko
>>
>> On Thu, Feb 14, 2013 at 3:31 AM,  <ramon.pin@accenture.com> wrote:
>>> Hi Arko.
>>>
>>>   Only thing you need to do is not running the TaskTracker and=20
>>> DataNode
> demons on your master machine. Ensure you do not have this on your=20
> slaves file con hadoop's config directory when you start the system.=20
> I'm supposing you are on Open Source release. For other distributions=20
> look at its documentation about how to remove those demons from your mast=
er
node machine.
>>>
>>>
>>> -----Original Message-----
>>> From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>>> Sent: mi=E9rcoles, 13 de febrero de 2013 20:32
>>> To: hdfs-user@hadoop.apache.org
>>> Subject: Managing space in Master Node
>>>
>>> Hello Gurus,
>>>
>>> I am managing a Hadoop Cluster to run some experiments.
>>>
>>> The issue I am continuously facing is that the Master Node runs out=20
>>> of disk space due to logs and data files.
>>>
>>> I can monitor and delete log files. However, I cannot delete the=20
>>> HDFS
> data.
>>>
>>> Thus, is there a way to force Hadoop not to save any HDFS data in=20
>>> the Master Node?
>>>
>>> Then I can use my master to handle the metadata only and store the logs.
>>>
>>> Thanks & regards
>>> Arko
>>>
>>>
>>> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information. If you have=20
> received it in error, please notify the sender immediately and delete=20
> the original. Any other use of the e-mail by you is prohibited.
>>>
>>> Where allowed by local law, electronic communications with Accenture=20
>>> and
> its affiliates, including e-mail and instant messaging (including=20
> content), may be scanned by our systems for the purposes of=20
> information security and assessment of internal compliance with Accenture
policy.
>>>
>>> ____________________________________________________________________
>>> _
>>> _________________
>>>
>>> www.accenture.com
>>>
> SDL Enterprise Technologies, Inc. - all rights reserved.  The information
contained in this email may be confidential and/or legally privileged. It h=
as
been sent for the sole use of the intended recipient(s). If you are not the
intended recipient of this mail, you are hereby notified that any
unauthorized review, use, disclosure, dissemination, distribution, or copyi=
ng
of this communication, or any of its contents, is strictly prohibited. If y=
ou
have received this communication in error, please reply to the sender and
destroy all copies of the message.
> Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA=20
> 01880, USA
>
SDL Enterprise Technologies, Inc. - all rights reserved.  The information c=
ontained in this email may be confidential and/or legally privileged. It ha=
s been sent for the sole use of the intended recipient(s). If you are not t=
he intended recipient of this mail, you are hereby notified that any unauth=
orized review, use, disclosure, dissemination, distribution, or copying of =
this communication, or any of its contents, is strictly prohibited. If you =
have received this communication in error, please reply to the sender and d=
estroy all copies of the message.
Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA