hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arko Provo Mukherjee <arkoprovomukher...@gmail.com>
Subject Re: Managing space in Master Node
Date Wed, 13 Feb 2013 23:32:53 GMT
Hi,

I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
(running DN and Task Tracker).

My issue is that the my NN is getting filled up pretty fast as the NN
acts as a DN.

I want my NN not to function as the DN.

Does including the IP for the NN in the "dfs.hosts.exclude" and
running "$ hadoop dfsadmin -refreshNodes" solve my problem?

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <dontariq@gmail.com> wrote:
> Stopping DN on one machine means excluding it. What exactly is the
> requirement? How many nodes do you have right now?
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
> <arkoprovomukherjee@gmail.com> wrote:
>>
>> Hi,
>>
>> I am not sure the provided link matches my situation.
>>
>> That is because, I am not trying to exclude any node but just trying
>> to stop the NN act as a DataNode.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
>> <arkoprovomukherjee@gmail.com> wrote:
>> > Thanks! Let me study and find out if it works for me.
>> > Thanks & regards
>> > Arko
>> >
>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
>> > <arkoprovomukherjee@gmail.com> wrote:
>> >> Hi,
>> >>
>> >> But the new location will also be the same NN physical server.
>> >> So how does that help?
>> >>
>> >> I would like to clear up the space in the master by removing all the
>> >> HDFS data to the slave nodes.
>> >>
>> >> Thanks & regards
>> >> Arko
>> >>
>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <dontariq@gmail.com>
>> >> wrote:
>> >>> No, copy this directory to the new location of your choice.
>> >>>
>> >>> Warm Regards,
>> >>> Tariq
>> >>> https://mtariq.jux.com/
>> >>> cloudfront.blogspot.com
>> >>>
>> >>>
>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>> >>> <arkoprovomukherjee@gmail.com> wrote:
>> >>>>
>> >>>> Thanks!
>> >>>> However, in that case, how do I handle the current data that is
>> >>>> already in
>> >>>> NN?
>> >>>> Do I have to reformat HDFS?
>> >>>> Thanks & regards
>> >>>> Arko
>> >>>>
>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <dontariq@gmail.com>
>> >>>> wrote:
>> >>>> > Just don't run the DN daemon on that machine. Dedicate that
machine
>> >>>> > to
>> >>>> > NN.
>> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
>> >>>> >
>> >>>> > Warm Regards,
>> >>>> > Tariq
>> >>>> > https://mtariq.jux.com/
>> >>>> > cloudfront.blogspot.com
>> >>>> >
>> >>>> >
>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>> >>>> > <arkoprovomukherjee@gmail.com> wrote:
>> >>>> >>
>> >>>> >> Hi,
>> >>>> >>
>> >>>> >> Thanks for the help!
>> >>>> >>
>> >>>> >> However, I am still unsure about how to "turn off" the
datanode
>> >>>> >> feature on the NN.
>> >>>> >>
>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir
is pointed
>> >>>> >> to
>> >>>> >> a directory.
>> >>>> >>
>> >>>> >> Should I just comment out that property? What would happen
to the
>> >>>> >> current data that there is in the Master? Will it get removed
>> >>>> >> automatically?
>> >>>> >>
>> >>>> >> Thanks & regards
>> >>>> >> Arko
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq
>> >>>> >> <dontariq@gmail.com>
>> >>>> >> wrote:
>> >>>> >> > You can specify the logging level as specified by
Charles. But
>> >>>> >> > turning
>> >>>> >> > logs
>> >>>> >> > off is never a good idea. Logs are really helpful
in problem
>> >>>> >> > diagnosis,
>> >>>> >> > which are eventual.
>> >>>> >> >
>> >>>> >> > Warm Regards,
>> >>>> >> > Tariq
>> >>>> >> > https://mtariq.jux.com/
>> >>>> >> > cloudfront.blogspot.com
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>> >>>> >> > <arkoprovomukherjee@gmail.com> wrote:
>> >>>> >> >>
>> >>>> >> >> Hi,
>> >>>> >> >>
>> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>> >>>> >> >>
>> >>>> >> >> I would like to "turn off" this feature.
>> >>>> >> >>
>> >>>> >> >> Request help regarding the same.
>> >>>> >> >>
>> >>>> >> >> Thanks & regards
>> >>>> >> >> Arko
>> >>>> >> >>
>> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker
<cbaker@sdl.com>
>> >>>> >> >> wrote:
>> >>>> >> >> > Hi Arko. Sounds like you may be running a
DataNode on the
>> >>>> >> >> > NameNode
>> >>>> >> >> > which
>> >>>> >> >> > is
>> >>>> >> >> > not recommended practice. Normally, the only
files the NN
>> >>>> >> >> > stores
>> >>>> >> >> > are
>> >>>> >> >> > the
>> >>>> >> >> > image and edits files. It does not store
any actual HDFS
>> >>>> >> >> > data. If
>> >>>> >> >> > you
>> >>>> >> >> > must
>> >>>> >> >> > run a DN on the NN, try turning down the
logging in
>> >>>> >> >> > /conf/log4j.properties:
>> >>>> >> >> >
>> >>>> >> >> > #hadoop.root.logger=INFO,console
>> >>>> >> >> > #hadoop.root.logger=WARN,console
>> >>>> >> >> > hadoop.root.logger=ERROR,console
>> >>>> >> >> >
>> >>>> >> >> > Depending on the logging information you
require, of course.
>> >>>> >> >> >
>> >>>> >> >> > -Chuck
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> > -----Original Message-----
>> >>>> >> >> > From: Arko Provo Mukherjee
>> >>>> >> >> > [mailto:arkoprovomukherjee@gmail.com]
>> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32
AM
>> >>>> >> >> > To: hdfs-user@hadoop.apache.org
>> >>>> >> >> > Subject: Managing space in Master Node
>> >>>> >> >> >
>> >>>> >> >> > Hello Gurus,
>> >>>> >> >> >
>> >>>> >> >> > I am managing a Hadoop Cluster to run some
experiments.
>> >>>> >> >> >
>> >>>> >> >> > The issue I am continuously facing is that
the Master Node
>> >>>> >> >> > runs
>> >>>> >> >> > out
>> >>>> >> >> > of
>> >>>> >> >> > disk
>> >>>> >> >> > space due to logs and data files.
>> >>>> >> >> >
>> >>>> >> >> > I can monitor and delete log files. However,
I cannot delete
>> >>>> >> >> > the
>> >>>> >> >> > HDFS
>> >>>> >> >> > data.
>> >>>> >> >> >
>> >>>> >> >> > Thus, is there a way to force Hadoop not
to save any HDFS
>> >>>> >> >> > data in
>> >>>> >> >> > the
>> >>>> >> >> > Master
>> >>>> >> >> > Node?
>> >>>> >> >> >
>> >>>> >> >> > Then I can use my master to handle the metadata
only and
>> >>>> >> >> > store the
>> >>>> >> >> > logs.
>> >>>> >> >> >
>> >>>> >> >> > Thanks & regards
>> >>>> >> >> > Arko
>> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights
reserved.  The
>> >>>> >> >> > information contained in this email may be
confidential
>> >>>> >> >> > and/or
>> >>>> >> >> > legally
>> >>>> >> >> > privileged. It has been sent for the sole
use of the intended
>> >>>> >> >> > recipient(s).
>> >>>> >> >> > If you are not the intended recipient of
this mail, you are
>> >>>> >> >> > hereby
>> >>>> >> >> > notified
>> >>>> >> >> > that any unauthorized review, use, disclosure,
dissemination,
>> >>>> >> >> > distribution,
>> >>>> >> >> > or copying of this communication, or any
of its contents, is
>> >>>> >> >> > strictly
>> >>>> >> >> > prohibited. If you have received this communication
in error,
>> >>>> >> >> > please
>> >>>> >> >> > reply
>> >>>> >> >> > to the sender and destroy all copies of the
message.
>> >>>> >> >> > Registered address: 201 Edgewater Drive,
Suite 225,
>> >>>> >> >> > Wakefield, MA
>> >>>> >> >> > 01880,
>> >>>> >> >> > USA
>> >>>> >> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >
>> >>>> >
>> >>>
>> >>>
>
>

Mime
View raw message