hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Tariq <donta...@gmail.com>
Subject Re: Managing space in Master Node
Date Wed, 13 Feb 2013 23:43:11 GMT
It should.

BTW what's the problem with decommissioning?It's the correct way of
removing the DN, as said by Charles earlier. Create the exclude file and
add the property in the hdfs-site.xml file.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com


On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:

> Hi,
>
> I see that the NameNode IP is present in conf/slaves file.
>
> Will removing the IP of the NN from the conf/slaves file and
> restarting hadoop help?
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
> <arkoprovomukherjee@gmail.com> wrote:
> > Hi,
> >
> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
> > (running DN and Task Tracker).
> >
> > My issue is that the my NN is getting filled up pretty fast as the NN
> > acts as a DN.
> >
> > I want my NN not to function as the DN.
> >
> > Does including the IP for the NN in the "dfs.hosts.exclude" and
> > running "$ hadoop dfsadmin -refreshNodes" solve my problem?
> >
> > Thanks & regards
> > Arko
> >
> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <dontariq@gmail.com>
> wrote:
> >> Stopping DN on one machine means excluding it. What exactly is the
> >> requirement? How many nodes do you have right now?
> >>
> >> Warm Regards,
> >> Tariq
> >> https://mtariq.jux.com/
> >> cloudfront.blogspot.com
> >>
> >>
> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
> >> <arkoprovomukherjee@gmail.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am not sure the provided link matches my situation.
> >>>
> >>> That is because, I am not trying to exclude any node but just trying
> >>> to stop the NN act as a DataNode.
> >>>
> >>> Thanks & regards
> >>> Arko
> >>>
> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
> >>> <arkoprovomukherjee@gmail.com> wrote:
> >>> > Thanks! Let me study and find out if it works for me.
> >>> > Thanks & regards
> >>> > Arko
> >>> >
> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> >>> > <arkoprovomukherjee@gmail.com> wrote:
> >>> >> Hi,
> >>> >>
> >>> >> But the new location will also be the same NN physical server.
> >>> >> So how does that help?
> >>> >>
> >>> >> I would like to clear up the space in the master by removing all
the
> >>> >> HDFS data to the slave nodes.
> >>> >>
> >>> >> Thanks & regards
> >>> >> Arko
> >>> >>
> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <dontariq@gmail.com
> >
> >>> >> wrote:
> >>> >>> No, copy this directory to the new location of your choice.
> >>> >>>
> >>> >>> Warm Regards,
> >>> >>> Tariq
> >>> >>> https://mtariq.jux.com/
> >>> >>> cloudfront.blogspot.com
> >>> >>>
> >>> >>>
> >>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
> >>> >>> <arkoprovomukherjee@gmail.com> wrote:
> >>> >>>>
> >>> >>>> Thanks!
> >>> >>>> However, in that case, how do I handle the current data
that is
> >>> >>>> already in
> >>> >>>> NN?
> >>> >>>> Do I have to reformat HDFS?
> >>> >>>> Thanks & regards
> >>> >>>> Arko
> >>> >>>>
> >>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <
> dontariq@gmail.com>
> >>> >>>> wrote:
> >>> >>>> > Just don't run the DN daemon on that machine. Dedicate
that
> machine
> >>> >>>> > to
> >>> >>>> > NN.
> >>> >>>> > Remove the hostname of the NN machine from the 'slaves'
file.
> >>> >>>> >
> >>> >>>> > Warm Regards,
> >>> >>>> > Tariq
> >>> >>>> > https://mtariq.jux.com/
> >>> >>>> > cloudfront.blogspot.com
> >>> >>>> >
> >>> >>>> >
> >>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> >>> >>>> > <arkoprovomukherjee@gmail.com> wrote:
> >>> >>>> >>
> >>> >>>> >> Hi,
> >>> >>>> >>
> >>> >>>> >> Thanks for the help!
> >>> >>>> >>
> >>> >>>> >> However, I am still unsure about how to "turn
off" the datanode
> >>> >>>> >> feature on the NN.
> >>> >>>> >>
> >>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir
is
> pointed
> >>> >>>> >> to
> >>> >>>> >> a directory.
> >>> >>>> >>
> >>> >>>> >> Should I just comment out that property? What
would happen to
> the
> >>> >>>> >> current data that there is in the Master? Will
it get removed
> >>> >>>> >> automatically?
> >>> >>>> >>
> >>> >>>> >> Thanks & regards
> >>> >>>> >> Arko
> >>> >>>> >>
> >>> >>>> >>
> >>> >>>> >>
> >>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq
> >>> >>>> >> <dontariq@gmail.com>
> >>> >>>> >> wrote:
> >>> >>>> >> > You can specify the logging level as specified
by Charles.
> But
> >>> >>>> >> > turning
> >>> >>>> >> > logs
> >>> >>>> >> > off is never a good idea. Logs are really
helpful in problem
> >>> >>>> >> > diagnosis,
> >>> >>>> >> > which are eventual.
> >>> >>>> >> >
> >>> >>>> >> > Warm Regards,
> >>> >>>> >> > Tariq
> >>> >>>> >> > https://mtariq.jux.com/
> >>> >>>> >> > cloudfront.blogspot.com
> >>> >>>> >> >
> >>> >>>> >> >
> >>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo
Mukherjee
> >>> >>>> >> > <arkoprovomukherjee@gmail.com> wrote:
> >>> >>>> >> >>
> >>> >>>> >> >> Hi,
> >>> >>>> >> >>
> >>> >>>> >> >> Yeah, my NameNode is also seconding as
a DataNode.
> >>> >>>> >> >>
> >>> >>>> >> >> I would like to "turn off" this feature.
> >>> >>>> >> >>
> >>> >>>> >> >> Request help regarding the same.
> >>> >>>> >> >>
> >>> >>>> >> >> Thanks & regards
> >>> >>>> >> >> Arko
> >>> >>>> >> >>
> >>> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles
Baker <
> cbaker@sdl.com>
> >>> >>>> >> >> wrote:
> >>> >>>> >> >> > Hi Arko. Sounds like you may be
running a DataNode on the
> >>> >>>> >> >> > NameNode
> >>> >>>> >> >> > which
> >>> >>>> >> >> > is
> >>> >>>> >> >> > not recommended practice. Normally,
the only files the NN
> >>> >>>> >> >> > stores
> >>> >>>> >> >> > are
> >>> >>>> >> >> > the
> >>> >>>> >> >> > image and edits files. It does not
store any actual HDFS
> >>> >>>> >> >> > data. If
> >>> >>>> >> >> > you
> >>> >>>> >> >> > must
> >>> >>>> >> >> > run a DN on the NN, try turning
down the logging in
> >>> >>>> >> >> > /conf/log4j.properties:
> >>> >>>> >> >> >
> >>> >>>> >> >> > #hadoop.root.logger=INFO,console
> >>> >>>> >> >> > #hadoop.root.logger=WARN,console
> >>> >>>> >> >> > hadoop.root.logger=ERROR,console
> >>> >>>> >> >> >
> >>> >>>> >> >> > Depending on the logging information
you require, of
> course.
> >>> >>>> >> >> >
> >>> >>>> >> >> > -Chuck
> >>> >>>> >> >> >
> >>> >>>> >> >> >
> >>> >>>> >> >> > -----Original Message-----
> >>> >>>> >> >> > From: Arko Provo Mukherjee
> >>> >>>> >> >> > [mailto:arkoprovomukherjee@gmail.com]
> >>> >>>> >> >> > Sent: Wednesday, February 13, 2013
11:32 AM
> >>> >>>> >> >> > To: hdfs-user@hadoop.apache.org
> >>> >>>> >> >> > Subject: Managing space in Master
Node
> >>> >>>> >> >> >
> >>> >>>> >> >> > Hello Gurus,
> >>> >>>> >> >> >
> >>> >>>> >> >> > I am managing a Hadoop Cluster to
run some experiments.
> >>> >>>> >> >> >
> >>> >>>> >> >> > The issue I am continuously facing
is that the Master Node
> >>> >>>> >> >> > runs
> >>> >>>> >> >> > out
> >>> >>>> >> >> > of
> >>> >>>> >> >> > disk
> >>> >>>> >> >> > space due to logs and data files.
> >>> >>>> >> >> >
> >>> >>>> >> >> > I can monitor and delete log files.
However, I cannot
> delete
> >>> >>>> >> >> > the
> >>> >>>> >> >> > HDFS
> >>> >>>> >> >> > data.
> >>> >>>> >> >> >
> >>> >>>> >> >> > Thus, is there a way to force Hadoop
not to save any HDFS
> >>> >>>> >> >> > data in
> >>> >>>> >> >> > the
> >>> >>>> >> >> > Master
> >>> >>>> >> >> > Node?
> >>> >>>> >> >> >
> >>> >>>> >> >> > Then I can use my master to handle
the metadata only and
> >>> >>>> >> >> > store the
> >>> >>>> >> >> > logs.
> >>> >>>> >> >> >
> >>> >>>> >> >> > Thanks & regards
> >>> >>>> >> >> > Arko
> >>> >>>> >> >> > SDL Enterprise Technologies, Inc.
- all rights reserved.
>  The
> >>> >>>> >> >> > information contained in this email
may be confidential
> >>> >>>> >> >> > and/or
> >>> >>>> >> >> > legally
> >>> >>>> >> >> > privileged. It has been sent for
the sole use of the
> intended
> >>> >>>> >> >> > recipient(s).
> >>> >>>> >> >> > If you are not the intended recipient
of this mail, you
> are
> >>> >>>> >> >> > hereby
> >>> >>>> >> >> > notified
> >>> >>>> >> >> > that any unauthorized review, use,
disclosure,
> dissemination,
> >>> >>>> >> >> > distribution,
> >>> >>>> >> >> > or copying of this communication,
or any of its contents,
> is
> >>> >>>> >> >> > strictly
> >>> >>>> >> >> > prohibited. If you have received
this communication in
> error,
> >>> >>>> >> >> > please
> >>> >>>> >> >> > reply
> >>> >>>> >> >> > to the sender and destroy all copies
of the message.
> >>> >>>> >> >> > Registered address: 201 Edgewater
Drive, Suite 225,
> >>> >>>> >> >> > Wakefield, MA
> >>> >>>> >> >> > 01880,
> >>> >>>> >> >> > USA
> >>> >>>> >> >> >
> >>> >>>> >> >
> >>> >>>> >> >
> >>> >>>> >
> >>> >>>> >
> >>> >>>
> >>> >>>
> >>
> >>
>

Mime
View raw message