hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Tariq <donta...@gmail.com>
Subject Re: Managing space in Master Node
Date Thu, 14 Feb 2013 00:02:29 GMT
Yes.

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com


On Thu, Feb 14, 2013 at 5:24 AM, Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:

> Sure, thanks!
> One last question before I go ahead and make the changes.
> My hdfs-site.xml doesn't have an entry for dfs.replication.
> So can I just assume that the system is using the default of 3?
> I ask this because if it is 1, then I will first have to increase it
> so as to not loose data when I stop the NN being a DN.
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 5:51 PM, Mohammad Tariq <dontariq@gmail.com>
> wrote:
> > Decommissioning is for removing DNs and not the NN.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Thu, Feb 14, 2013 at 5:18 AM, Arko Provo Mukherjee
> > <arkoprovomukherjee@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> I was just wondering if it is okay to exclude the NN IP.
> >>
> >> I hope it won't create any "side effects".
> >>
> >> This is because I don't want to actually decommission the NN node, but
> >> just make it stop working as a DataNode.
> >>
> >> However, if you are sure there are no problems, I will go ahead and
> >> make the changes.
> >>
> >> Thanks & regards
> >> Arko
> >>
> >> On Wed, Feb 13, 2013 at 5:43 PM, Mohammad Tariq <dontariq@gmail.com>
> >> wrote:
> >> > It should.
> >> >
> >> > BTW what's the problem with decommissioning?It's the correct way of
> >> > removing
> >> > the DN, as said by Charles earlier. Create the exclude file and add
> the
> >> > property in the hdfs-site.xml file.
> >> >
> >> > Warm Regards,
> >> > Tariq
> >> > https://mtariq.jux.com/
> >> > cloudfront.blogspot.com
> >> >
> >> >
> >> > On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee
> >> > <arkoprovomukherjee@gmail.com> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> I see that the NameNode IP is present in conf/slaves file.
> >> >>
> >> >> Will removing the IP of the NN from the conf/slaves file and
> >> >> restarting hadoop help?
> >> >>
> >> >> Thanks & regards
> >> >> Arko
> >> >>
> >> >> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
> >> >> <arkoprovomukherjee@gmail.com> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave
> >> >> > Nodes
> >> >> > (running DN and Task Tracker).
> >> >> >
> >> >> > My issue is that the my NN is getting filled up pretty fast as
the
> NN
> >> >> > acts as a DN.
> >> >> >
> >> >> > I want my NN not to function as the DN.
> >> >> >
> >> >> > Does including the IP for the NN in the "dfs.hosts.exclude" and
> >> >> > running "$ hadoop dfsadmin -refreshNodes" solve my problem?
> >> >> >
> >> >> > Thanks & regards
> >> >> > Arko
> >> >> >
> >> >> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <
> dontariq@gmail.com>
> >> >> > wrote:
> >> >> >> Stopping DN on one machine means excluding it. What exactly
is the
> >> >> >> requirement? How many nodes do you have right now?
> >> >> >>
> >> >> >> Warm Regards,
> >> >> >> Tariq
> >> >> >> https://mtariq.jux.com/
> >> >> >> cloudfront.blogspot.com
> >> >> >>
> >> >> >>
> >> >> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
> >> >> >> <arkoprovomukherjee@gmail.com> wrote:
> >> >> >>>
> >> >> >>> Hi,
> >> >> >>>
> >> >> >>> I am not sure the provided link matches my situation.
> >> >> >>>
> >> >> >>> That is because, I am not trying to exclude any node but
just
> >> >> >>> trying
> >> >> >>> to stop the NN act as a DataNode.
> >> >> >>>
> >> >> >>> Thanks & regards
> >> >> >>> Arko
> >> >> >>>
> >> >> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
> >> >> >>> <arkoprovomukherjee@gmail.com> wrote:
> >> >> >>> > Thanks! Let me study and find out if it works for
me.
> >> >> >>> > Thanks & regards
> >> >> >>> > Arko
> >> >> >>> >
> >> >> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> >> >> >>> > <arkoprovomukherjee@gmail.com> wrote:
> >> >> >>> >> Hi,
> >> >> >>> >>
> >> >> >>> >> But the new location will also be the same NN
physical server.
> >> >> >>> >> So how does that help?
> >> >> >>> >>
> >> >> >>> >> I would like to clear up the space in the master
by removing
> all
> >> >> >>> >> the
> >> >> >>> >> HDFS data to the slave nodes.
> >> >> >>> >>
> >> >> >>> >> Thanks & regards
> >> >> >>> >> Arko
> >> >> >>> >>
> >> >> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq
> >> >> >>> >> <dontariq@gmail.com>
> >> >> >>> >> wrote:
> >> >> >>> >>> No, copy this directory to the new location
of your choice.
> >> >> >>> >>>
> >> >> >>> >>> Warm Regards,
> >> >> >>> >>> Tariq
> >> >> >>> >>> https://mtariq.jux.com/
> >> >> >>> >>> cloudfront.blogspot.com
> >> >> >>> >>>
> >> >> >>> >>>
> >> >> >>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo
Mukherjee
> >> >> >>> >>> <arkoprovomukherjee@gmail.com> wrote:
> >> >> >>> >>>>
> >> >> >>> >>>> Thanks!
> >> >> >>> >>>> However, in that case, how do I handle
the current data that
> >> >> >>> >>>> is
> >> >> >>> >>>> already in
> >> >> >>> >>>> NN?
> >> >> >>> >>>> Do I have to reformat HDFS?
> >> >> >>> >>>> Thanks & regards
> >> >> >>> >>>> Arko
> >> >> >>> >>>>
> >> >> >>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad
Tariq
> >> >> >>> >>>> <dontariq@gmail.com>
> >> >> >>> >>>> wrote:
> >> >> >>> >>>> > Just don't run the DN daemon on
that machine. Dedicate
> that
> >> >> >>> >>>> > machine
> >> >> >>> >>>> > to
> >> >> >>> >>>> > NN.
> >> >> >>> >>>> > Remove the hostname of the NN machine
from the 'slaves'
> >> >> >>> >>>> > file.
> >> >> >>> >>>> >
> >> >> >>> >>>> > Warm Regards,
> >> >> >>> >>>> > Tariq
> >> >> >>> >>>> > https://mtariq.jux.com/
> >> >> >>> >>>> > cloudfront.blogspot.com
> >> >> >>> >>>> >
> >> >> >>> >>>> >
> >> >> >>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM,
Arko Provo Mukherjee
> >> >> >>> >>>> > <arkoprovomukherjee@gmail.com>
wrote:
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> Hi,
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> Thanks for the help!
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> However, I am still unsure about
how to "turn off" the
> >> >> >>> >>>> >> datanode
> >> >> >>> >>>> >> feature on the NN.
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> I checked the "hdfs-site.xml"
file and the dfs.data.dir
> is
> >> >> >>> >>>> >> pointed
> >> >> >>> >>>> >> to
> >> >> >>> >>>> >> a directory.
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> Should I just comment out that
property? What would
> happen
> >> >> >>> >>>> >> to
> >> >> >>> >>>> >> the
> >> >> >>> >>>> >> current data that there is in
the Master? Will it get
> >> >> >>> >>>> >> removed
> >> >> >>> >>>> >> automatically?
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> Thanks & regards
> >> >> >>> >>>> >> Arko
> >> >> >>> >>>> >>
> >> >> >>> >>>> >>
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> On Wed, Feb 13, 2013 at 1:55
PM, Mohammad Tariq
> >> >> >>> >>>> >> <dontariq@gmail.com>
> >> >> >>> >>>> >> wrote:
> >> >> >>> >>>> >> > You can specify the logging
level as specified by
> >> >> >>> >>>> >> > Charles.
> >> >> >>> >>>> >> > But
> >> >> >>> >>>> >> > turning
> >> >> >>> >>>> >> > logs
> >> >> >>> >>>> >> > off is never a good idea.
Logs are really helpful in
> >> >> >>> >>>> >> > problem
> >> >> >>> >>>> >> > diagnosis,
> >> >> >>> >>>> >> > which are eventual.
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >> > Warm Regards,
> >> >> >>> >>>> >> > Tariq
> >> >> >>> >>>> >> > https://mtariq.jux.com/
> >> >> >>> >>>> >> > cloudfront.blogspot.com
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >> > On Thu, Feb 14, 2013 at
1:22 AM, Arko Provo Mukherjee
> >> >> >>> >>>> >> > <arkoprovomukherjee@gmail.com>
wrote:
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> Hi,
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> Yeah, my NameNode is
also seconding as a DataNode.
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> I would like to "turn
off" this feature.
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> Request help regarding
the same.
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> Thanks & regards
> >> >> >>> >>>> >> >> Arko
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> On Wed, Feb 13, 2013
at 1:38 PM, Charles Baker
> >> >> >>> >>>> >> >> <cbaker@sdl.com>
> >> >> >>> >>>> >> >> wrote:
> >> >> >>> >>>> >> >> > Hi Arko. Sounds
like you may be running a DataNode
> on
> >> >> >>> >>>> >> >> > the
> >> >> >>> >>>> >> >> > NameNode
> >> >> >>> >>>> >> >> > which
> >> >> >>> >>>> >> >> > is
> >> >> >>> >>>> >> >> > not recommended
practice. Normally, the only files
> the
> >> >> >>> >>>> >> >> > NN
> >> >> >>> >>>> >> >> > stores
> >> >> >>> >>>> >> >> > are
> >> >> >>> >>>> >> >> > the
> >> >> >>> >>>> >> >> > image and edits
files. It does not store any actual
> >> >> >>> >>>> >> >> > HDFS
> >> >> >>> >>>> >> >> > data. If
> >> >> >>> >>>> >> >> > you
> >> >> >>> >>>> >> >> > must
> >> >> >>> >>>> >> >> > run a DN on the
NN, try turning down the logging in
> >> >> >>> >>>> >> >> > /conf/log4j.properties:
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > #hadoop.root.logger=INFO,console
> >> >> >>> >>>> >> >> > #hadoop.root.logger=WARN,console
> >> >> >>> >>>> >> >> > hadoop.root.logger=ERROR,console
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > Depending on the
logging information you require, of
> >> >> >>> >>>> >> >> > course.
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > -Chuck
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > -----Original
Message-----
> >> >> >>> >>>> >> >> > From: Arko Provo
Mukherjee
> >> >> >>> >>>> >> >> > [mailto:arkoprovomukherjee@gmail.com]
> >> >> >>> >>>> >> >> > Sent: Wednesday,
February 13, 2013 11:32 AM
> >> >> >>> >>>> >> >> > To: hdfs-user@hadoop.apache.org
> >> >> >>> >>>> >> >> > Subject: Managing
space in Master Node
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > Hello Gurus,
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > I am managing
a Hadoop Cluster to run some
> >> >> >>> >>>> >> >> > experiments.
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > The issue I am
continuously facing is that the
> Master
> >> >> >>> >>>> >> >> > Node
> >> >> >>> >>>> >> >> > runs
> >> >> >>> >>>> >> >> > out
> >> >> >>> >>>> >> >> > of
> >> >> >>> >>>> >> >> > disk
> >> >> >>> >>>> >> >> > space due to logs
and data files.
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > I can monitor
and delete log files. However, I
> cannot
> >> >> >>> >>>> >> >> > delete
> >> >> >>> >>>> >> >> > the
> >> >> >>> >>>> >> >> > HDFS
> >> >> >>> >>>> >> >> > data.
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > Thus, is there
a way to force Hadoop not to save any
> >> >> >>> >>>> >> >> > HDFS
> >> >> >>> >>>> >> >> > data in
> >> >> >>> >>>> >> >> > the
> >> >> >>> >>>> >> >> > Master
> >> >> >>> >>>> >> >> > Node?
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > Then I can use
my master to handle the metadata only
> >> >> >>> >>>> >> >> > and
> >> >> >>> >>>> >> >> > store the
> >> >> >>> >>>> >> >> > logs.
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > Thanks & regards
> >> >> >>> >>>> >> >> > Arko
> >> >> >>> >>>> >> >> > SDL Enterprise
Technologies, Inc. - all rights
> >> >> >>> >>>> >> >> > reserved.
> >> >> >>> >>>> >> >> > The
> >> >> >>> >>>> >> >> > information contained
in this email may be
> >> >> >>> >>>> >> >> > confidential
> >> >> >>> >>>> >> >> > and/or
> >> >> >>> >>>> >> >> > legally
> >> >> >>> >>>> >> >> > privileged. It
has been sent for the sole use of the
> >> >> >>> >>>> >> >> > intended
> >> >> >>> >>>> >> >> > recipient(s).
> >> >> >>> >>>> >> >> > If you are not
the intended recipient of this mail,
> >> >> >>> >>>> >> >> > you
> >> >> >>> >>>> >> >> > are
> >> >> >>> >>>> >> >> > hereby
> >> >> >>> >>>> >> >> > notified
> >> >> >>> >>>> >> >> > that any unauthorized
review, use, disclosure,
> >> >> >>> >>>> >> >> > dissemination,
> >> >> >>> >>>> >> >> > distribution,
> >> >> >>> >>>> >> >> > or copying of
this communication, or any of its
> >> >> >>> >>>> >> >> > contents,
> >> >> >>> >>>> >> >> > is
> >> >> >>> >>>> >> >> > strictly
> >> >> >>> >>>> >> >> > prohibited. If
you have received this communication
> in
> >> >> >>> >>>> >> >> > error,
> >> >> >>> >>>> >> >> > please
> >> >> >>> >>>> >> >> > reply
> >> >> >>> >>>> >> >> > to the sender
and destroy all copies of the message.
> >> >> >>> >>>> >> >> > Registered address:
201 Edgewater Drive, Suite 225,
> >> >> >>> >>>> >> >> > Wakefield, MA
> >> >> >>> >>>> >> >> > 01880,
> >> >> >>> >>>> >> >> > USA
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >
> >> >> >>> >>>> >
> >> >> >>> >>>
> >> >> >>> >>>
> >> >> >>
> >> >> >>
> >> >
> >> >
> >
> >
>

Mime
View raw message