hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammad Tariq <donta...@gmail.com>
Subject Re: Managing space in Master Node
Date Wed, 13 Feb 2013 23:22:40 GMT
Stopping DN on one machine means excluding it. What exactly is the
requirement? How many nodes do you have right now?

Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com


On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:

> Hi,
>
> I am not sure the provided link matches my situation.
>
> That is because, I am not trying to exclude any node but just trying
> to stop the NN act as a DataNode.
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
> <arkoprovomukherjee@gmail.com> wrote:
> > Thanks! Let me study and find out if it works for me.
> > Thanks & regards
> > Arko
> >
> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> > <arkoprovomukherjee@gmail.com> wrote:
> >> Hi,
> >>
> >> But the new location will also be the same NN physical server.
> >> So how does that help?
> >>
> >> I would like to clear up the space in the master by removing all the
> >> HDFS data to the slave nodes.
> >>
> >> Thanks & regards
> >> Arko
> >>
> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <dontariq@gmail.com>
> wrote:
> >>> No, copy this directory to the new location of your choice.
> >>>
> >>> Warm Regards,
> >>> Tariq
> >>> https://mtariq.jux.com/
> >>> cloudfront.blogspot.com
> >>>
> >>>
> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
> >>> <arkoprovomukherjee@gmail.com> wrote:
> >>>>
> >>>> Thanks!
> >>>> However, in that case, how do I handle the current data that is
> already in
> >>>> NN?
> >>>> Do I have to reformat HDFS?
> >>>> Thanks & regards
> >>>> Arko
> >>>>
> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <dontariq@gmail.com>
> >>>> wrote:
> >>>> > Just don't run the DN daemon on that machine. Dedicate that machine
> to
> >>>> > NN.
> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
> >>>> >
> >>>> > Warm Regards,
> >>>> > Tariq
> >>>> > https://mtariq.jux.com/
> >>>> > cloudfront.blogspot.com
> >>>> >
> >>>> >
> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> >>>> > <arkoprovomukherjee@gmail.com> wrote:
> >>>> >>
> >>>> >> Hi,
> >>>> >>
> >>>> >> Thanks for the help!
> >>>> >>
> >>>> >> However, I am still unsure about how to "turn off" the datanode
> >>>> >> feature on the NN.
> >>>> >>
> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is
pointed
> to
> >>>> >> a directory.
> >>>> >>
> >>>> >> Should I just comment out that property? What would happen
to the
> >>>> >> current data that there is in the Master? Will it get removed
> >>>> >> automatically?
> >>>> >>
> >>>> >> Thanks & regards
> >>>> >> Arko
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <
> dontariq@gmail.com>
> >>>> >> wrote:
> >>>> >> > You can specify the logging level as specified by Charles.
But
> >>>> >> > turning
> >>>> >> > logs
> >>>> >> > off is never a good idea. Logs are really helpful in problem
> >>>> >> > diagnosis,
> >>>> >> > which are eventual.
> >>>> >> >
> >>>> >> > Warm Regards,
> >>>> >> > Tariq
> >>>> >> > https://mtariq.jux.com/
> >>>> >> > cloudfront.blogspot.com
> >>>> >> >
> >>>> >> >
> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> >>>> >> > <arkoprovomukherjee@gmail.com> wrote:
> >>>> >> >>
> >>>> >> >> Hi,
> >>>> >> >>
> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
> >>>> >> >>
> >>>> >> >> I would like to "turn off" this feature.
> >>>> >> >>
> >>>> >> >> Request help regarding the same.
> >>>> >> >>
> >>>> >> >> Thanks & regards
> >>>> >> >> Arko
> >>>> >> >>
> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cbaker@sdl.com>
> >>>> >> >> wrote:
> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode
on the
> NameNode
> >>>> >> >> > which
> >>>> >> >> > is
> >>>> >> >> > not recommended practice. Normally, the only
files the NN
> stores
> >>>> >> >> > are
> >>>> >> >> > the
> >>>> >> >> > image and edits files. It does not store any
actual HDFS
> data. If
> >>>> >> >> > you
> >>>> >> >> > must
> >>>> >> >> > run a DN on the NN, try turning down the logging
in
> >>>> >> >> > /conf/log4j.properties:
> >>>> >> >> >
> >>>> >> >> > #hadoop.root.logger=INFO,console
> >>>> >> >> > #hadoop.root.logger=WARN,console
> >>>> >> >> > hadoop.root.logger=ERROR,console
> >>>> >> >> >
> >>>> >> >> > Depending on the logging information you require,
of course.
> >>>> >> >> >
> >>>> >> >> > -Chuck
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> > -----Original Message-----
> >>>> >> >> > From: Arko Provo Mukherjee [mailto:
> arkoprovomukherjee@gmail.com]
> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
> >>>> >> >> > To: hdfs-user@hadoop.apache.org
> >>>> >> >> > Subject: Managing space in Master Node
> >>>> >> >> >
> >>>> >> >> > Hello Gurus,
> >>>> >> >> >
> >>>> >> >> > I am managing a Hadoop Cluster to run some experiments.
> >>>> >> >> >
> >>>> >> >> > The issue I am continuously facing is that the
Master Node
> runs
> >>>> >> >> > out
> >>>> >> >> > of
> >>>> >> >> > disk
> >>>> >> >> > space due to logs and data files.
> >>>> >> >> >
> >>>> >> >> > I can monitor and delete log files. However,
I cannot delete
> the
> >>>> >> >> > HDFS
> >>>> >> >> > data.
> >>>> >> >> >
> >>>> >> >> > Thus, is there a way to force Hadoop not to save
any HDFS
> data in
> >>>> >> >> > the
> >>>> >> >> > Master
> >>>> >> >> > Node?
> >>>> >> >> >
> >>>> >> >> > Then I can use my master to handle the metadata
only and
> store the
> >>>> >> >> > logs.
> >>>> >> >> >
> >>>> >> >> > Thanks & regards
> >>>> >> >> > Arko
> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights
reserved.  The
> >>>> >> >> > information contained in this email may be confidential
and/or
> >>>> >> >> > legally
> >>>> >> >> > privileged. It has been sent for the sole use
of the intended
> >>>> >> >> > recipient(s).
> >>>> >> >> > If you are not the intended recipient of this
mail, you are
> hereby
> >>>> >> >> > notified
> >>>> >> >> > that any unauthorized review, use, disclosure,
dissemination,
> >>>> >> >> > distribution,
> >>>> >> >> > or copying of this communication, or any of its
contents, is
> >>>> >> >> > strictly
> >>>> >> >> > prohibited. If you have received this communication
in error,
> >>>> >> >> > please
> >>>> >> >> > reply
> >>>> >> >> > to the sender and destroy all copies of the message.
> >>>> >> >> > Registered address: 201 Edgewater Drive, Suite
225,
> Wakefield, MA
> >>>> >> >> > 01880,
> >>>> >> >> > USA
> >>>> >> >> >
> >>>> >> >
> >>>> >> >
> >>>> >
> >>>> >
> >>>
> >>>
>

Mime
View raw message