hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arko Provo Mukherjee <arkoprovomukher...@gmail.com>
Subject Re: Managing space in Master Node
Date Wed, 13 Feb 2013 23:06:40 GMT
Hi,

I am not sure the provided link matches my situation.

That is because, I am not trying to exclude any node but just trying
to stop the NN act as a DataNode.

Thanks & regards
Arko

On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
<arkoprovomukherjee@gmail.com> wrote:
> Thanks! Let me study and find out if it works for me.
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> <arkoprovomukherjee@gmail.com> wrote:
>> Hi,
>>
>> But the new location will also be the same NN physical server.
>> So how does that help?
>>
>> I would like to clear up the space in the master by removing all the
>> HDFS data to the slave nodes.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <dontariq@gmail.com> wrote:
>>> No, copy this directory to the new location of your choice.
>>>
>>> Warm Regards,
>>> Tariq
>>> https://mtariq.jux.com/
>>> cloudfront.blogspot.com
>>>
>>>
>>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>>> <arkoprovomukherjee@gmail.com> wrote:
>>>>
>>>> Thanks!
>>>> However, in that case, how do I handle the current data that is already in
>>>> NN?
>>>> Do I have to reformat HDFS?
>>>> Thanks & regards
>>>> Arko
>>>>
>>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <dontariq@gmail.com>
>>>> wrote:
>>>> > Just don't run the DN daemon on that machine. Dedicate that machine
to
>>>> > NN.
>>>> > Remove the hostname of the NN machine from the 'slaves' file.
>>>> >
>>>> > Warm Regards,
>>>> > Tariq
>>>> > https://mtariq.jux.com/
>>>> > cloudfront.blogspot.com
>>>> >
>>>> >
>>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>>>> > <arkoprovomukherjee@gmail.com> wrote:
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> Thanks for the help!
>>>> >>
>>>> >> However, I am still unsure about how to "turn off" the datanode
>>>> >> feature on the NN.
>>>> >>
>>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed
to
>>>> >> a directory.
>>>> >>
>>>> >> Should I just comment out that property? What would happen to the
>>>> >> current data that there is in the Master? Will it get removed
>>>> >> automatically?
>>>> >>
>>>> >> Thanks & regards
>>>> >> Arko
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <dontariq@gmail.com>
>>>> >> wrote:
>>>> >> > You can specify the logging level as specified by Charles.
But
>>>> >> > turning
>>>> >> > logs
>>>> >> > off is never a good idea. Logs are really helpful in problem
>>>> >> > diagnosis,
>>>> >> > which are eventual.
>>>> >> >
>>>> >> > Warm Regards,
>>>> >> > Tariq
>>>> >> > https://mtariq.jux.com/
>>>> >> > cloudfront.blogspot.com
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>>>> >> > <arkoprovomukherjee@gmail.com> wrote:
>>>> >> >>
>>>> >> >> Hi,
>>>> >> >>
>>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>>>> >> >>
>>>> >> >> I would like to "turn off" this feature.
>>>> >> >>
>>>> >> >> Request help regarding the same.
>>>> >> >>
>>>> >> >> Thanks & regards
>>>> >> >> Arko
>>>> >> >>
>>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cbaker@sdl.com>
>>>> >> >> wrote:
>>>> >> >> > Hi Arko. Sounds like you may be running a DataNode
on the NameNode
>>>> >> >> > which
>>>> >> >> > is
>>>> >> >> > not recommended practice. Normally, the only files
the NN stores
>>>> >> >> > are
>>>> >> >> > the
>>>> >> >> > image and edits files. It does not store any actual
HDFS data. If
>>>> >> >> > you
>>>> >> >> > must
>>>> >> >> > run a DN on the NN, try turning down the logging in
>>>> >> >> > /conf/log4j.properties:
>>>> >> >> >
>>>> >> >> > #hadoop.root.logger=INFO,console
>>>> >> >> > #hadoop.root.logger=WARN,console
>>>> >> >> > hadoop.root.logger=ERROR,console
>>>> >> >> >
>>>> >> >> > Depending on the logging information you require,
of course.
>>>> >> >> >
>>>> >> >> > -Chuck
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > -----Original Message-----
>>>> >> >> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>>>> >> >> > To: hdfs-user@hadoop.apache.org
>>>> >> >> > Subject: Managing space in Master Node
>>>> >> >> >
>>>> >> >> > Hello Gurus,
>>>> >> >> >
>>>> >> >> > I am managing a Hadoop Cluster to run some experiments.
>>>> >> >> >
>>>> >> >> > The issue I am continuously facing is that the Master
Node runs
>>>> >> >> > out
>>>> >> >> > of
>>>> >> >> > disk
>>>> >> >> > space due to logs and data files.
>>>> >> >> >
>>>> >> >> > I can monitor and delete log files. However, I cannot
delete the
>>>> >> >> > HDFS
>>>> >> >> > data.
>>>> >> >> >
>>>> >> >> > Thus, is there a way to force Hadoop not to save any
HDFS data in
>>>> >> >> > the
>>>> >> >> > Master
>>>> >> >> > Node?
>>>> >> >> >
>>>> >> >> > Then I can use my master to handle the metadata only
and store the
>>>> >> >> > logs.
>>>> >> >> >
>>>> >> >> > Thanks & regards
>>>> >> >> > Arko
>>>> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved.
 The
>>>> >> >> > information contained in this email may be confidential
and/or
>>>> >> >> > legally
>>>> >> >> > privileged. It has been sent for the sole use of the
intended
>>>> >> >> > recipient(s).
>>>> >> >> > If you are not the intended recipient of this mail,
you are hereby
>>>> >> >> > notified
>>>> >> >> > that any unauthorized review, use, disclosure, dissemination,
>>>> >> >> > distribution,
>>>> >> >> > or copying of this communication, or any of its contents,
is
>>>> >> >> > strictly
>>>> >> >> > prohibited. If you have received this communication
in error,
>>>> >> >> > please
>>>> >> >> > reply
>>>> >> >> > to the sender and destroy all copies of the message.
>>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225,
Wakefield, MA
>>>> >> >> > 01880,
>>>> >> >> > USA
>>>> >> >> >
>>>> >> >
>>>> >> >
>>>> >
>>>> >
>>>
>>>

Mime
View raw message