hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arko Provo Mukherjee <arkoprovomukher...@gmail.com>
Subject Re: Managing space in Master Node
Date Wed, 13 Feb 2013 20:31:18 GMT
Thanks! Let me study and find out if it works for me.
Thanks & regards
Arko

On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
<arkoprovomukherjee@gmail.com> wrote:
> Hi,
>
> But the new location will also be the same NN physical server.
> So how does that help?
>
> I would like to clear up the space in the master by removing all the
> HDFS data to the slave nodes.
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <dontariq@gmail.com> wrote:
>> No, copy this directory to the new location of your choice.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>> <arkoprovomukherjee@gmail.com> wrote:
>>>
>>> Thanks!
>>> However, in that case, how do I handle the current data that is already in
>>> NN?
>>> Do I have to reformat HDFS?
>>> Thanks & regards
>>> Arko
>>>
>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <dontariq@gmail.com>
>>> wrote:
>>> > Just don't run the DN daemon on that machine. Dedicate that machine to
>>> > NN.
>>> > Remove the hostname of the NN machine from the 'slaves' file.
>>> >
>>> > Warm Regards,
>>> > Tariq
>>> > https://mtariq.jux.com/
>>> > cloudfront.blogspot.com
>>> >
>>> >
>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>>> > <arkoprovomukherjee@gmail.com> wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> Thanks for the help!
>>> >>
>>> >> However, I am still unsure about how to "turn off" the datanode
>>> >> feature on the NN.
>>> >>
>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>>> >> a directory.
>>> >>
>>> >> Should I just comment out that property? What would happen to the
>>> >> current data that there is in the Master? Will it get removed
>>> >> automatically?
>>> >>
>>> >> Thanks & regards
>>> >> Arko
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <dontariq@gmail.com>
>>> >> wrote:
>>> >> > You can specify the logging level as specified by Charles. But
>>> >> > turning
>>> >> > logs
>>> >> > off is never a good idea. Logs are really helpful in problem
>>> >> > diagnosis,
>>> >> > which are eventual.
>>> >> >
>>> >> > Warm Regards,
>>> >> > Tariq
>>> >> > https://mtariq.jux.com/
>>> >> > cloudfront.blogspot.com
>>> >> >
>>> >> >
>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>>> >> > <arkoprovomukherjee@gmail.com> wrote:
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>>> >> >>
>>> >> >> I would like to "turn off" this feature.
>>> >> >>
>>> >> >> Request help regarding the same.
>>> >> >>
>>> >> >> Thanks & regards
>>> >> >> Arko
>>> >> >>
>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cbaker@sdl.com>
>>> >> >> wrote:
>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on
the NameNode
>>> >> >> > which
>>> >> >> > is
>>> >> >> > not recommended practice. Normally, the only files the
NN stores
>>> >> >> > are
>>> >> >> > the
>>> >> >> > image and edits files. It does not store any actual HDFS
data. If
>>> >> >> > you
>>> >> >> > must
>>> >> >> > run a DN on the NN, try turning down the logging in
>>> >> >> > /conf/log4j.properties:
>>> >> >> >
>>> >> >> > #hadoop.root.logger=INFO,console
>>> >> >> > #hadoop.root.logger=WARN,console
>>> >> >> > hadoop.root.logger=ERROR,console
>>> >> >> >
>>> >> >> > Depending on the logging information you require, of course.
>>> >> >> >
>>> >> >> > -Chuck
>>> >> >> >
>>> >> >> >
>>> >> >> > -----Original Message-----
>>> >> >> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>>> >> >> > To: hdfs-user@hadoop.apache.org
>>> >> >> > Subject: Managing space in Master Node
>>> >> >> >
>>> >> >> > Hello Gurus,
>>> >> >> >
>>> >> >> > I am managing a Hadoop Cluster to run some experiments.
>>> >> >> >
>>> >> >> > The issue I am continuously facing is that the Master
Node runs
>>> >> >> > out
>>> >> >> > of
>>> >> >> > disk
>>> >> >> > space due to logs and data files.
>>> >> >> >
>>> >> >> > I can monitor and delete log files. However, I cannot
delete the
>>> >> >> > HDFS
>>> >> >> > data.
>>> >> >> >
>>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS
data in
>>> >> >> > the
>>> >> >> > Master
>>> >> >> > Node?
>>> >> >> >
>>> >> >> > Then I can use my master to handle the metadata only and
store the
>>> >> >> > logs.
>>> >> >> >
>>> >> >> > Thanks & regards
>>> >> >> > Arko
>>> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved.
 The
>>> >> >> > information contained in this email may be confidential
and/or
>>> >> >> > legally
>>> >> >> > privileged. It has been sent for the sole use of the intended
>>> >> >> > recipient(s).
>>> >> >> > If you are not the intended recipient of this mail, you
are hereby
>>> >> >> > notified
>>> >> >> > that any unauthorized review, use, disclosure, dissemination,
>>> >> >> > distribution,
>>> >> >> > or copying of this communication, or any of its contents,
is
>>> >> >> > strictly
>>> >> >> > prohibited. If you have received this communication in
error,
>>> >> >> > please
>>> >> >> > reply
>>> >> >> > to the sender and destroy all copies of the message.
>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield,
MA
>>> >> >> > 01880,
>>> >> >> > USA
>>> >> >> >
>>> >> >
>>> >> >
>>> >
>>> >
>>
>>

Mime
View raw message