hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jens Scheidtmann <jens.scheidtm...@gmail.com>
Subject Re: Is there any way to partially process HDFS edits?
Date Sun, 29 Sep 2013 09:09:12 GMT
Tom,

I would file a jira, if I were you and my Hadoop Version was recent
enough.  Should be pretty easy to reproduce.

Jens

Am Donnerstag, 26. September 2013 schrieb Tom Brown :

> They were created and deleted in quick succession. I thought that meant
> the edits for both the create and delete would be logically next to each
> other in the file allowing it to release the memory almost as soon as it
> had been allocated.
>
> In any case, after finding a VM host that could give me more RAM, I was
> able to get the namenode started. The process used 25GB at it's peak.
>
> Thanks for your help!
>
>
> On Thu, Sep 26, 2013 at 11:07 AM, Harsh J <harsh@cloudera.com> wrote:
>
> Tom,
>
> That is valuable info. When we "replay" edits, we would be creating
> and then deleting those files - so memory would grow in between until
> the delete events begin appearing in the edit log segment.
>
> On Thu, Sep 26, 2013 at 10:07 PM, Tom Brown <tombrown52@gmail.com> wrote:
> > A simple estimate puts the total number of blocks somewhere around
> 500,000.
> > Due to an HBase bug (HBASE-9648), there were approximately 50,000,000
> files
> > that were created and quickly deleted (about 10/sec for 6 weeks) in the
> > cluster, and that activity is what is contained in the edits.
> >
> > Since those files don't exist (quickly created and deleted), shouldn't
> they
> > be inconsequential to the memory requirements of the namenode as it
> starts
> > up.
> >
> > --Tom
> >
> >
> > On Thu, Sep 26, 2013 at 10:25 AM, Nitin Pawar <nitinpawar432@gmail.com>
> > wrote:
> >>
> >> Can you share how many blocks does your cluster have? how many
> >> directories? how many files?
> >>
> >> There is a JIRA https://issues.apache.org/jira/browse/HADOOP-1687 which
> >> explains how much RAM will be used for your namenode.
> >> Its pretty old by hadoop version but its a good starting point.
> >>
> >> According to Cloudera's blog "A good rule of thumb is to assume 1GB of
> >> NameNode memory for every 1 million blocks stored in the distributed
> file
> >> system"
> >>
> >>
> http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/
> >>
> >>
> >>
> >> On Thu, Sep 26, 2013 at 9:26 PM, Tom Brown <tombrown52@gmail.com>
> wrote:
> >>>
> >>> It ran again for about 15 hours before dying again. I'm seeing what
> extra
> >>> RAM resources we can throw at this VM (maybe up to 32GB), but until
> then I'm
> >>> trying to figure out if I'm hitting some strange bug.
> >>>
> >>> When the edits were originally made (over the course of 6 weeks), the
> >>> namenode only had 512MB and was able to contain the filesystem
> completely in
> >>> memory. I don't understand why it's running out of memory. If 512MB was
> >>> enough while the edits were first made, shouldn't it be enough to
> process
> >>> them again?
> >>>
> >>> --Tom
> >>>
> >>>
> >>> On Thu, Sep 26, 2013 at 6:05 AM, Harsh J <harsh@cloudera.com> wrote:
> >>>>
> >>>> Hi Tom,
> >>>>
> >>>> The edits are processed sequentially, and aren't all held in memory.
> >>>> Right now there's no mid-way-checkpoint when it is loaded, such that
> >>>> it could resume only with remaining work if interrupted. Normally this
> >>>> is not a problem in deployments given that SNN or SBN runs for
> >>>> checkpointing the images and keeping the edits collection small
> >>>> periodically.
> >>>>
> >>>> If your NameNode is running out of memory _applying_ the edits, then
> >>>> the cause is not the edits but a growing namespace. You most-likely
> >>>> have more files now than before, and thats going to take up permanent
> >>>> memory from the NameNode heap size.
> >>>>
> >>>> On Thu, Sep 26, 2013 at 3:00 AM, Tom Brown <tombrown52@gmail.com>
> wrote:
> >>>> > Unfortunately, I cannot give it that much RAM. The machine has
4GB
> >>>> > total
> >>>> > (though could be expanded somewhat-- it's a VM).
> >>>> >
> >>>> > Though if each edit is processed sequentially (in a
>
>

Mime
View raw message