Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=ymail.com;
  h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=Zmw5ef2Oi4w8Wv1Q443RbxW4zlBI3T98Hs8ryrRfSEXPeh7GMn4zXBCC4TzOx8dPn5PQ545f2dWXpMiYcLLbAV21mZN2uvnszQXSEc93GopWCBw6OLOVQ5ciFe0uVnIs7yXkwqMD83o5Ok1Czv043HUgKs88VoVhYbVFqHSaBJc=;
References: 
 <CAHtRq2fs8YdiUKV4vhqgjy4++DWvdwUiXiTYzmc62aJvVoCswA@mail.gmail.com>
Message-ID: <1380138830.85455.YahooMailNeo@web141202.mail.bf1.yahoo.com>
Date: Wed, 25 Sep 2013 12:53:50 -0700 (PDT)
From: Ravi Prakash <ravihoo@ymail.com>
Reply-To: Ravi Prakash <ravihoo@ymail.com>
Subject: Re: Is there any way to partially process HDFS edits?
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
In-Reply-To: 
 <CAHtRq2fs8YdiUKV4vhqgjy4++DWvdwUiXiTYzmc62aJvVoCswA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="624387052-725184109-1380138830=:85455"

--624387052-725184109-1380138830=:85455
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Tom! I would guess that just giving the NN JVM lots of memory (64Gb / 96Gb)=
 should be the easiest way.=0A=0A=0A=0A=0A________________________________=
=0A From: Tom Brown <tombrown52@gmail.com>=0ATo: "user@hadoop.apache.org" <=
user@hadoop.apache.org> =0ASent: Wednesday, September 25, 2013 11:29 AM=0AS=
ubject: Is there any way to partially process HDFS edits?=0A =0A=0A=0AI hav=
e an edits file on my namenode that is 35GB. This is quite a bit larger tha=
n it should be (the secondary namenode wasn't running for some time, and=A0=
HBASE-9648 caused a huge number of additional edits).=0A=0AThe first time I=
 tried to start the namenode, it chewed on the edits for about 4 hours and =
then ran out of memory. I have increased the memory available to the nameno=
de (was 512MB, now 2GB), and started the process again.=0A=0AIs there any w=
ay that the edits file can be partially processed to avoid having to re-pro=
cess the same edits over and over until I can allocate enough memory for it=
 to be done in one shot?=0A=0A=0AHow long should it take (hours? days?) to =
process an edits file of that size?=0A=0AAny help is appreciated!=0A=0A--To=
m
--624387052-725184109-1380138830=:85455
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

<html><body><div style=3D"color:#000; background-color:#fff; font-family:ti=
mes new roman, new york, times, serif;font-size:10pt">Tom! I would guess th=
at just giving the NN JVM lots of memory (64Gb / 96Gb) should be the easies=
t way.<br><div><span><br></span></div><div><br></div>  <div style=3D"font-f=
amily: times new roman, new york, times, serif; font-size: 10pt;"> <div sty=
le=3D"font-family: times new roman, new york, times, serif; font-size: 12pt=
;"> <div dir=3D"ltr"> <hr size=3D"1">  <font face=3D"Arial" size=3D"2"> <b>=
<span style=3D"font-weight:bold;">From:</span></b> Tom Brown &lt;tombrown52=
@gmail.com&gt;<br> <b><span style=3D"font-weight: bold;">To:</span></b> "us=
er@hadoop.apache.org" &lt;user@hadoop.apache.org&gt; <br> <b><span style=3D=
"font-weight: bold;">Sent:</span></b> Wednesday, September 25, 2013 11:29 A=
M<br> <b><span style=3D"font-weight: bold;">Subject:</span></b> Is there an=
y way to partially process HDFS edits?<br> </font> </div> <div
 class=3D"y_msg_container"><br><div id=3D"yiv9638761225"><div dir=3D"ltr">I=
 have an edits file on my namenode that is 35GB. This is quite a bit larger=
 than it should be (the secondary namenode wasn't running for some time, an=
d&nbsp;HBASE-9648 caused a huge number of additional edits).<div>=0A<br></d=
iv><div>The first time I tried to start the namenode, it chewed on the edit=
s for about 4 hours and then ran out of memory. I have increased the memory=
 available to the namenode (was 512MB, now 2GB), and started the process ag=
ain.</div>=0A<div><br></div><div>Is there any way that the edits file can b=
e partially processed to avoid having to re-process the same edits over and=
 over until I can allocate enough memory for it to be done in one shot?<br>=
</div><div>=0A<br></div><div>How long should it take (hours? days?) to proc=
ess an edits file of that size?</div><div><br></div><div>Any help is apprec=
iated!</div><div><br></div><div>--Tom</div></div></div><br><br></div> </div=
> </div>  </div></body></html>
--624387052-725184109-1380138830=:85455--