Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates
 209.85.223.179 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <50FDC9C0.40703@salesforce.com>
References: <50FDC9C0.40703@salesforce.com>
From: Harsh J <harsh@cloudera.com>
Date: Tue, 22 Jan 2013 23:08:14 +0530
Message-ID: 
 <CAOcnVr2rmCi-ff-fBR=WgK6y69M2HbCthB2AGvPD7n4RuWNPvw@mail.gmail.com>
Subject: Re: CDH412/Hadoop 2.0.3 Upgrade instructions
To: dbebortha@salesforce.com, "cdh-user@cloudera.org" <cdh-user@cloudera.org>
Content-Type: multipart/alternative; boundary=14dae9340c0108c1ab04d3e40c6c

--14dae9340c0108c1ab04d3e40c6c
Content-Type: text/plain; charset=ISO-8859-1

Moving to cdh-user@cloudera.org as your question is CDH related.

My answers inline:

On Tue, Jan 22, 2013 at 4:35 AM, Dheeren bebortha
<dbebortha@salesforce.com>wrote:

> I am trying to upgrade a Hadoop Cluster with 0.20.X and MRv1 to a hadoop
> Cluster with CDH412 with HA+QJM+YARN (aka Hadoop 2.0.3) without any  data
> loss and minimal down time. The documentation on cloudera site iis OK, but
> very confusing. BTW I do not plan on using Cloudera manager. Has anyone
> attempted a clean upgrade using hadoop native commands?
>

The upgrade process for any 0.20/1.x/CDH3 release to CDH4 is documented at
https://ccp.cloudera.com/display/CDH4DOC/Upgrading+from+CDH3+to+CDH4. The
only difference you may see is in use of packaging (tarballs or RPMs/DEBs?)
and therefore, of usernames used in the guide.

The basic process is to stop the older HDFS, remove older installation and
all its traces carefully, and start the newer HDFS with the -upgrade flag.
This takes care of HDFS metadata upgrades. Once done and you've verified
that files/etc. are all perfectly readable and state's good, you can
dfsadmin -finalizeUpgrade your cluster to commit the upgrade permanently.
QJM is documented in a separate guide, found on the same portal mentioned
above and can be upgraded in a second step after upgrade, to achieve full
HA.

For MR side, all your MR1 jobs will need to be recompiled before they may
be run on the newer YARN+MR2 cluster due to some binary incompatible
changes made between the versions you're upgrading. Other than a recompile,
you may mostly not require to do anything else.

May we also know your reason to not use CM when its aimed to make all this
much easier to do and manage? We appreciate any form of feedback, thanks!

-- 
Harsh J

--14dae9340c0108c1ab04d3e40c6c
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Moving to <a href=3D"mailto:cdh-user@cloudera.org">cdh-use=
r@cloudera.org</a> as your question is CDH related.<div><br></div><div styl=
e>My answers inline:</div><div class=3D"gmail_extra"><br><div class=3D"gmai=
l_quote">

On Tue, Jan 22, 2013 at 4:35 AM, Dheeren bebortha <span dir=3D"ltr">&lt;<a =
href=3D"mailto:dbebortha@salesforce.com" target=3D"_blank">dbebortha@salesf=
orce.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=
=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(20=
4,204,204);border-left-style:solid;padding-left:1ex">

I am trying to upgrade a Hadoop Cluster with 0.20.X and MRv1 to a hadoop Cl=
uster with CDH412 with HA+QJM+YARN (aka Hadoop 2.0.3) without any =A0data l=
oss and minimal down time. The documentation on cloudera site iis OK, but v=
ery confusing. BTW I do not plan on using Cloudera manager. Has anyone atte=
mpted a clean upgrade using hadoop native commands?<br>

</blockquote><div><br></div><div style>The upgrade process for any 0.20/1.x=
/CDH3 release to CDH4 is documented at=A0<a href=3D"https://ccp.cloudera.co=
m/display/CDH4DOC/Upgrading+from+CDH3+to+CDH4">https://ccp.cloudera.com/dis=
play/CDH4DOC/Upgrading+from+CDH3+to+CDH4</a>. The only difference you may s=
ee is in use of packaging (tarballs or RPMs/DEBs?) and therefore, of userna=
mes used in the guide.</div>

<div style><br></div><div style>The basic process is to stop the older HDFS=
, remove older installation and all its traces carefully, and start the new=
er HDFS with the -upgrade flag. This takes care of HDFS metadata upgrades. =
Once done and you&#39;ve verified that files/etc. are all perfectly readabl=
e and state&#39;s good, you can dfsadmin -finalizeUpgrade your cluster to c=
ommit the upgrade permanently. QJM is documented in a separate guide, found=
 on the same portal mentioned above and can be upgraded in a second step af=
ter upgrade, to achieve full HA.</div>

<div style><br></div><div style>For MR side, all your MR1 jobs will need to=
 be recompiled before they may be run on the newer YARN+MR2 cluster due to =
some binary incompatible changes made between the versions you&#39;re upgra=
ding. Other than a recompile, you may mostly not require to do anything els=
e.</div>

<div style><br></div><div style>May we also know your reason to not use CM =
when its aimed to make all this much easier to do and manage? We appreciate=
 any form of feedback, thanks!</div></div><div><br></div>-- <br>Harsh J
</div></div>

--14dae9340c0108c1ab04d3e40c6c--