hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Kerzner <markkerz...@gmail.com>
Subject Re: hadoop versions
Date Mon, 16 Nov 2009 02:56:59 GMT
Oops, I just switched to 0.19. There is no .20 on the EC2 that I see. Can I
risk it - I have few MR-specific thing - or go back to 0.18?

On Sun, Nov 15, 2009 at 8:47 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Mark,
>
> You should be good to go.
>
> However, I wouldn't recommend running 0.19 at all. Upgrade to 0.20 - 19 has
> lots of bugs and I know of few people running it in production - 0.19.0
> especially.
>
> -Todd
>
> On Sun, Nov 15, 2009 at 4:30 PM, Mark Kerzner <markkerzner@gmail.com>
> wrote:
>
> > A bit more specifically, I am planning to use 0.19.0 for EC2 cluster,
> > because that's the only .19 version available on EC2, and the version on
> my
> > workstation is 0.19.2, because it was easily available for download.
> Since
> > my complete cluster runs on 0.19.0, and my jar is also there, I should
> have
> > no problem, right?
> >
> > Thank you,
> > Mark
> >
> > On Sun, Nov 15, 2009 at 3:54 PM, Todd Lipcon <todd@cloudera.com> wrote:
> >
> > > Hi Mark,
> > >
> > > The simple answer is "yes, to be safest, they should match".
> > >
> > > In truth, the answer is a bit more complex. Since Java is dynamically
> > > "linked" (classloaded) at runtime, as long as the method signatures and
> > > class names you're using in your code haven't changed between versions,
> > > your
> > > jar compiled against one version will run against another. Between
> > > different
> > > versions of the same major release (eg 0.20 to 0.20.1) this is almost
> > > always
> > > the case except for the occasional backported new API in a later
> version.
> > > So, if you've compiled against 0.20.0 and then run against 0.20.1, or a
> > > Cloudera build like 0.20.1+152 you should be fine.
> > >
> > > If you're using a jar compiled against 0.18.3 and trying to run on
> 0.20,
> > > though, your luck will be much more varied. Most of the APIs from
> 0.18.3
> > > are
> > > still present but there are a few things that will break with strange
> > > errors. So, I *strongly* recommend that you compile against the same
> > major
> > > version you plan on running against.
> > >
> > > All of the above is referring to the case when you're compiling a jar
> > > locally, and then copying it to the cluster and using the cluster's
> > "hadoop
> > > jar foo.jar" command to submit the job. If you're trying to use your
> > local
> > > Hadoop installation pointed at the jobtracker from the remote cluster,
> > the
> > > requirements are a bit more strict - the protocol version numbers must
> > > match, which means you *must* run the same major release (0.18 hadoop
> > > cannot
> > > submit to an 0.20 cluster).
> > >
> > > Thanks
> > > -Todd
> > >
> > > On Sun, Nov 15, 2009 at 1:37 PM, Mark Kerzner <markkerzner@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > when I am building my jar for a MapReduce job, I include the version
> of
> > > > Hadoop I am running on my workstation. When I run Hadoop on a
> cluster,
> > > > there
> > > > is a version that runs on a cluster. Do they have to match? In other
> > > words,
> > > > how does my Hadoop jar interacts with the cluster's Hadoop?
> > > >
> > > > Thank you,
> > > > Mark
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message