hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Kerzner <markkerz...@gmail.com>
Subject Re: hadoop versions
Date Mon, 16 Nov 2009 00:30:21 GMT
A bit more specifically, I am planning to use 0.19.0 for EC2 cluster,
because that's the only .19 version available on EC2, and the version on my
workstation is 0.19.2, because it was easily available for download. Since
my complete cluster runs on 0.19.0, and my jar is also there, I should have
no problem, right?

Thank you,
Mark

On Sun, Nov 15, 2009 at 3:54 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Mark,
>
> The simple answer is "yes, to be safest, they should match".
>
> In truth, the answer is a bit more complex. Since Java is dynamically
> "linked" (classloaded) at runtime, as long as the method signatures and
> class names you're using in your code haven't changed between versions,
> your
> jar compiled against one version will run against another. Between
> different
> versions of the same major release (eg 0.20 to 0.20.1) this is almost
> always
> the case except for the occasional backported new API in a later version.
> So, if you've compiled against 0.20.0 and then run against 0.20.1, or a
> Cloudera build like 0.20.1+152 you should be fine.
>
> If you're using a jar compiled against 0.18.3 and trying to run on 0.20,
> though, your luck will be much more varied. Most of the APIs from 0.18.3
> are
> still present but there are a few things that will break with strange
> errors. So, I *strongly* recommend that you compile against the same major
> version you plan on running against.
>
> All of the above is referring to the case when you're compiling a jar
> locally, and then copying it to the cluster and using the cluster's "hadoop
> jar foo.jar" command to submit the job. If you're trying to use your local
> Hadoop installation pointed at the jobtracker from the remote cluster, the
> requirements are a bit more strict - the protocol version numbers must
> match, which means you *must* run the same major release (0.18 hadoop
> cannot
> submit to an 0.20 cluster).
>
> Thanks
> -Todd
>
> On Sun, Nov 15, 2009 at 1:37 PM, Mark Kerzner <markkerzner@gmail.com>
> wrote:
>
> > Hi,
> >
> > when I am building my jar for a MapReduce job, I include the version of
> > Hadoop I am running on my workstation. When I run Hadoop on a cluster,
> > there
> > is a version that runs on a cluster. Do they have to match? In other
> words,
> > how does my Hadoop jar interacts with the cluster's Hadoop?
> >
> > Thank you,
> > Mark
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message