hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Kerzner <markkerz...@gmail.com>
Subject Re: hadoop versions
Date Sun, 15 Nov 2009 22:25:16 GMT
Thank you, Todd, for this information.

Mark

On Sun, Nov 15, 2009 at 3:54 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Mark,
>
> The simple answer is "yes, to be safest, they should match".
>
> In truth, the answer is a bit more complex. Since Java is dynamically
> "linked" (classloaded) at runtime, as long as the method signatures and
> class names you're using in your code haven't changed between versions,
> your
> jar compiled against one version will run against another. Between
> different
> versions of the same major release (eg 0.20 to 0.20.1) this is almost
> always
> the case except for the occasional backported new API in a later version.
> So, if you've compiled against 0.20.0 and then run against 0.20.1, or a
> Cloudera build like 0.20.1+152 you should be fine.
>
> If you're using a jar compiled against 0.18.3 and trying to run on 0.20,
> though, your luck will be much more varied. Most of the APIs from 0.18.3
> are
> still present but there are a few things that will break with strange
> errors. So, I *strongly* recommend that you compile against the same major
> version you plan on running against.
>
> All of the above is referring to the case when you're compiling a jar
> locally, and then copying it to the cluster and using the cluster's "hadoop
> jar foo.jar" command to submit the job. If you're trying to use your local
> Hadoop installation pointed at the jobtracker from the remote cluster, the
> requirements are a bit more strict - the protocol version numbers must
> match, which means you *must* run the same major release (0.18 hadoop
> cannot
> submit to an 0.20 cluster).
>
> Thanks
> -Todd
>
> On Sun, Nov 15, 2009 at 1:37 PM, Mark Kerzner <markkerzner@gmail.com>
> wrote:
>
> > Hi,
> >
> > when I am building my jar for a MapReduce job, I include the version of
> > Hadoop I am running on my workstation. When I run Hadoop on a cluster,
> > there
> > is a version that runs on a cluster. Do they have to match? In other
> words,
> > how does my Hadoop jar interacts with the cluster's Hadoop?
> >
> > Thank you,
> > Mark
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message