hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiller, Dean (Contractor)" <dean.hil...@broadridge.com>
Subject RE: any plans to deploy OSGi bundles on cluster?
Date Tue, 04 Jan 2011 18:30:35 GMT
I guess I meant in the setting for number of tasks in child JVM before
teardown.  In that case, it is nice to separate/unload my previous
classes from the child JVM which OSGi does.  I was thinking we may do 10
tasks / JVM setting which I thought meant have a "Child" process run 10
tasks before shutting down....4 may be from one job and 4 from a new job
with conflicting classes maybe.  Will that work or is it not advised?

-----Original Message-----
From: Allen Wittenauer [mailto:awittenauer@linkedin.com] 
Sent: Monday, January 03, 2011 9:28 PM
To: <mapreduce-user@hadoop.apache.org>
Subject: Re: any plans to deploy OSGi bundles on cluster?

On Jan 2, 2011, at 9:51 AM, Hiller, Dean (Contractor) wrote:

> I was looking at distributed cache and how I need to copy local jars
> hdfs.  I was wondering if there was any plans to just deploy an OSGi
> bundle(ie. Introspect and auto deploy jars from bundle to the
> distributed cache and then make the api calls to deploy them to the
> slave nodes so there is no work for the developer to do except deploy
> OSGi bundles).

	AFAIK, no.

> Not to mention, the OSGi classloader mechanism is so sweet, that I
> deploy jar A to be used by all my jobs and also deploy jar B version 1
> and jar B version 2 which could be used at the same time by different
> jobs without classloading problems.  

	Given that distributed caches are set per-job, this isn't a
problem with Hadoop either.  Each job's task gets its own JVM.  The only
time that I know of versioning being an issue is when one conflicts with
a bundled Hadoop jar.  [... and that problem is either fixed or will be
committed soon to trunk]
This message and any attachments are intended only for the use of the addressee and
may contain information that is privileged and confidential. If the reader of the 
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.

View raw message