Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of tucu00@gmail.com designates
 72.14.220.152 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=cD/Ookkcub+rxgD1aYPqXNG4q4YmkTGDL5WMF0sGMfCOa+WQH2PE99ovCSlo+zbom1n7a9WgJ68HupY/eY65Ry7h42UxzPJaSZMMqzVcEtLz9p3JZh+cYOL5ZG4icReaWKqkqbOTBUsjOlUL0JRmi7Iiuo/gwKrVg+SinA+QiWw=
Message-ID: <b76e3e240806020520p2662192bxb150ca527bc7041@mail.gmail.com>
Date: Mon, 2 Jun 2008 17:50:20 +0530
From: "Alejandro Abdelnur" <tucu00@gmail.com>
To: core-user@hadoop.apache.org
Subject: Re: Realtime Map Reduce = Supercomputing for the Masses?
In-Reply-To: <4843CA1F.1020309@apache.org>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_519_31992698.1212409220377"
References: <C49CD854-FCE3-4A7F-A997-35628C332C67@gmail.com>
	 <c7d45fc70806010108k2e752308k9d1ea2c047fbb77e@mail.gmail.com>
	 <3ea1b3240806010126k42964e45jf6908a0ebd35dc95@mail.gmail.com>
	 <4843CA1F.1020309@apache.org>

------=_Part_519_31992698.1212409220377
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Yes you would have to do it with classloaders (not 'hello world' but not
'rocket science' either).

You'll be limited on using native libraries, even if you use classloaders
properly as native libs can be loaded only once.

You will have to ensure you get rid of the task classloader once the task is
over (thus removing all singleton stuff that may be in it).

You will have to put in place a security manager for the code running out
the task classloader.

You'll end up doing somemthing similar to servlet containers webapp
classloading model with the extra burden of hot-loading for each task run.
Which in the end may have a similar overhead of bootstrapping a JVM for the
task, this should be measured to see what is the time delta to see if it is
worth the effort.

A


On Mon, Jun 2, 2008 at 3:53 PM, Steve Loughran <stevel@apache.org> wrote:

> Christophe Taton wrote:
>
>> Actually Hadoop could be made more friendly to such realtime Map/Reduce
>> jobs.
>> For instance, we could consider running all tasks inside the task tracker
>> jvm as separate threads, which could be implemented as another personality
>> of the TaskRunner.
>> I have been looking into this a couple of weeks ago...
>> Would you be interested in such a feature?
>>
>
> Why does that have benefits? So that you can share stuff via local data
> structures? Because you'd better be sharing classloaders if you are going to
> play that game. And that is very hard to get right (to the extent that I
> dont think any apache project other than Felix does it well)
>

------=_Part_519_31992698.1212409220377--