Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:
	content-type:content-transfer-encoding:mime-version;
	b=Pjm62foivRm6PPq8w5c8d9MxmagfHNC121qPCAiX7IgaV+cZi+Iu5Hfo3hxXGqrB
From: Amogh Vasekar <amogh@yahoo-inc.com>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>,
        "core-user@hadoop.apache.org" <core-user@hadoop.apache.org>
Date: Wed, 16 Sep 2009 11:23:14 +0530
Subject: RE: about hadoop jvm allocation in job excution
Thread-Topic: about hadoop jvm allocation in job excution
Thread-Index: Aco2KWRkpyKK+sbLSduwTz6Hoc2pzwAZ9GLQ
Message-ID: 
 <616DA47B2EF5B944B91846785B512FF4CFC727F9EA@EGL-EX07VS01.ds.corp.yahoo.com>
References: <25458201.post@talk.nabble.com>
In-Reply-To: <25458201.post@talk.nabble.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi,
Funny enough was looking at it just yesterday.
http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Task+JVM+=
Reuse


Thanks,
Amogh

-----Original Message-----
From: Zhimin [mailto:wangzm@cs.umb.edu]=20
Sent: Tuesday, September 15, 2009 10:53 PM
To: core-user@hadoop.apache.org
Subject: about hadoop jvm allocation in job excution


We have a project which needs to support similarity queries against items
from a huge amount of data.  One approach we have tried is to use Hbase as
the data repository and Hadoop as the query execution engine. We adopted
Hadoop because Map-Reduce is a very good model of our underlying task and
the programming was straightforward. However, we found that Hadoop will
always allocate a new JVM for each individual task on a node. This is
inefficient for us because in our case the whole Hadoop platform is
dedicated to some relatively stable  parametrized querries, and security an=
d
strict isolation of different tasks is not our main concern. To save the
task setup time, I wonder if there are some existing mechanism in Hadoop or
some extension of Hadoop in other open source projects that can let us
reside our classes in a JVM on the job node, with task nodes waiting for
requests. =20


--=20
View this message in context: http://www.nabble.com/about-hadoop-jvm-alloca=
tion-in-job-excution-tp25458201p25458201.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.