Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: Sanjay Sharma <sanjay.sharma@impetus.co.in>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Wed, 17 Mar 2010 11:33:05 +0530
Subject: RE: Storing Custom Java Objects in Hadoop Distibuted Cache
Thread-Topic: Storing Custom Java Objects in Hadoop Distibuted Cache
Thread-Index: AcrFjqbGvcxFV279SvCV2Fmw6soMsgABjDPQ
Message-ID: <8419CE074608E24FB834DB9D270C78D6A4727C8E3E@mail1.impetus.co.in>
References: <4d371b591003162201s2f94fcber7ba83d295a802592@mail.gmail.com>
In-Reply-To: <4d371b591003162201s2f94fcber7ba83d295a802592@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Ninad,
You can always use Java object serialization to store custom objects as fil=
es in Hadoop distributed cache before map/reducer start running.
The thumb rule steps of such usage is-
a. Create the object while configuring your job, serialize it to a file and=
 put it is distributed cache
b. Initialize the same object in Map or Reducer init/setup

The known issues with this approach is -
a. additional hit of file based object ser-de(serialization-de-serializatio=
n). So do check whether the cumulative object creation time is more than th=
e  time in object ser-de
b. works only in case of write-once read-only object

In case the object size is a problem, suggest going for memcache or BDB or =
tokyotyrant or similar fast store for such distributed caching

Regards,
Sanjay Sharma

-----Original Message-----
From: Ninad Raut [mailto:hbase.user.ninad@gmail.com]
Sent: Wednesday, March 17, 2010 10:32 AM
To: common-user@hadoop.apache.org
Subject: Storing Custom Java Objects in Hadoop Distibuted Cache

Hi All,

Currently hadoop cache only allows files to be stored in the distributed
cache.

Can I store custom objects in hadoop cache so that all my map reduce
programs use those objects?


Pleas let me know if any one has a solution to this problem. The Object
creation is a very heavy task and I want to do it only once for all Map -
Reduce Routines.


Regards,
Ninad .....

Impetus Technologies is participating at the CTIA Wireless 2010 from 23rd t=
o 25th March 2010. Meet Impetus in Las Vegas to experience our mobile and w=
ireless domain expertise. Click http://impetus.com/events to know more.

Follow our updates on www.twitter.com/impetuscalling.

NOTE: This message may contain information that is confidential, proprietar=
y, privileged or otherwise protected by law. The message is intended solely=
 for the named addressee. If received in error, please destroy and notify t=
he sender. Any use of this email is prohibited when received in error. Impe=
tus does not represent, warrant and/or guarantee, that the integrity of thi=
s communication has been maintained nor that the communication is free of e=
rrors, virus, interception or interference.