hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjay Sharma <sanjay.sha...@impetus.co.in>
Subject RE: Storing Custom Java Objects in Hadoop Distibuted Cache
Date Wed, 17 Mar 2010 06:03:05 GMT
Hi Ninad,
You can always use Java object serialization to store custom objects as files in Hadoop distributed
cache before map/reducer start running.
The thumb rule steps of such usage is-
a. Create the object while configuring your job, serialize it to a file and put it is distributed
b. Initialize the same object in Map or Reducer init/setup

The known issues with this approach is -
a. additional hit of file based object ser-de(serialization-de-serialization). So do check
whether the cumulative object creation time is more than the  time in object ser-de
b. works only in case of write-once read-only object

In case the object size is a problem, suggest going for memcache or BDB or tokyotyrant or
similar fast store for such distributed caching

Sanjay Sharma

-----Original Message-----
From: Ninad Raut [mailto:hbase.user.ninad@gmail.com]
Sent: Wednesday, March 17, 2010 10:32 AM
To: common-user@hadoop.apache.org
Subject: Storing Custom Java Objects in Hadoop Distibuted Cache

Hi All,

Currently hadoop cache only allows files to be stored in the distributed

Can I store custom objects in hadoop cache so that all my map reduce
programs use those objects?

Pleas let me know if any one has a solution to this problem. The Object
creation is a very heavy task and I want to do it only once for all Map -
Reduce Routines.

Ninad .....

Impetus Technologies is participating at the CTIA Wireless 2010 from 23rd to 25th March 2010.
Meet Impetus in Las Vegas to experience our mobile and wireless domain expertise. Click http://impetus.com/events
to know more.

Follow our updates on www.twitter.com/impetuscalling.

NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.

View raw message