hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devajyoti Sarkar" <dsar...@q-kk.com>
Subject Re: Sharing an object across mappers
Date Fri, 03 Oct 2008 14:49:46 GMT
Hi Arun,

Briefly going through the DistributedCache information, it seems to be a way
to distribute files to mappers/reducers. One still needs to read the
contents into each map/reduce task VM. Therefore, the data gets replicated
across the VMs in a single node. It seems it does not address my basic
problem which is to have a large shared object across multiple map/reduce
tasks at a given node without having to replicate it across the VMs.

Is there a setting in Hadoop where one can tell Hadoop to create the
individual map/reduce tasks in the same JVM?


On Fri, Oct 3, 2008 at 10:32 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:

> On Oct 3, 2008, at 1:10 AM, Devajyoti Sarkar wrote:
>  Hi Alan,
>> Thanks for your message.
>> The object can be read-only once it is initialized - I do not need to
>> modify
> Please take a look at DistributedCache:
> http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#DistributedCache
> An example:
> http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#Example%3A+WordCount+v2.0
> Arun
>> it. Essentially it is an object that allows me to analyze/modify data that
>> I
>> am mapping/reducing. It comes to about 3-4GB of RAM. The problem I have is
>> that if I run multiple mappers, this object gets replicated in the
>> different
>> VMs and I run out of memory on my node. I pretty much need to have the
>> full
>> object in memory to do my processing. It is possible (though quite
>> difficult) to have it partially on disk and query it (like a lucene store
>> implementation) but there is a significant performance hit. As an e.g.,
>> let
>> us say I use the xlarge CPU instance at Amazon (8CPUs, 8GB RAM). In this
>> scenario, I can really only have 1 mapper per node whereas there are 8
>> CPUs.
>> But if the overhead of sharing the object (e.g. RMI) or persisting the
>> object (e.g. lucene) is greater than 8 times the memory speed, then it is
>> cheaper to run 1 mapper/node. I tried sharing with Terracotta and I was
>> getting a roughly 600 times decrease in performance versus in-memory
>> access.
>> So ideally, if I could have all the mappers in the same VM, then I can
>> create a singleton and still have multiple mappers access it at memory
>> speeds.
>> Please do let me know if I am looking at this correctly and if the above
>> is
>> possible.
>> Thanks a lot for all your help.
>> Cheers,
>> Dev
>> On Fri, Oct 3, 2008 at 12:49 PM, Alan Ho <karlunho@yahoo.ca> wrote:
>>  It really depends on what type of data you are sharing, how you are
>>> looking
>>> up the data, whether the data is Read-write, and whether you care about
>>> consistency. If you don't care about consistency, I suggest that you
>>> shove
>>> the data into a BDB store (for key-value lookup) or a lucene store, and
>>> copy
>>> the data to all the nodes. That way all data access will be in-process,
>>> no
>>> gc problems, and you will get very fast results. BDB and lucene both have
>>> easy replication strategies.
>>> If the data is RW, and you need consistency, you should probably forget
>>> about MapReduce and just run everything on big-iron.
>>> Regards,
>>> Alan Ho
>>> ----- Original Message ----
>>> From: Devajyoti Sarkar <dsarkar@q-kk.com>
>>> To: core-user@hadoop.apache.org
>>> Sent: Thursday, October 2, 2008 8:41:04 PM
>>> Subject: Sharing an object across mappers
>>> I think each mapper/reducer runs in its own JVM which makes it impossible
>>> to
>>> share objects. I need to share a large object so that I can access it at
>>> memory speeds across all the mappers. Is it possible to have all the
>>> mappers
>>> run in the same VM? Or is there a way to do this across VMs at high
>>> speed?
>>> I
>>> guess JMI and others such methods will be just too slow.
>>> Thanks,
>>> Dev
>>>     __________________________________________________________________
>>> Instant Messaging, free SMS, sharing photos and more... Try the new
>>> Yahoo!
>>> Canada Messenger at http://ca.beta.messenger.yahoo.com/

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message