hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stuart Sierra" <m...@stuartsierra.com>
Subject Re: Serving contents of large MapFiles/SequenceFiles from memory across many machines
Date Fri, 19 Sep 2008 16:40:14 GMT
On Thu, Sep 18, 2008 at 1:05 AM, Chris Dyer <redpony@umd.edu> wrote:
> Basically, I'd like to be able to
> load the entire contents of a file key-value map file in DFS into
> memory across many machines in my cluster so that I can access any of
> it with ultra-low latencies.

I think the simplest way, which I've used, is to put your key-value
file into DistributedCache, then load it into a HashMap or ArrayList
in the configure method of each Map/Reduce task.

-Stuart

Mime
View raw message