hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chandraprakash Bhagtani <cpbhagt...@gmail.com>
Subject Re: Storing contents of a file in a java object
Date Mon, 05 Oct 2009 08:42:56 GMT
I had run a big mapred job (160GB data) on a small cluster of 7 nodes. I had
started 15 Memcached server instances on 7 nodes and I noticed that a single
memcached server was processing 1 million requests per second (in my case),
however it was definitely 3-4 times slower than in-memory approach. I had to
increase the limit of open file descriptors for that.

Since memcached was not performing up to my expectations I used Tokyocabinet
( A file based database) and its performance was near to in-memory approach.

On Fri, Oct 2, 2009 at 12:40 AM, Andrzej Jan Taramina
<andrzej@chaeron.com>wrote:

> Chandraprakash:
>
> > I used Memcached with hadoop, so that i can store (read/write) data on
> > memcached server. I can refer this data
> >     from any task by just connecting the memcached server.
>
> What kind of impact did that have on your network utilization when running
> big mapred jobs?   Did you saturate your
> network links, having all Hadoop tasktracker nodes accessing the same
> memcached box?
>
> Thx!
>
> --
> Andrzej Taramina
> Chaeron Corporation: Enterprise System Solutions
> http://www.chaeron.com
>
>


-- 
Thanks & Regards,
Chandra Prakash Bhagtani,

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message