hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@veoh.com>
Subject Re: Hadoop HDFS vs. NFS latency
Date Sat, 22 Mar 2008 23:05:07 GMT

We at Veoh face a similar problem of serving files from a distributed store,
probably on somewhat larger scale than your are looking at.

We evaluated several systems and determined that for our needs, hadoop was
not a good choice.  The advantages of hadoop (map-reduce, blocking) were
substantially out-weighed by issues with scale (we have nearly a billion
files that we need to work with) and hackability (we need to have system
admins be able to modify placement policies).  We also were unable to find
anybody else using hadoop in this way.

Commercial distributed stores were eliminated based on cost alone.

In the end, we chose to use mogile for origin file service and have been
pretty happy with the choice, although we did wind up making significant
changes to the system in order to facilitate scaling.  One major advantage
for the choice of mogile was the fact that we knew that it was already being
used in some high volume sites.

On 3/22/08 10:32 AM, "Robert Kr├╝ger" <krueger@signal7.de> wrote:

> Hi,
> we're currently evaluating the use of Hadoop's HDFS for a project where
> most of the data will be large files and latency will not matter that
> much, so it should be suited perfectly in those cases, but some of the
> data will be smaller files (a few K) and will be accessed quite
> frequently (e.g. by an HTTP server serving user requests, however not
> with a very high load). Now, while the docs state that thruput rather
> than latency was optimized, I am curious as to what order of magnitude
> of an overhead is to be expected for, e.g. reading a file's metadata
> such as modification timestamp or streaming (reading) a small file as a
> whole as compare to things like NFS access or access to theses resources
> via another HTTP server using the same network hardware and setup?
> Paying a 10% penalty in latency compared to NFS would probably be OK but
> at the moment I have no idea if we are talking abou 1, 10 or 100%
> difference.
> Has anyone done similar things, i.e. serve not so large files directly
> from a HDFS cluster via an HTTP server?
> Thanks in advance,
> Robert

View raw message