hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@veoh.com>
Subject Re: Using HDFS as native storage
Date Thu, 27 Mar 2008 17:38:32 GMT

We evaluated several options for just this problem and eventually settled on
MogileFS.  That said, Mogile needed several weeks of work to get it ready
for prime time.  It will work pretty well for modest sized collections, but
for our stuff (many hundreds of millions of files, approaching PB of
storage), it just wasn't ready.  The fixes had to do with sharding the name
database across many mySQL instances and improving the handling of storage
system up-state.

On 3/27/08 2:13 AM, "Robert Kr├╝ger" <krueger@signal7.de> wrote:

> Hi,
> we're looking for options for creating a scalable storage solution based
> on commodity hardware for media files (spacewise dominated video files
> of a few hundred MB but also to store up to a few million smaller files
> such as thumbnails). The system will start with a few TB and should be
> able to scale to about a PB.
> Is anyone using HDFS for native storage for critical files or is it just
> common to use HDFS for large amounts of temporary more or less
> non-critical data? What would be the trade-offs to decide whether to use
> HDFS or something like GlusterFS? Note that we'r ecurrently not planning
> on using MapReduce.
> Thanks in advance,
> Robert

View raw message