hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norbert Burger <norbert.bur...@gmail.com>
Subject Re: Using HDFS to serve www requests
Date Thu, 26 Mar 2009 20:18:25 GMT
Have you looked into MogileFS already?  Seems like a good fit, based
on your description.  This question has come up more than once here,
and MogileFS is an oft-recommended solution.

Norbert

On 3/26/09, phil cryer <phil@cryer.us> wrote:
> > When you say that you have huge images, how big is "huge?"
>
>
> Yes, we're looking at some images that are 100Megs in size, but
>  nothing like what you're speaking of.  This helps me understand
>  Hadoop's usage better and unfortunately it won't be the fit I was
>  hoping for.
>
>
>  > You can use the API or the FUSE module to mount hadoop but that is not
>  > a direct goal of hadoop. Hope that helps.
>
>
> Very interesting, and yes, that indeed does help, not to veer off
>  thread too much, but does Sun's Lustre follow in the steps of Gluster
>  then?  I know Lustre requires kernel patches to install, so it's at a
>  different level than the others, but I have seen some articles about
>  large scale clusters built with Lustre and want to look at that as
>  another option.
>
>  Again, thanks for the info, if anyone has general information on
>  cluster software, or know of a more appropriate list, I'd appreciate
>  the advice.
>
>  Thanks
>
>
>  P
>
>
>  On Thu, Mar 26, 2009 at 12:32 PM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
>  > It is a little more natural to connect to HDFS from apache tomcat.
>  > This will allow you to skip the FUSE mounts and just use the HDFS-API.
>  >
>  > I have modified this code to run inside tomcat.
>  > http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample
>  >
>  > I will not testify to how well this setup will perform under internet
>  > traffic, but it does work.
>  >
>  > GlusterFS is more like a traditional POSIX filesystem. It supports
>  > locking and appends and you can do things like put the mysql data
>  > directory on it.
>  >
>  > GLUSTERFS is geared for storing data to be accessed with low latency.
>  > Nodes (Bricks) are normally connected via GIG-E or infiniban. The
>  > GlusterFS volume is mounted directly on a unix system.
>  >
>  > Hadoop is a user space file system. The latency is higher. Nodes are
>  > connected by GIG-E. It is closely coupled with MAP/REDUCE.
>  >
>  > You can use the API or the FUSE module to mount hadoop but that is not
>  > a direct goal of hadoop. Hope that helps.
>  >
>

Mime
View raw message