hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reshu Jain <jai...@us.ibm.com>
Subject Fw: Alternative distributed filesystem.
Date Wed, 18 Nov 2009 23:08:22 GMT

I wanted to propose IBM's Global Parallel File System™ (GPFS™ )  as the
distributed filesystem.  GPFS™  is well known for its unmatched scalability
leadership in file system performance, and it is now IBM’s premier storage
virtualization solution. More information at
GPFS™  is POSIX compliant and designed for general workloads. The
filesystem performance is designed for random access behavior on I/O

We at IBM have been running Hadoop on GPFS™  and have seen comparable
performance number to HDFS.  We are in the process of putting the GPFS
plugin into Hadoop code. In which case you will also be able to run
Map-Reduce jobs natively on GPFS™  along with other applications on the
same cluster."

The following papers talk about the performance comparison between HDFS and
GPFS™  and changes made to GPFS™  to support map reduce data loads.

Rajagopal Ananthanarayanan, Karan Gupta, Prashant Pandey, Himabindu Pucha,
Prasenjit Sarkar, Mansi Shah, Renu Tewari: Cloud analytics: Do we really
need to reinvent the storage stack? in Workshop on Hot Topics in Cloud
Computing (HotCloud '09) June 15, San Diego, CA

Guanying Wang, Ali R. Butt (Virgiana Tech), Prashant Pandey, Karan Gupta
(IBM Almaden): Using Realistic Simulation for Performance Analysis of
MapReduce Setups in Workshop on Large-Scale System and Application
Performance (LSAP) June 10, 2009 Munich

Guanying Wang, Ali R. Butt, Prashant Pandey and Karan Gupta. A Simulation
Approach to Evaluating Design Decisions in MapReduce Setup in IEEE/ACM
International Symposium on Modelling, Analysis and Simulation of Computer
and Telecommunication Systems 2009


----- Forwarded by David Fallside/Santa Teresa/IBM on 11/13/2009 04:12 PM
             <umka@stanford.ed                                          To 
             u>                        <common-user@hadoop.apache.org>     
             11/13/2009 01:56                                              
             PM                                                    Subject 
                                       Alternative distributed filesystem. 
             Please respond to                                             

Dear Hadoop users,

One of our hadoop clusters is being converted to SGE to run very specific
application and we're thinking about how to utilize these huge hard-drives
that are there. Since there will be no hadoop installed on these nodes
looking for alternative distributed filesystem that will have decent
concurrent read/write performance (compared to HDFS) for large amounts of
data. Using single filestorage - like NAS RAID arrays proved to be very
ineffective when someone is pushing gigabytes of data on them.

What other systems can we look at? We would like that FS to be mounted on
every node, open source, hopefully we'd like to have POSIX compliance and
decent random access performance (yet it isn't critical).

HDFS doesn't fit the bill because mounting it via fuse_dfs and using
any mapred jobs (i.e. data will typically be pushed from 1-2 nodes at most
at different times) seems slightly "ass-backward" to say the least.


  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message