Mailing-List: contact cloudstack-users-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-users@incubator.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Date: Mon, 10 Sep 2012 01:02:03 -0700
From: Shanker Balan <mail@shankerbalan.net>
To: cloudstack-users@incubator.apache.org
Subject: Re: How to integrate Hadoop to CloudStack
Message-ID: <20120910080203.GB23689@shankerbalan.net>
References: 
 <CA+PKkn38QkJaV6_QzQz=pfcVouU=a0ZonMPVfwpDSCui60mXRg@mail.gmail.com>
 <20120907080929.GA25403@shankerbalan.net>
 <CA+PKkn0LESsTN+__DO5KfOfTeu7szE+HeeTaFcnj1sCC_=A_7Q@mail.gmail.com>
 <20120910065308.GA23689@shankerbalan.net>
 <CABDQNHhsiBkfpMQLHCGZkgNyvy4-=t9jpbPURTP_UCNzRqqQ=Q@mail.gmail.com>
 <CA+PKkn22wg6j3xQdu8LTa0CWTMfZZkReX7t6VjUhTnpAjjmOyA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CA+PKkn22wg6j3xQdu8LTa0CWTMfZZkReX7t6VjUhTnpAjjmOyA@mail.gmail.com>
Organisation: http://shankerbalan.net/
User-Agent: Mutt/1.5.20 (2009-06-14)

Hello,

Nguyen Anh Tu wrote,
> Hi Shanker,
> 
> You right. I just want to use Hadoop HDFS for CS, no need any sub project.
> Namenode is exactly a single point of failure. I found a good solution
> instead. That's GlusterFS, which have no master node. All node is peer.
> However, CloudStack and Hadoop are both written by Java. They are belong to
> ASF too. Personally, CloudStack and Hadoop are concerned.

I had to do a bit of planning for the private cloud at my previous work and I
pretty much went down the same line of thought for the storage as you did. NFS
-> HDFS -> GlusterFS. Also considered DRBD.

Unfortunately, CS does not support GlusterFS naively so you cant achieve HA.
In order to use Gluster, you need to export the vols over NFS and you will run
into all the limitations of NFS once more.

I liked Isilon's OneFS solution as a replacement for Gluster though.

Hth.


> 2012/9/10 Hieu Le <hieulq89@gmail.com>
> 
> >
> >
> > ---------- Forwarded message ----------
> > From: Shanker Balan <mail@shankerbalan.net>
> > Date: Mon, Sep 10, 2012 at 1:53 PM
> > Subject: Re: How to integrate Hadoop to CloudStack
> > To: cloudstack-users@incubator.apache.org
> >
> >
> > Hello,
> >
> > Nguyen Anh Tu wrote,
> >
> > > Hello Shanker,
> > >
> > > I mean that with CS, I want to replace NFS to HDFS. You know NFS is not a
> > > suitable solution for storage, because it has not fault-tolerant feature.
> > > So I want to use HDFS for Secondary Storage in CS. I see this
> > > http://www.slideshare.net/kkitase/cloudstack-architecture-future. I
> > think
> > > in near future, Hadoop will be used as a storage solution in CS.
> >
> > You mean "HDFS" as a storage solution for CS? HDFS is just one of the
> > components in the Hadoop project. Hadoop also includes non storage sub
> > projects like MR, Pig, ZK etc.
> >
> > Regarding HDFS, The NameNode machine is a single point of failure for an
> > HDFS
> > cluster at this time. See
> >
> > http://hadoop.apache.org/common/docs/r0.20.2/hdfs_design.html#Metadata+Disk+Failure
> >
> > I have seen the NameNode fail many times at my previous $work place and its
> > not fun. There is a HA NameNode solution in the works but I dont think its
> > reached stable status.
> >
> > http://www.cloudera.com/blog/2012/03/high-availability-for-the-hadoop-distributed-file-system-hdfs/
> >
> > In the end its about the scalability, availability, manageability and cost
> > considerations you wish to achieve that decides every aspect of your cloud
> > solution.
> >
> > Regards.
> >
> >
> > > 2012/9/7 Shanker Balan <mail@shankerbalan.net>
> > >
> > > > (Moving to cloudstack-users@ with Bcc to
> > > > cloudstack-dev@incubator.apache.org)
> > > >
> > > > Hello Nguyen,
> > > >
> > > > Nguyen Anh Tu wrote,
> > > > > Hi guy,
> > > > >
> > > > > Anyone can help me to integrate Hadoop to CloudStack. I read the
> > article
> > > > > "CloudStack and Hadoop: a match made in the cloud" but can not find
> > a way
> > > > > to do this.
> > > >
> > > > Could you explain a bit more on what you mean by "Integrating Hadoop To
> > > > Cloudstack"? I am not using CS yet, but I have a bunch of use cases I
> > have
> > > > been thinking about lately.
> > > >
> > > > You can use Cloudstack to provision Hadoop instances very easily.
> > > > Cloudstack's
> > > > bare metal provisioning capabilities allows you to build high
> > performance
> > > > clusters.
> > > >
> > > >
> > > >
> > http://www.cloudstack.org/blog/63-cloudstack-the-best-kept-secret-in-cloud-computing.html.html
> > > >
> > > > Cloudstack also provides an S3 compatible interface over supported
> > object
> > > > stores like Swift and Caringo. So instead of using HDFS, you can
> > choose to
> > > > store your data on CS backed by object store+s3 bridge.
> > > >
> > > > http://www.slideshare.net/sebastiengoasguen/cloudstack-s3
> > > > http://wiki.apache.org/hadoop/AmazonS3
> > > >
> > > > On the other hand, if you are expecting a hosted Hadoop solution (like
> > AWS
> > > > EMR), I dont think that's quite ready yet (or if its even on the
> > roadmap
> > > > anytime soon).
> > > >
> > > > --
> > > > http://shankerbalan.net/
> > > >
> > > > PS: cloudstack-users@ might be a more appropriate list to discuss this
> > > > further.
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > N.g.U.y.e.N.A.n.H.t.U
> >
> > --
> > http://shankerbalan.net/
> >
> >
> >
> > --
> > ..:: Hieu LE ::..
> >
> > Class: Information System - Course 52
> > School of Information and Communication Technology
> > Hanoi University of Technology
> > No 1, Dai Co Viet street - Hai Ba Trung district - Hanoi
> >
> > High Performance Computing Center
> > Cloud Computing Group
> > Gmail: hieulq89@gmail.com
> >
> 
> 
> 
> -- 
> 
> N.g.U.y.e.N.A.n.H.t.U

-- 
http://shankerbalan.net/