incubator-cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Krzysztof Janiszewski - ecenter sp. z o.o." <k.janiszew...@ecenter.pl>
Subject RE: How to integrate Hadoop to CloudStack
Date Mon, 10 Sep 2012 08:44:41 GMT
Try MooseFS and mount it to CloudStack as shared mount point. Works fine for
us.

Best regards
Krzysztof Janiszewski


-----Original Message-----
From: Shanker Balan [mailto:mail@shankerbalan.net] 
Sent: Monday, September 10, 2012 10:02 AM
To: cloudstack-users@incubator.apache.org
Subject: Re: How to integrate Hadoop to CloudStack

Hello,

Nguyen Anh Tu wrote,
> Hi Shanker,
> 
> You right. I just want to use Hadoop HDFS for CS, no need any sub project.
> Namenode is exactly a single point of failure. I found a good solution 
> instead. That's GlusterFS, which have no master node. All node is peer.
> However, CloudStack and Hadoop are both written by Java. They are 
> belong to ASF too. Personally, CloudStack and Hadoop are concerned.

I had to do a bit of planning for the private cloud at my previous work and
I pretty much went down the same line of thought for the storage as you did.
NFS
-> HDFS -> GlusterFS. Also considered DRBD.

Unfortunately, CS does not support GlusterFS naively so you cant achieve HA.
In order to use Gluster, you need to export the vols over NFS and you will
run into all the limitations of NFS once more.

I liked Isilon's OneFS solution as a replacement for Gluster though.

Hth.


> 2012/9/10 Hieu Le <hieulq89@gmail.com>
> 
> >
> >
> > ---------- Forwarded message ----------
> > From: Shanker Balan <mail@shankerbalan.net>
> > Date: Mon, Sep 10, 2012 at 1:53 PM
> > Subject: Re: How to integrate Hadoop to CloudStack
> > To: cloudstack-users@incubator.apache.org
> >
> >
> > Hello,
> >
> > Nguyen Anh Tu wrote,
> >
> > > Hello Shanker,
> > >
> > > I mean that with CS, I want to replace NFS to HDFS. You know NFS 
> > > is not a suitable solution for storage, because it has not
fault-tolerant feature.
> > > So I want to use HDFS for Secondary Storage in CS. I see this 
> > > http://www.slideshare.net/kkitase/cloudstack-architecture-future. 
> > > I
> > think
> > > in near future, Hadoop will be used as a storage solution in CS.
> >
> > You mean "HDFS" as a storage solution for CS? HDFS is just one of 
> > the components in the Hadoop project. Hadoop also includes non 
> > storage sub projects like MR, Pig, ZK etc.
> >
> > Regarding HDFS, The NameNode machine is a single point of failure 
> > for an HDFS cluster at this time. See
> >
> > http://hadoop.apache.org/common/docs/r0.20.2/hdfs_design.html#Metada
> > ta+Disk+Failure
> >
> > I have seen the NameNode fail many times at my previous $work place 
> > and its not fun. There is a HA NameNode solution in the works but I 
> > dont think its reached stable status.
> >
> > http://www.cloudera.com/blog/2012/03/high-availability-for-the-hadoo
> > p-distributed-file-system-hdfs/
> >
> > In the end its about the scalability, availability, manageability 
> > and cost considerations you wish to achieve that decides every 
> > aspect of your cloud solution.
> >
> > Regards.
> >
> >
> > > 2012/9/7 Shanker Balan <mail@shankerbalan.net>
> > >
> > > > (Moving to cloudstack-users@ with Bcc to
> > > > cloudstack-dev@incubator.apache.org)
> > > >
> > > > Hello Nguyen,
> > > >
> > > > Nguyen Anh Tu wrote,
> > > > > Hi guy,
> > > > >
> > > > > Anyone can help me to integrate Hadoop to CloudStack. I read 
> > > > > the
> > article
> > > > > "CloudStack and Hadoop: a match made in the cloud" but can not 
> > > > > find
> > a way
> > > > > to do this.
> > > >
> > > > Could you explain a bit more on what you mean by "Integrating 
> > > > Hadoop To Cloudstack"? I am not using CS yet, but I have a bunch 
> > > > of use cases I
> > have
> > > > been thinking about lately.
> > > >
> > > > You can use Cloudstack to provision Hadoop instances very easily.
> > > > Cloudstack's
> > > > bare metal provisioning capabilities allows you to build high
> > performance
> > > > clusters.
> > > >
> > > >
> > > >
> > http://www.cloudstack.org/blog/63-cloudstack-the-best-kept-secret-in
> > -cloud-computing.html.html
> > > >
> > > > Cloudstack also provides an S3 compatible interface over 
> > > > supported
> > object
> > > > stores like Swift and Caringo. So instead of using HDFS, you can
> > choose to
> > > > store your data on CS backed by object store+s3 bridge.
> > > >
> > > > http://www.slideshare.net/sebastiengoasguen/cloudstack-s3
> > > > http://wiki.apache.org/hadoop/AmazonS3
> > > >
> > > > On the other hand, if you are expecting a hosted Hadoop solution 
> > > > (like
> > AWS
> > > > EMR), I dont think that's quite ready yet (or if its even on the
> > roadmap
> > > > anytime soon).
> > > >
> > > > --
> > > > http://shankerbalan.net/
> > > >
> > > > PS: cloudstack-users@ might be a more appropriate list to 
> > > > discuss this further.
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > N.g.U.y.e.N.A.n.H.t.U
> >
> > --
> > http://shankerbalan.net/
> >
> >
> >
> > --
> > ..:: Hieu LE ::..
> >
> > Class: Information System - Course 52 School of Information and 
> > Communication Technology Hanoi University of Technology No 1, Dai Co 
> > Viet street - Hai Ba Trung district - Hanoi
> >
> > High Performance Computing Center
> > Cloud Computing Group
> > Gmail: hieulq89@gmail.com
> >
> 
> 
> 
> --
> 
> N.g.U.y.e.N.A.n.H.t.U

--
http://shankerbalan.net/


Mime
View raw message