incubator-cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nguyen Anh Tu <ng.t...@gmail.com>
Subject Re: How to integrate Hadoop to CloudStack
Date Mon, 10 Sep 2012 09:01:11 GMT
@Shanker: You right. We use CloudStack -> NFS -> GlusterFS. Bottle neck is
NFS. OneFS is seemly an enterprise solution, we prefer open source.
@Krzysztof Janiszewski: How can I mount MooseFS to CloudStack?
@both: you know that CloudStack supports some ways. As Secondary Storage,
you can only use NFS (maybe Swift if you enabled). As Primary Storage, you
have NFS, PreSetup and iSCSI. So, how can I use MooseFS or GlusterFS or
something else without NFS?

2012/9/10 Krzysztof Janiszewski - ecenter sp. z o.o. <
k.janiszewski@ecenter.pl>

> Try MooseFS and mount it to CloudStack as shared mount point. Works fine
> for
> us.
>
> Best regards
> Krzysztof Janiszewski
>
>
> -----Original Message-----
> From: Shanker Balan [mailto:mail@shankerbalan.net]
> Sent: Monday, September 10, 2012 10:02 AM
> To: cloudstack-users@incubator.apache.org
> Subject: Re: How to integrate Hadoop to CloudStack
>
> Hello,
>
> Nguyen Anh Tu wrote,
> > Hi Shanker,
> >
> > You right. I just want to use Hadoop HDFS for CS, no need any sub
> project.
> > Namenode is exactly a single point of failure. I found a good solution
> > instead. That's GlusterFS, which have no master node. All node is peer.
> > However, CloudStack and Hadoop are both written by Java. They are
> > belong to ASF too. Personally, CloudStack and Hadoop are concerned.
>
> I had to do a bit of planning for the private cloud at my previous work and
> I pretty much went down the same line of thought for the storage as you
> did.
> NFS
> -> HDFS -> GlusterFS. Also considered DRBD.
>
> Unfortunately, CS does not support GlusterFS naively so you cant achieve
> HA.
> In order to use Gluster, you need to export the vols over NFS and you will
> run into all the limitations of NFS once more.
>
> I liked Isilon's OneFS solution as a replacement for Gluster though.
>
> Hth.
>
>
> > 2012/9/10 Hieu Le <hieulq89@gmail.com>
> >
> > >
> > >
> > > ---------- Forwarded message ----------
> > > From: Shanker Balan <mail@shankerbalan.net>
> > > Date: Mon, Sep 10, 2012 at 1:53 PM
> > > Subject: Re: How to integrate Hadoop to CloudStack
> > > To: cloudstack-users@incubator.apache.org
> > >
> > >
> > > Hello,
> > >
> > > Nguyen Anh Tu wrote,
> > >
> > > > Hello Shanker,
> > > >
> > > > I mean that with CS, I want to replace NFS to HDFS. You know NFS
> > > > is not a suitable solution for storage, because it has not
> fault-tolerant feature.
> > > > So I want to use HDFS for Secondary Storage in CS. I see this
> > > > http://www.slideshare.net/kkitase/cloudstack-architecture-future.
> > > > I
> > > think
> > > > in near future, Hadoop will be used as a storage solution in CS.
> > >
> > > You mean "HDFS" as a storage solution for CS? HDFS is just one of
> > > the components in the Hadoop project. Hadoop also includes non
> > > storage sub projects like MR, Pig, ZK etc.
> > >
> > > Regarding HDFS, The NameNode machine is a single point of failure
> > > for an HDFS cluster at this time. See
> > >
> > > http://hadoop.apache.org/common/docs/r0.20.2/hdfs_design.html#Metada
> > > ta+Disk+Failure
> > >
> > > I have seen the NameNode fail many times at my previous $work place
> > > and its not fun. There is a HA NameNode solution in the works but I
> > > dont think its reached stable status.
> > >
> > > http://www.cloudera.com/blog/2012/03/high-availability-for-the-hadoo
> > > p-distributed-file-system-hdfs/
> > >
> > > In the end its about the scalability, availability, manageability
> > > and cost considerations you wish to achieve that decides every
> > > aspect of your cloud solution.
> > >
> > > Regards.
> > >
> > >
> > > > 2012/9/7 Shanker Balan <mail@shankerbalan.net>
> > > >
> > > > > (Moving to cloudstack-users@ with Bcc to
> > > > > cloudstack-dev@incubator.apache.org)
> > > > >
> > > > > Hello Nguyen,
> > > > >
> > > > > Nguyen Anh Tu wrote,
> > > > > > Hi guy,
> > > > > >
> > > > > > Anyone can help me to integrate Hadoop to CloudStack. I read
> > > > > > the
> > > article
> > > > > > "CloudStack and Hadoop: a match made in the cloud" but can not
> > > > > > find
> > > a way
> > > > > > to do this.
> > > > >
> > > > > Could you explain a bit more on what you mean by "Integrating
> > > > > Hadoop To Cloudstack"? I am not using CS yet, but I have a bunch
> > > > > of use cases I
> > > have
> > > > > been thinking about lately.
> > > > >
> > > > > You can use Cloudstack to provision Hadoop instances very easily.
> > > > > Cloudstack's
> > > > > bare metal provisioning capabilities allows you to build high
> > > performance
> > > > > clusters.
> > > > >
> > > > >
> > > > >
> > > http://www.cloudstack.org/blog/63-cloudstack-the-best-kept-secret-in
> > > -cloud-computing.html.html
> > > > >
> > > > > Cloudstack also provides an S3 compatible interface over
> > > > > supported
> > > object
> > > > > stores like Swift and Caringo. So instead of using HDFS, you can
> > > choose to
> > > > > store your data on CS backed by object store+s3 bridge.
> > > > >
> > > > > http://www.slideshare.net/sebastiengoasguen/cloudstack-s3
> > > > > http://wiki.apache.org/hadoop/AmazonS3
> > > > >
> > > > > On the other hand, if you are expecting a hosted Hadoop solution
> > > > > (like
> > > AWS
> > > > > EMR), I dont think that's quite ready yet (or if its even on the
> > > roadmap
> > > > > anytime soon).
> > > > >
> > > > > --
> > > > > http://shankerbalan.net/
> > > > >
> > > > > PS: cloudstack-users@ might be a more appropriate list to
> > > > > discuss this further.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > N.g.U.y.e.N.A.n.H.t.U
> > >
> > > --
> > > http://shankerbalan.net/
> > >
> > >
> > >
> > > --
> > > ..:: Hieu LE ::..
> > >
> > > Class: Information System - Course 52 School of Information and
> > > Communication Technology Hanoi University of Technology No 1, Dai Co
> > > Viet street - Hai Ba Trung district - Hanoi
> > >
> > > High Performance Computing Center
> > > Cloud Computing Group
> > > Gmail: hieulq89@gmail.com
> > >
> >
> >
> >
> > --
> >
> > N.g.U.y.e.N.A.n.H.t.U
>
> --
> http://shankerbalan.net/
>
>


-- 

N.g.U.y.e.N.A.n.H.t.U

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message