cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nguyen Anh Tu <ng.t...@gmail.com>
Subject Re: How to integrate Hadoop to CloudStack
Date Tue, 11 Sep 2012 07:21:44 GMT
@Krzysztof: I see the ShareMountPoint option, but it seemly does not
support on Xen hypervisor, only KVM. Unfortunately, I'm using Xen.

2012/9/10 Krzysztof Janiszewski - ecenter sp. z o.o. <
k.janiszewski@ecenter.pl>

> You need Advanced Setup. Then you can setup Primary Storage as
> SharedMountPoint and point it to, for example, /export/primary. Then, on
> each Host you mount MooseFS storage to /export/primary and thats all.
>
> Secondary storage cannot be mounted as shared mount point.
>
> Best regards
> Krzysztof Janiszewski
>
>
> -----Original Message-----
> From: Nguyen Anh Tu [mailto:ng.tuna@gmail.com]
> Sent: Monday, September 10, 2012 11:01 AM
> To: cloudstack-users@incubator.apache.org
> Subject: Re: How to integrate Hadoop to CloudStack
>
> @Shanker: You right. We use CloudStack -> NFS -> GlusterFS. Bottle neck is
> NFS. OneFS is seemly an enterprise solution, we prefer open source.
> @Krzysztof Janiszewski: How can I mount MooseFS to CloudStack?
> @both: you know that CloudStack supports some ways. As Secondary Storage,
> you can only use NFS (maybe Swift if you enabled). As Primary Storage, you
> have NFS, PreSetup and iSCSI. So, how can I use MooseFS or GlusterFS or
> something else without NFS?
>
> 2012/9/10 Krzysztof Janiszewski - ecenter sp. z o.o. <
> k.janiszewski@ecenter.pl>
>
> > Try MooseFS and mount it to CloudStack as shared mount point. Works
> > fine for us.
> >
> > Best regards
> > Krzysztof Janiszewski
> >
> >
> > -----Original Message-----
> > From: Shanker Balan [mailto:mail@shankerbalan.net]
> > Sent: Monday, September 10, 2012 10:02 AM
> > To: cloudstack-users@incubator.apache.org
> > Subject: Re: How to integrate Hadoop to CloudStack
> >
> > Hello,
> >
> > Nguyen Anh Tu wrote,
> > > Hi Shanker,
> > >
> > > You right. I just want to use Hadoop HDFS for CS, no need any sub
> > project.
> > > Namenode is exactly a single point of failure. I found a good
> > > solution instead. That's GlusterFS, which have no master node. All node
> is peer.
> > > However, CloudStack and Hadoop are both written by Java. They are
> > > belong to ASF too. Personally, CloudStack and Hadoop are concerned.
> >
> > I had to do a bit of planning for the private cloud at my previous
> > work and I pretty much went down the same line of thought for the
> > storage as you did.
> > NFS
> > -> HDFS -> GlusterFS. Also considered DRBD.
> >
> > Unfortunately, CS does not support GlusterFS naively so you cant
> > achieve HA.
> > In order to use Gluster, you need to export the vols over NFS and you
> > will run into all the limitations of NFS once more.
> >
> > I liked Isilon's OneFS solution as a replacement for Gluster though.
> >
> > Hth.
> >
> >
> > > 2012/9/10 Hieu Le <hieulq89@gmail.com>
> > >
> > > >
> > > >
> > > > ---------- Forwarded message ----------
> > > > From: Shanker Balan <mail@shankerbalan.net>
> > > > Date: Mon, Sep 10, 2012 at 1:53 PM
> > > > Subject: Re: How to integrate Hadoop to CloudStack
> > > > To: cloudstack-users@incubator.apache.org
> > > >
> > > >
> > > > Hello,
> > > >
> > > > Nguyen Anh Tu wrote,
> > > >
> > > > > Hello Shanker,
> > > > >
> > > > > I mean that with CS, I want to replace NFS to HDFS. You know NFS
> > > > > is not a suitable solution for storage, because it has not
> > fault-tolerant feature.
> > > > > So I want to use HDFS for Secondary Storage in CS. I see this
> > > > > http://www.slideshare.net/kkitase/cloudstack-architecture-future.
> > > > > I
> > > > think
> > > > > in near future, Hadoop will be used as a storage solution in CS.
> > > >
> > > > You mean "HDFS" as a storage solution for CS? HDFS is just one of
> > > > the components in the Hadoop project. Hadoop also includes non
> > > > storage sub projects like MR, Pig, ZK etc.
> > > >
> > > > Regarding HDFS, The NameNode machine is a single point of failure
> > > > for an HDFS cluster at this time. See
> > > >
> > > > http://hadoop.apache.org/common/docs/r0.20.2/hdfs_design.html#Meta
> > > > da
> > > > ta+Disk+Failure
> > > >
> > > > I have seen the NameNode fail many times at my previous $work
> > > > place and its not fun. There is a HA NameNode solution in the
> > > > works but I dont think its reached stable status.
> > > >
> > > > http://www.cloudera.com/blog/2012/03/high-availability-for-the-had
> > > > oo
> > > > p-distributed-file-system-hdfs/
> > > >
> > > > In the end its about the scalability, availability, manageability
> > > > and cost considerations you wish to achieve that decides every
> > > > aspect of your cloud solution.
> > > >
> > > > Regards.
> > > >
> > > >
> > > > > 2012/9/7 Shanker Balan <mail@shankerbalan.net>
> > > > >
> > > > > > (Moving to cloudstack-users@ with Bcc to
> > > > > > cloudstack-dev@incubator.apache.org)
> > > > > >
> > > > > > Hello Nguyen,
> > > > > >
> > > > > > Nguyen Anh Tu wrote,
> > > > > > > Hi guy,
> > > > > > >
> > > > > > > Anyone can help me to integrate Hadoop to CloudStack. I
read
> > > > > > > the
> > > > article
> > > > > > > "CloudStack and Hadoop: a match made in the cloud" but
can
> > > > > > > not find
> > > > a way
> > > > > > > to do this.
> > > > > >
> > > > > > Could you explain a bit more on what you mean by "Integrating
> > > > > > Hadoop To Cloudstack"? I am not using CS yet, but I have a
> > > > > > bunch of use cases I
> > > > have
> > > > > > been thinking about lately.
> > > > > >
> > > > > > You can use Cloudstack to provision Hadoop instances very easily.
> > > > > > Cloudstack's
> > > > > > bare metal provisioning capabilities allows you to build high
> > > > performance
> > > > > > clusters.
> > > > > >
> > > > > >
> > > > > >
> > > > http://www.cloudstack.org/blog/63-cloudstack-the-best-kept-secret-
> > > > in
> > > > -cloud-computing.html.html
> > > > > >
> > > > > > Cloudstack also provides an S3 compatible interface over
> > > > > > supported
> > > > object
> > > > > > stores like Swift and Caringo. So instead of using HDFS, you
> > > > > > can
> > > > choose to
> > > > > > store your data on CS backed by object store+s3 bridge.
> > > > > >
> > > > > > http://www.slideshare.net/sebastiengoasguen/cloudstack-s3
> > > > > > http://wiki.apache.org/hadoop/AmazonS3
> > > > > >
> > > > > > On the other hand, if you are expecting a hosted Hadoop
> > > > > > solution (like
> > > > AWS
> > > > > > EMR), I dont think that's quite ready yet (or if its even on
> > > > > > the
> > > > roadmap
> > > > > > anytime soon).
> > > > > >
> > > > > > --
> > > > > > http://shankerbalan.net/
> > > > > >
> > > > > > PS: cloudstack-users@ might be a more appropriate list to
> > > > > > discuss this further.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > N.g.U.y.e.N.A.n.H.t.U
> > > >
> > > > --
> > > > http://shankerbalan.net/
> > > >
> > > >
> > > >
> > > > --
> > > > ..:: Hieu LE ::..
> > > >
> > > > Class: Information System - Course 52 School of Information and
> > > > Communication Technology Hanoi University of Technology No 1, Dai
> > > > Co Viet street - Hai Ba Trung district - Hanoi
> > > >
> > > > High Performance Computing Center
> > > > Cloud Computing Group
> > > > Gmail: hieulq89@gmail.com
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > N.g.U.y.e.N.A.n.H.t.U
> >
> > --
> > http://shankerbalan.net/
> >
> >
>
>
> --
>
> N.g.U.y.e.N.A.n.H.t.U
>
>


-- 

N.g.U.y.e.N.A.n.H.t.U

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message