incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Chamard <da...@cloud.ca>
Subject Re: HDFS and CloudStack? was: [SCM] CloudStack OSS branch s3-hdfs
Date Thu, 24 May 2012 03:39:27 GMT
Alex,

Very interesting since we are considering improving the openstack swift  s3
middleware.  We are also interested and will be testing heavily the
cloudstack integration with swift for backend templates/snapshots

https://github.com/cloudops/cs_auth
There is the link for the auth system that Will stevens wrote that allows
swift to authenticate through the cloudstack api.
We can do a webex or teamviewer to show the UI if you are interested.

Cheers,

David

On Wed, May 23, 2012 at 4:21 PM, Alex Huang <Alex.Huang@citrix.com> wrote:

> For this, it's more about abstracting object storage behind a S3 API
> proxy, for which Swift is definitely a candidate.  CloudStack 3.0 already
> has integration with Swift but this is a cleaner integration.
>
> I definitely would like to see what you've done.  I wanted to see it in
> the last OpenStack conference but was unable to make the meeting.  Any
> links where we can take peak?
>
> --Alex
>
> > -----Original Message-----
> > From: David Chamard [mailto:david@cloud.ca]
> > Sent: Wednesday, May 23, 2012 1:15 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Re: HDFS and CloudStack? was: [SCM] CloudStack OSS branch s3-
> > hdfs
> >
> > Hi Guys,
> >
> > Have you guys considered swift? Rackspace have a 100%  availability with
> > swift in the last 2 years.. Having the meta data included in the object
> make it
> > very scalable + having a database for every container . The key with
> swift is a
> > good infrastructure architecture to get the desired performance.. We
> > actually built a tab in the Cloudstack UI for swift and an auth  system.
> >
> > Would you guys be interested with screenshots of the UIs. Our latest
> version
> > is becoming stable and very clean.
> >
> > Cheers,
> >
> > David
> >
> >
> > On Tue, May 22, 2012 at 4:41 PM, Chiradeep Vittal <
> > Chiradeep.Vittal@citrix.com> wrote:
> >
> > >
> > > On 5/22/12 12:58 PM, "David Nalley" <david@gnsa.us> wrote:
> > >
> > > >Hi Chiradeep -
> > > >
> > > >I just saw this branch created - which has a fascinating name - sadly
> > > >I can't find any docs that talk about Hadoop or HDFS on the wiki
> > > >(there is a parenthetical statement on the wiki that says 'like
> > > >Hadoop' but nothing much past that) Can you tell us about what your
> > > >plan is here, objectives? Functional spec exist for any of this work?
> > > >
> > > >--David
> > >
> > > This is a follow-up to
> > > http://www.cloudstack.org/blog/121-cloudstack-and-hadoop-a-match-
> > made-
> > > in-th
> > > e-cloud.html
> > > "Because of its scalability, reliability, performance, and maturity,
> > > HDFS is a great object store solution for IaaS cloud. We have started
> > > the development of an S3 API front-end for HDFS"
> > >
> > > The S3 API in the 3.0.x branch supports a simple filesystem backing
> > > store to store objects and MySQL to store metadata such as ACLs and
> > policies.
> > > The code is actually written to be plugged in with different backend
> > > storage systems.
> > > Really, most of it is implementing the S3BucketAdapter interface.
> > > However, there's quite a bit of assumptions about using a filesystem.
> > >
> > > What I checked into the s3-hdfs branch is a proof-of-concept to plugin
> > > HDFS as the backing store for objects.
> > > It still uses MySQL as the metadata store.
> > > The code isn't very clean yet, and does require a couple of magical
> > > incantations to configure on the client and the HDFS side.
> > > I wanted to push something since I will be out-of-office till next
> week.
> > > I'll follow up with more detailed documentation then.
> > >
> > > I also have a bunch of thoughts on where this could go including
> > >  - scalable metadata storage (no MySQL)
> > >  - using a CloudStack-managed cohort of service vms as S3 API servers
> > >  - testing performance and scalability
> > >  - replacing (or deprecating) the S3 API frontend with a CloudStack
> > > object store API
> > >  - interaction with the Hadoop community for enhancements to HDFS
> > > (multi-datacenter storage for example)
> > >  - support for other Object Storage APIs (e.g., Azure, Google Storage,
> > > all of which are similar)
> > >
> > > Again, I'll write these incoherent thoughts more clearly when I get
> > > back in town.
> > >
> > > --
> > > Chiradeep
> > >
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message