cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Min Chen <min.c...@citrix.com>
Subject Re: Query String Request Authentication(QSRA) support by S3 providers
Date Fri, 28 Jun 2013 00:26:57 GMT
Hi Tom,

	Are you saying that you cannot see a Download Template button from UI
when Amazon S3 is added as secondary storage? I only tested with RiakCS
and Cloudian, so didn't see this issue. But I am CC Jessica her to confirm
what special handling is done in UI to enable/disable a button from UI.

	Thanks
	-min

On 6/27/13 5:23 PM, "Thomas O'Dowd" <tpodowd@cloudian.com> wrote:

>Hi Min,
>
>Can you check this bug? I'm trying to test this feature for Amazon but
>having no luck getting the Download template link/button to appear.
>
>https://issues.apache.org/jira/browse/CLOUDSTACK-3220
>
>Thanks,
>
>Tom.
>
>On Fri, 2013-06-21 at 17:21 +0000, Min Chen wrote:
>> John,
>> 
>> 	For S3, the api call createEntityExtractUrl is done on management
>>server
>> side; while for NFS secondary storage, if the implementation of
>> createEntityExtractUrl will involve some code be executed in ssvm to
>>copy
>> template from the install location to a public accessible web server
>> location.
>> 	I don't quite understand some of your comments below. This API is not
>> used to write any information to S3 bucket/directory. This is used for
>> object already existed on S3, and we just provide a URL for user to
>> download a template from S3, just like how Amazon provided user a way to
>> user to extract a S3 object through generatePresignedUrl. We can discuss
>> more on this on collaboration conference.
>> 
>> 	Thanks	
>> 	-min
>> 
>> 
>> 
>> On 6/21/13 7:25 AM, "John Burwell" <jburwell@basho.com> wrote:
>> 
>> >Min,
>> >
>> >(I apologize for my belated reply -- I lost track of this draft in the
>> >chaos of the last couple of days.)
>> >
>> >Upon further review, I think I feel into the confusion between
>>management
>> >server and ssvm.  This code is executing on the management server side,
>> >correct?  Based on my "corrected" understanding is correct, I would
>>like
>> >to amend my thoughts.  Namely, I would like to see the driver
>>operations
>> >pushed out to the SSVM where we can use the stream.  As I think about
>>it,
>> >the management server should not need to interact with the driver.
>> >Simply yard up the DataStore attributes + details map and other extract
>> >parameters, and send them to the SSVM.  Using this information, the S3
>> >driver could open a stream to write the template out to the
>> >bucket/directory.  I recognize it changes the protocol between the
>> >management server and SSVM, but it simply both sides of the operation
>>by
>> >allowing the DataStore information to be treated opaquely until it is
>> >consumed by the driver to execute the write operation.  I also
>>recognize
>> >that we may a little late in the cycle to address it for 4.2, and it
>>may
>> >need to be part of the 4.3 enhancements.
>> >
>> >Thanks,
>> >-John
>> >
>> >On Jun 18, 2013, at 3:55 PM, Min Chen <min.chen@citrix.com> wrote:
>> >
>> >> John,
>> >> 	In that case, how do we keep backward compatibility of
>>extractTemplate
>> >> api, which requires a URL in the response?
>> >> 
>> >> 	Thanks
>> >> 	-min
>> >> 
>> >> On 6/18/13 11:53 AM, "John Burwell" <jburwell@basho.com> wrote:
>> >> 
>> >>> Min,
>> >>> 
>> >>> Looking through the code, I think we can simplify driver operation
>>and
>> >>> increase robustness by changing
>> >>>ImageStoreDriver#createEntityExtractUrl()
>> >>> : String to ImageStoreDriver#readEntity(Š) : InputStream.  My first
>> >>> concern with the current implementation is that it circumvents any
>> >>> connection pooling/resource management underlying client libraries
>> >>> provide.  I/O streams provide a higher-level abstraction that allows
>> >>> drivers to provide the orchestration components with actual
>>resources
>> >>> rather String references.  Second, the current interface seems to
>> >>>appears
>> >>> to assume that an http/https URL will be returned.  With I/O
>>streams,
>> >>>we
>> >>> can support any client library capable of using the standard I/O
>> >>> framework -- enabling us to support other protocols for downloading
>> >>> templates in the future (e.g. RBD, local filesystem, NBD, etc).
>> >>> 
>> >>> Thanks,
>> >>> -John
>> >>> 
>> >>> On Jun 18, 2013, at 1:11 PM, Min Chen <min.chen@citrix.com> wrote:
>> >>> 
>> >>>> A new version of using generatePresignedUrl in
>>S3ImageStoreDriverImpl
>> >>>>is
>> >>>> checked into object_store.
>> >>>> 
>> >>>> THanks
>> >>>> -min
>> >>>> 
>> >>>> On 6/18/13 8:29 AM, "Min Chen" <min.chen@citrix.com> wrote:
>> >>>> 
>> >>>>> Yes, current code is in
>> >>>>>S3ImageStoreDriverImpl.createEntityExtractUrl,
>> >>>>> which has a security issue mentioned in CLOUDSTACK-3030. I am
>>going
>> >>>>>to
>> >>>>> change it to use generatePresignedUrl api from AWS S3 api.
>> >>>>> 
>> >>>>> Thanks
>> >>>>> -min
>> >>>>> 
>> >>>>> From: John Burwell <jburwell@basho.com<mailto:jburwell@basho.com>>
>> >>>>> Date: Tuesday, June 18, 2013 8:07 AM
>> >>>>> To: Min Chen <min.chen@citrix.com<mailto:min.chen@citrix.com>>
>> >>>>> Cc: Thomas O'Dowd
>> >>>>><tpodowd@cloudian.com<mailto:tpodowd@cloudian.com>>,
>> >>>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>"
>> >>>>> <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
>> >>>>> Subject: Re: Query String Request Authentication(QSRA) support
by
>>S3
>> >>>>> providers
>> >>>>> 
>> >>>>> Min,
>> >>>>> 
>> >>>>> Is the code checked into the object_store branch?  If so, which
>>lines
>> >>>>> in
>> >>>>> S3TemplateDownloader?
>> >>>>> 
>> >>>>> Thanks,
>> >>>>> -John
>> >>>>> 
>> >>>>> On Jun 18, 2013, at 12:39 AM, Min Chen
>> >>>>> <min.chen@citrix.com<mailto:min.chen@citrix.com>>
wrote:
>> >>>>> 
>> >>>>> Hi John,
>> >>>>> 
>> >>>>> This is regarding extractTemplate api, where for extractable
>> >>>>>template,
>> >>>>> users can click "Download Template" button from UI to get a
http
>>url
>> >>>>>to
>> >>>>> download the template already stored at S3 without providing
S3
>> >>>>> credentials. In 4.1, we don't have this issue, since the URL
>>returned
>> >>>>> is
>> >>>>> the public web server location hosted in ssvm, and in 4.2, we
are
>> >>>>> returning URL pointing to s3 object. Without setting ACL to
the S3
>> >>>>> object, user cannot directly click the URL returned  from
>> >>>>> extractTemplate
>> >>>>> api to download the template without providing credentials.
By
>> >>>>>reading
>> >>>>> the AWS SDK doc today, I ran across the following API that I
may
>>be
>> >>>>> able
>> >>>>> to use for this purpose:
>> >>>>> 
>> >>>>> 
>> >>>>> 
>> >>>>> 
>> 
>>>>>>>URL<http://java.sun.com/j2se/1.5.0/docs/api/java/net/URL.html?is-ext
>>>>>>>er
>> >>>>>na
>> >>>>> l=
>> >>>>> true>     
>> >>>>> 
>> >>>>> 
>> 
>>>>>>>generatePresignedUrl<http://docs.aws.amazon.com/AWSJavaSDK/latest/ja
>>>>>>>va
>> >>>>>do
>> >>>>> c/
>> >>>>> 
>> >>>>> 
>> 
>>>>>>>com/amazonaws/services/s3/AmazonS3Client.html#generatePresignedUrl%2
>>>>>>>8j
>> >>>>>av
>> >>>>> a.
>> >>>>> 
>> >>>>> 
>> 
>>>>>>>lang.String,%20java.lang.String,%20java.util.Date,%20com.amazonaws.H
>>>>>>>tt
>> >>>>>pM
>> >>>>> et
>> >>>>> 
>> >>>>> 
>> 
>>>>>>>hod%29>(String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Str
>>>>>>>in
>> >>>>>g.
>> >>>>> ht
>> >>>>> ml?is-external=true> bucketName,
>> >>>>> 
>> >>>>> 
>> 
>>>>>>>String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/String.html
>>>>>>>?i
>> >>>>>s-
>> >>>>> ex
>> >>>>> ternal=true> key,
>> >>>>> 
>> >>>>> 
>> 
>>>>>>>Date<http://java.sun.com/j2se/1.5.0/docs/api/java/util/Date.html?is-
>>>>>>>ex
>> >>>>>te
>> >>>>> rn
>> >>>>> al=true> expiration,
>> >>>>> 
>> >>>>> 
>> 
>>>>>>>HttpMethod<http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/
>>>>>>>am
>> >>>>>az
>> >>>>> on
>> >>>>> aws/HttpMethod.html> method)
>> >>>>>         Returns a pre-signed URL for accessing an Amazon S3
>>resource.
>> >>>>> 
>> >>>>> This is along the same line as QSRA mentioned by Tom, by wrapped
>>in
>> >>>>> AmazonS3Client for easy consumption. By using this method, I
think
>> >>>>> that I
>> >>>>> don't need to change ACL of S3 object to open a security hole.
>> >>>>> 
>> >>>>> Thanks
>> >>>>> -min
>> >>>>> 
>> >>>>> From: John Burwell <jburwell@basho.com<mailto:jburwell@basho.com>>
>> >>>>> Date: Monday, June 17, 2013 7:38 PM
>> >>>>> To: Min Chen <min.chen@citrix.com<mailto:min.chen@citrix.com>>
>> >>>>> Cc: Thomas O'Dowd
>> >>>>><tpodowd@cloudian.com<mailto:tpodowd@cloudian.com>>,
>> >>>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>"
>> >>>>> <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
>> >>>>> Subject: Re: Query String Request Authentication(QSRA) support
by
>>S3
>> >>>>> providers
>> >>>>> 
>> >>>>> Min,
>> >>>>> 
>> >>>>> Why are we mucking with ACLs at all?  The best security practice
>> >>>>>would
>> >>>>> be
>> >>>>> to create a bucket for CloudStack's use and assign it a dedicated
>> >>>>> access
>> >>>>> key and secret key pair with read/write access only to that
>>bucket.
>> >>>>> Requiring an administrative account to an object store opens
an
>> >>>>> unnecessarily large attack surface.  Therefore, as implemented
in
>> >>>>>4.1,
>> >>>>> we
>> >>>>> should defer bucket creation, ACL assignment, and credential
>>creation
>> >>>>> to
>> >>>>> the administrator/operator.
>> >>>>> 
>> >>>>> Thanks,
>> >>>>> -John
>> >>>>> 
>> >>>>> On Jun 17, 2013, at 1:15 PM, Min Chen
>> >>>>> <min.chen@citrix.com<mailto:min.chen@citrix.com>>
wrote:
>> >>>>> 
>> >>>>> Tom filed a very good bug for ACL setting change on S3 object
when
>> >>>>> users
>> >>>>> issue extractTemplate API
>> >>>>> (https://issues.apache.org/jira/browse/CLOUDSTACK-3030), and
his
>> >>>>> recommendation of using Query String Request Authentication
(QSRA)
>> >>>>> alternative sounds like a right approach to fix this bug. Before
>> >>>>> implementing it, I would like to confirm if QSRA should be
>>supported
>> >>>>>by
>> >>>>> all S3 providers if they claim that they are AWS s3 compatible.
If
>> >>>>>so,
>> >>>>> we
>> >>>>> will make this assumption in our code. Based on Tom, Cloudian
is
>> >>>>> supporting it. How about RiakCS, John?
>> >>>>> 
>> >>>>> Thanks
>> >>>>> -min
>> >>>>> 
>> >>>>> 
>> >>>> 
>> >>> 
>> >> 
>> >
>> 
>
>-- 
>Cloudian KK - http://www.cloudian.com/get-started.html
>Fancy 100TB of full featured S3 Storage?
>Checkout the Cloudian(R) Community Edition!
>


Mime
View raw message