cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sebgoa <run...@gmail.com>
Subject Re: S3 use with simulator
Date Mon, 23 Jun 2014 13:45:02 GMT

On Jun 23, 2014, at 3:33 PM, Prasanna Santhanam <tsp@apache.org> wrote:

> Be advised that advanced.cfg might be adding the NFS store as
> secondary storage as opposed to cacheStorage that the S3 image store
> needs it to be. At least that's how it was the last time I ran it.

Thanks. 

That's exactly what I am trying to figure out. Can we migrate from a NFS secondary storage
to an object store.

If I move the NFS to be a cache store then I can add a new ImageStore. But the updatecloudstoobjectstore
which is supposed to do everything at once, does not work for me.

> 
> On Mon, Jun 23, 2014 at 02:45:55PM +0200, sebgoa wrote:
>> 
>> On Jun 23, 2014, at 2:30 PM, Koushik Das <koushik.das@citrix.com> wrote:
>> 
>>> This has nothing to do with the zoneId. If you search the source code for the
error message "Failed to add data store: DataCenter id is null, and simulator image store
has to be associated with a data center", it points toSimulatorImageStoreLifeCycleImpl.initialize()
method. This indicates that somehow the image store provider for Simulator is getting picked
up. For S3 provider type, the S3ImageStoreLifeCycleImpl should get invoked and in that the
zoneId is not used so there won't be issues.
>> 
>> Happy to discuss it on IRC, if you could join.
>> 
>> in SimulatorImageStoreLifeCycleImpl.initialize() on 4.4-forward I see:
>> 
>> Long dcId = (Long)dsInfos.get("zoneId");
>>        String url = (String)dsInfos.get("url");
>>        String name = (String)dsInfos.get("name");
>>        if (name == null) {
>>            name = url;
>>        }
>>     ...
>> 
>>        if (dcId == null) {
>>            throw new InvalidParameterValueException("DataCenter id is null, and simulator
image store has to be associated with a data center");
>>        }
>> 
>> So my non Java expertise tells me that dcId is null because zoneId is not set ???
>> 
>> 
>>> 
>>> 
>>> -----Original Message-----
>>> From: Sebastien Goasguen [mailto:runseb@gmail.com] 
>>> Sent: Monday, 23 June 2014 5:47 PM
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: S3 use with simulator
>>> 
>>> 
>>> On Jun 23, 2014, at 7:14 AM, Koushik Das <koushik.das@citrix.com> wrote:
>>> 
>>>> This is the request
>>>> 
>>>> http://localhost:8096/api?command=updateCloudToUseObjectStore&url=http
>>>> ://localhost:1234/test&name=test&provider=S3&details[0].key=accesskey&
>>>> details[0].value=dummy&details[1].key=secretkey&details[1].value=dummy
>>> 
>>> Any chance you can try with a signed api request ?
>>> 
>>> It looks like it's not finding a zoneId ...hence dcId is null....
>>> 
>>>> 
>>>> 
>>>> -----Original Message-----
>>>> From: sebgoa [mailto:runseb@gmail.com]
>>>> Sent: Monday, 23 June 2014 4:40 PM
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: S3 use with simulator
>>>> 
>>>> 
>>>> On Jun 23, 2014, at 12:51 PM, Koushik Das <koushik.das@citrix.com>
wrote:
>>>> 
>>>>> inline
>>>>> 
>>>>> On 23-Jun-2014, at 4:05 PM, Sebastien Goasguen <runseb@gmail.com>
wrote:
>>>>> 
>>>>>> Hi Koushik, thanks for the hint, I checked out 4.4-forward and made
that small change.
>>>>>> 
>>>>>> I still add the same error:
>>>>>> 
>>>>>>> update cloudtouseobjectstore url=http://localhost:9081/riak-cs

>>>>>>> name=riakcs provider=s3 details[0].key=accesskey 
>>>>>>> details[0].value=ZMJD6-90S2MST4NZMK1Z details[1].key=secretkey

>>>>>>> details[1].value=ecJlaZebrYKj_qYaIfzlRR_1izojGVWjBRFx0Q==
>>>>>> 530: Failed to add data store: DataCenter id is null, and simulator

>>>>>> image store has to be associated with a data center
>>>>>> 
>>>>>> However this actually *removed* the NFS store (from the simulated
basic zone).
>>>>>> 
>>>>> 
>>>>> I verified using advanced zone, not sure if there are any issues with
basic zone. The API response I get is the following:
>>>>> <updatecloudtouseobjectstoreresponse
>>>>> cloud-stack-version="4.4.0-SNAPSHOT"><imagestore><id>55b7e6f7-ec27-4d
>>>>> a 
>>>>> 8-b4a0-b209a3e3d825</id><name>test</name><url>http://localhost:1234/t
>>>>> e 
>>>>> st</url><protocol>http</protocol><providername>S3</providername><scop
>>>>> e
>>>>>> REGION</scope><details><name>secretkey</name><value>dummy</value></d
>>>>>> e
>>>>> tails><details><name>accesskey</name><value>dummy</value></details></
>>>>> tails>i
>>>>> magestore></updatecloudtouseobjectstoreresponse>
>>>>> 
>>>> 
>>>> Hum, strange. I just checked with an advanced zone from setup/dev/advanced.cfg
and I get the same error 'datacenter id is null'.
>>>> 
>>>> Can you send the exact request that you sent ?
>>>> 
>>>>> Also in the image_store table the entries look like below, note that
the 'role' of NFS store changed to 'ImageCache'.
>>>>> 
>>>>> '1', 'nfs://10.147.28.6:/export/home/sandbox/secondary', 'NFS', 
>>>>> 'nfs', 'nfs://10.147.28.6:/export/home/sandbox/secondary', '1', 
>>>>> 'ZONE', 'ImageCache', '8eb3e701-7e2b-4925-87f8-f8ea5f97b863',
>>>>> '/mnt/05b33ef7-da6a-33eb-9981-b4efb90ed99b/', '2014-06-23 06:27:58',

>>>>> NULL, NULL, NULL '2', 'test', 'S3', 'http', 
>>>>> 'http://localhost:1234/test', NULL, 'REGION', 'Image', 
>>>>> '55b7e6f7-ec27-4da8-b4a0-b209a3e3d825', NULL, '2014-06-23 07:37:55',

>>>>> NULL, NULL, NULL
>>>>> 
>>>> 
>>>> Yes I can confirm this. It gets listed in the UI under 'secondary staging
store', and it's in the db.
>>>> 
>>>>> 
>>>>>> I was then able to add:
>>>>>> 
>>>>>>> add imagestore name=riakcs provider=S3 
>>>>>>> url=http://localhost:9081/riak-cs details[0].key=accesskey 
>>>>>>> details[0].value=ZMJD6-90S2MST4NZMK1Z details[1].key=secretkey

>>>>>>> details[1].value=ecJlaZebrYKj_qYaIfzlRR_1izojGVWjBRFx0Q==
>>>>>> imagestore:
>>>>>> name = riakcs
>>>>>> id = 1a60d62a-c1e9-4d1c-8b35-d5cd687f6de4
>>>>>> details:
>>>>>> +-----------+------------------------------------------+
>>>>>> |    name   |                  value                   |
>>>>>> +-----------+------------------------------------------+
>>>>>> | secretkey | ecJlaZebrYKj_qYaIfzlRR_1izojGVWjBRFx0Q== |
>>>>>> | accesskey |           ZMJD6-90S2MST4NZMK1Z           |
>>>>>> +-----------+------------------------------------------+
>>>>>> protocol = http
>>>>>> providername = S3
>>>>>> scope = REGION
>>>>>> url = http://localhost:9081/riak-cs
>>>>>> 
>>>>>> Apparently successfully, even though the removal of the NFS store
entirely is a bit worrisome.
>>>>>> 
>>>>>> On Jun 23, 2014, at 4:32 AM, Koushik Das <koushik.das@citrix.com>
wrote:
>>>>>> 
>>>>>>> The error message is from the simulator storage plugin, not sure
why you are getting it even after setting provider types as S3.
>>>>>>> I tried the same on a 4.4-forward setup and got a different error
(as below).
>>>>>>> "You can only add new image stores from the same provider NFS
already added"
>>>>>>> 
>>>>>>> This error is there as the simulator storage provider doesn't
support migration to object store. Once I fixed it, the image store got added successfully
and the NFS store was converted to a cache store.
>>>>>>> This is the change I made
>>>>>>> 
>>>>>>> diff --git
>>>>>>> a/plugins/hypervisors/simulator/src/org/apache/cloudstack/storage/d
>>>>>>> a tastore/lifecycle/SimulatorImageStoreLifeCycleImpl.java
>>>>>>> b/plugins/hypervisors/simulator/src/org/apache/cloudstack/storage/d
>>>>>>> a tastore/lifecycle/SimulatorImageStoreLifeCycleImpl.java
>>>>>>> index edf1e28..537c7bd 100644
>>>>>>> ---
>>>>>>> a/plugins/hypervisors/simulator/src/org/apache/cloudstack/storage/d
>>>>>>> a tastore/lifecycle/SimulatorImageStoreLifeCycleImpl.java
>>>>>>> +++ b/plugins/hypervisors/simulator/src/org/apache/cloudstack/stora
>>>>>>> +++ g e/datastore/lifecycle/SimulatorImageStoreLifeCycleImpl.java
>>>>>>> @@ -133,7 +133,7 @@ public class SimulatorImageStoreLifeCycleImpl
implements ImageStoreLifeCycle {
>>>>>>> */
>>>>>>> @Override
>>>>>>> public boolean migrateToObjectStore(DataStore store) {
>>>>>>> -        return false;
>>>>>>> +        return imageStoreHelper.convertToStagingStore(store);
>>>>>>> }
>>>>>>> 
>>>>>>> }
>>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: Min Chen [mailto:min.chen@citrix.com]
>>>>>>> Sent: Saturday, 21 June 2014 3:44 AM
>>>>>>> To: dev@cloudstack.apache.org
>>>>>>> Subject: Re: S3 use with simulator
>>>>>>> 
>>>>>>> Got it. The error seems from simulator, maybe Koushik has better
idea on that. I didn't use that on simulator env though.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> -min
>>>>>>> 
>>>>>>> On 6/20/14 11:05 AM, "Sebastien Goasguen" <runseb@gmail.com>
wrote:
>>>>>>> 
>>>>>>>> I already have an Nsf secondary storage. I understood that
i could 
>>>>>>>> not add a second secondary storage that is s3 based, and
that the 
>>>>>>>> only way was to use the updatecloud.... Api
>>>>>>>> 
>>>>>>>> -Sebastien
>>>>>>>> 
>>>>>>>>> On 20 Jun 2014, at 18:49, Min Chen <min.chen@citrix.com>
wrote:
>>>>>>>>> 
>>>>>>>>> Hi Sebastien,
>>>>>>>>> 
>>>>>>>>> I am curious why you are using updateCloudToUseObjectStore
api to 
>>>>>>>>> add S3  as a secondary store in your data center? To
set up a 
>>>>>>>>> data center using  S3, you need to use addImageStore
api.
>>>>>>>>> 
>>>>>>>>> Thanks
>>>>>>>>> -min
>>>>>>>>> 
>>>>>>>>>> On 6/20/14 7:47 AM, "Sebastien Goasguen" <runseb@gmail.com>
wrote:
>>>>>>>>>> 
>>>>>>>>>> I switch to provider=S3 but I get:
>>>>>>>>>> 
>>>>>>>>>>> 530: Failed to add data store: DataCenter id
is null, and 
>>>>>>>>>>> simulator image store has to be associated with
a data center
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Jun 20, 2014, at 7:07 AM, Koushik Das 
>>>>>>>>>>> <koushik.das@citrix.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> As per the code the following are the supported
image store 
>>>>>>>>>>> provider types
>>>>>>>>>>> 
>>>>>>>>>>> // constants for provider names static final
String NFS_IMAGE = 
>>>>>>>>>>> "NFS"; static final String S3_IMAGE = "S3"; static
final String 
>>>>>>>>>>> SWIFT_IMAGE = "Swift"; static final String SAMPLE_IMAGE
= 
>>>>>>>>>>> "Sample"; static final String SMB = "NFS"; static
final String 
>>>>>>>>>>> DEFAULT_PRIMARY = "DefaultPrimary";
>>>>>>>>>>> 
>>>>>>>>>>> Try passing S3 in the provider.
>>>>>>>>>>> BTW when you setup simulator the image provider
is NFS.
>>>>>>>>>>> 
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>>>>>>>>>>> Sent: Thursday, 19 June 2014 5:08 PM
>>>>>>>>>>> To: dev@cloudstack.apache.org
>>>>>>>>>>> Subject: S3 use with simulator
>>>>>>>>>>> 
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>> I am using the simulator and started a basic
zone.
>>>>>>>>>>> I have an S3 object store locally (riakCS), and
I am trying to 
>>>>>>>>>>> add it to the infra using the 'cloudtouseobjectstore'
api with cloudmonkey.
>>>>>>>>>>> 
>>>>>>>>>>> I tried with:
>>>>>>>>>>> 
>>>>>>>>>>>> update cloudtouseobjectstore url=http://localhost:9081/riak-cs

>>>>>>>>>>>> name=riak provider=riakcs
>>>>>>>>>>>> 
>>>>>>>>>>>> details[0].key=accesskey&details[0].value=STU6Z-ZMK1TPMDAXL9I1
>>>>>>>>>>>> &
>>>>>>>>>>>> deta
>>>>>>>>>>>> ils
>>>>>>>>>>>> [1
>>>>>>>>>>>> 
>>>>>>>>>>>> ].key=secretkey&details[1].value=8OuY3mHDXihu0Tdb2aVJ4vuYZLBAl
>>>>>>>>>>>> 5
>>>>>>>>>>>> Z7Ni
>>>>>>>>>>>> WKs
>>>>>>>>>>>> g=
>>>>>>>>>>>> =
>>>>>>>>>>> 530: Failed to add data store: DataCenter id
is null, and 
>>>>>>>>>>> simulator image store has to be associated with
a data center
>>>>>>>>>>> 
>>>>>>>>>>> I am not sure if the arguments are right, has
anyone done this 
>>>>>>>>>>> before ?
>>>>>>>>>>> 
>>>>>>>>>>> thanks,
>>>>>>>>>>> 
>>>>>>>>>>> -sebastien
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
> 
> -- 
> Prasanna.,
> 
> ------------------------
> Powered by BigRock.com
> 


Mime
View raw message