Mailing-List: contact dev-help@deltacloud.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@deltacloud.apache.org
Received-SPF: pass (nike.apache.org: domain of mandreou@redhat.com designates
 209.132.183.28 as permitted sender)
Message-ID: <51024927.4050805@redhat.com>
Date: Fri, 25 Jan 2013 10:58:15 +0200
From: "marios@redhat.com" <mandreou@redhat.com>
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/17.0 Thunderbird/17.0
MIME-Version: 1.0
To: dev@deltacloud.apache.org
CC: "Koper, Dies" <diesk@fast.au.fujitsu.com>
Subject: Re: api test failures - how to resolve?
References: <434A0ECB689CAF49A3A2321F30F2AB8323CF09B9@FALEX03.au.fjanz.com>
 <434A0ECB689CAF49A3A2321F30F2AB8324902D20@FALEX03.au.fjanz.com>
In-Reply-To: <434A0ECB689CAF49A3A2321F30F2AB8324902D20@FALEX03.au.fjanz.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 25/01/13 09:33, Koper, Dies wrote:
> Hi Marios,
> 
> I've been working on getting the api tests to run on the fgcp.
> It's been a good experience as I found some bugs/room for improvement
> for the driver.
> 
> I have just sent a patch to the ML with changes to both fgcp driver and
> tests that address most issues.
> Please review.

thanks - can you please check if that one applies for you? I couldn't
get it to apply - may need a rebase (or otherwise if it works for you
let me know I can try again...)

> 
> May I have your feedback on the following:
> 
> 1. With the fgcp, an instance needs to be started at least once (and
> then stopped again) or else you can't create an image of it. So this
> test fails. Do you have any suggestion how I can address it? There is no
> API to tell whether an instance has been started once.

... can't think of an 'obvious' solution here. You could add some
waiting logic (as you suggest for 3 below) but I agree with you that
this may be unacceptable for other drivers; these tests are meant to me
'driver agnostic' - i.e. you just point them at some running deltacloud
server with little/no knowledge of what's running there and they
discover stuff/skip accordingly. So with this in mind - for this kind of
'fgcp specific' scenario, it may be better to look at adding to the
white-box tests under /deltacloud/server/tests/drivers... i can't see
fgcp tests there yet.

I think this may be indicative of a problem with how we advertise
actions. The name of the test:

239 "it 'should allow to snapshot running instance if supported by
provider' do"

gives a clue here. We are talking about creating an image from a running
server. In FGCP case, this is not possible. For starters, we shouldn't
be advertising the 'create_image' action for running instances, when you
can't actually create the image from a running instance. If that logic
were working correctly, this test would just be skipped (i.e. fgcp
instance running... no create_image...):

243 skip "no create image support for instance #{@@my_instance_id}"
unless instance_actions.include?("create_image")$

This needs more thought - perhaps a JIRA would be a good start.

> 
> 2. Storage snapshots assume there are snapshots present. I get an error
> saying storage_snapshots must have some storage_snapshot elements
> inside. I suppose the only way to address this one is to create a volume
> and then create a snapshot.
> 

right - either what you suggest here - which is what other tests... like
'instances_test' for example do with the 'setup' stage of creating the
given resource before running the tests... or rethink the test itself to
be more lenient. Either way - its something that should be changed in
the test - i.e. not in the fgcp driver.


> 3. Fgcp won't let me create an instance while another resource is being
> created. I need to add some waiting logic. I have a patch for that,
> which works for fgcp, but I'm not sure it adds inacceptable waiting time
> for other drivers. I'll send it separately, as I don't want a NACK for
> this one delay an ACK for the first one.

well... ok let's deal with that one when we have the patch to point at
and discuss,


thanks, marios

> 
> Cheers,
> Dies Koper
> 
> 
>> -----Original Message-----
>> From: Koper, Dies [mailto:diesk@fast.au.fujitsu.com]
>> Sent: Friday, 30 November 2012 1:12 AM
>> To: dev@deltacloud.apache.org
>> Subject: api test failures - how to resolve?
>>
>> Hi,
>>
>> I've run the API tests on fgcp, i.e. rake test:deltacloud:
>>
>> 166 tests, 1340 assertions, 2 failures, 8 errors, 43 skips
>>
>> I'd like your suggestions on how to resolve the failures. My goal is
> to
>> make changes so that a test run like this shows no failures for fgcp
> so
>> that it can be used for easy regression testing.
>>
>>   1) Failure:
>> test_0001_must have the "state" element defined for each instance in
>> collection(Deltacloud API instances collection)
>> [d:/sources/OSS/cloud/deltacloud/tests/delt
>> acloud/instances_test.rb:90]:
>> Expected [] to not be empty.
>>
>> Fgcp driver requires to make an additional backend call for each
>> instance in the collection, which caused http timeouts to occur with
> my
>> (big) contract.
>> Therefore, the fgcp driver only includes state when a single instance
> is
>> retrieved, not for the whole collection.
>> I thought that was acceptable, but here it says 'must'?
>>
>>   2) Error:
>> test_0012_should allow to snapshot running instance if supported by
>> provider(Deltacloud API instances collection):
>> RestClient::ResourceNotFound: 404 Resource Not Found
>>
>> Fgcp only allows snapshots for stopped instances.
>>
>> I'll investigate the remaining ones tomorrow.
>>
>> Regards,
>> Dies Koper
>>
>>
> 
>