Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Received-SPF: error (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <008A60A0-82BE-48BE-BAAE-F245570B49B4@basho.com>
References: <20130514205655.GU24552@USLT-205755.sungardas.corp>
	<CA3CCAA5-9AD0-45BC-9DC1-94B365DD83DD@basho.com>
	<8B75117D-AA85-45B9-8F52-CCE601BFEF7B@basho.com>
	<008A60A0-82BE-48BE-BAAE-F245570B49B4@basho.com>
Date: Tue, 14 May 2013 19:37:51 -0400
Message-ID: 
 <CA+96GG6AObYDqNAt3Tj-b9ip=Vu5QPscCNiYPWi52y0=CF8q0Q@mail.gmail.com>
Subject: Re: [ACS41] Next release candidate coming in a few hours!
From: Chip Childers <chip.childers@sungard.com>
To: John Burwell <jburwell@basho.com>
Cc: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
Content-Type: multipart/alternative; boundary=001a11c2f40222424b04dcb61fee

--001a11c2f40222424b04dcb61fee
Content-Type: text/plain; charset=ISO-8859-1

On Tuesday, May 14, 2013, John Burwell wrote:

> Chip,
>
> After some further discussion about this issue on IRC, Alex and I
> determined that system VM clock drift issue not only breaks S3, but has
> other significant impacts that merit it being a blocker for 4.1 (e.g.
> timestamps of files written by the SSVM being incorrect, log file
> correlation difficulties, sensitivity of other storage protocols to time
> sync, etc).  Based on this conversation, I have opened CLOUDSTACK-2492<https://issues.apache.org/jira/browse/CLOUDSTACK-2492> to
> capture the issue and track resolution.
>
> Thanks,
> -John
>

Well shoot. Issuing a new system VM is going to require a bunch of testing
if it's not done as a hand rolled edit of the existing system VMs.

Who can take this?


>
> On May 14, 2013, at 6:09 PM, John Burwell <jburwell@basho.com<javascript:_e({}, 'cvml', 'jburwell@basho.com');>>
> wrote:
>
> Chip,
>
> The source of the problem appears to be clock drift between the SSVM and
> S3 per following stack trace:
>
> 2013-05-14 06:51:55,400 DEBUG [cloud.utils.S3Utils]
> (agentRequest-Handler-3:) Putting directory
> /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5 in
> S3 bucket jsb-cloudstack-templates.
> 2013-05-14 06:51:55,401 DEBUG [cloud.utils.S3Utils]
> (agentRequest-Handler-3:) Creating S3 client with configuration: [protocol:
> https, connectionTimeOut: 50000, maxErrorRetry: 3, socketTimeout: 50000]
> 2013-05-14 06:51:55,403 DEBUG
> [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:)
> Determining key using account id 1 and template id 5
> 2013-05-14 06:51:55,403 DEBUG [cloud.utils.S3Utils]
> (agentRequest-Handler-3:) Putting file
> /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5/template.properties
> into bucket jsb-cloudstack-templates with key
> template/tmpl/1/5/template.properties.
> 2013-05-14 06:51:55,578 ERROR
> [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:)
> Failed to upload template id 5
> Status Code: 403, AWS Service: Amazon S3, AWS Request ID:
> 970A274E132A9ACB, AWS Error Code: RequestTimeTooSkewed, AWS Error Message:
> The difference between the request time and the current time is too large.,
> S3 Extended Request ID:
> 9w8a6YBxTn+WlBg96s9stxWuuP8oQ7ksZtg6++wVRHJfE2qmucrilhoEJVetJui4
>         at
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:609)
>         at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:309)
>         at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
>         at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2863)
>         at
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1100)
>         at
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:963)
>         at com.cloud.utils.S3Utils.putDirectory(S3Utils.java:282)
>         at
> com.cloud.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:414)
>         at
> com.cloud.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:212)
>         at com.cloud.agent.Agent.processRequest(Agent.java:525)
>         at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
>         at com.cloud.utils.nio.Task.run(Task.java:83)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> It does not appear that the System VM image has NTP nor does it seem like
> a valid assumption that an SSVM would have the connectivity to reach an NTP
> server.  Therefore, do we have some other means to sync the clock on the
> SSVM (e.g. with the host through hypervisor extensions)?  In my previous
> tests, the SSVM seemed to be syncing its time to the host which addressed
> the problem.
>
> Thanks,
> -John
>
> On May 14, 2013, at 4:58 PM, John Burwell <jburwell@basho.com<javascript:_e({}, 'cvml', 'jburwell@basho.com');>>
> wrote:
>
> Chip,
>
> I am looking into the issue now.  There is a failure when the S3 upload
> template command is issued.  I working to determine whether or not the
> cause is environmental or code.
>
> Thanks,
> -John
>
> On May 14, 2013, at 4:56 PM, Chip Childers <chip.childers@sungard.com<javascript:_e({}, 'cvml', 'chip.childers@sungard.com');>>
> wrote:
>
> Hi all,
>
> We have a clear bug list (blockers and critical) for 4.1.0.  I'm going
> to cut a new release candidate tonight.
>
> If there are *any* outstanding issues known, now's the time to raise
> them.
>
> (I'm specifically looking for an ACK from jburwell here, since he
> mentioned a possible S3 feature issue.  John, if you can please give me
> a heads up on where you stand in the next few hours, I'd appreciate it.)
>
> -chip
>
>
>
>
>

--001a11c2f40222424b04dcb61fee--