cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Burwell <jburw...@basho.com>
Subject Re: [ACS41] Next release candidate coming in a few hours!
Date Tue, 14 May 2013 22:54:34 GMT
Chip,

After some further discussion about this issue on IRC, Alex and I determined that system VM
clock drift issue not only breaks S3, but has other significant impacts that merit it being
a blocker for 4.1 (e.g. timestamps of files written by the SSVM being incorrect, log file
correlation difficulties, sensitivity of other storage protocols to time sync, etc).  Based
on this conversation, I have opened CLOUDSTACK-2492 to capture the issue and track resolution.

Thanks,
-John

On May 14, 2013, at 6:09 PM, John Burwell <jburwell@basho.com> wrote:

> Chip,
> 
> The source of the problem appears to be clock drift between the SSVM and S3 per following
stack trace:
> 
> 2013-05-14 06:51:55,400 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) Putting
directory /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5 in S3 bucket
jsb-cloudstack-templates.
> 2013-05-14 06:51:55,401 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) Creating
S3 client with configuration: [protocol: https, connectionTimeOut: 50000, maxErrorRetry: 3,
socketTimeout: 50000]
> 2013-05-14 06:51:55,403 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:)
Determining key using account id 1 and template id 5
> 2013-05-14 06:51:55,403 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) Putting
file /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5/template.properties
into bucket jsb-cloudstack-templates with key template/tmpl/1/5/template.properties.
> 2013-05-14 06:51:55,578 ERROR [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:)
Failed to upload template id 5
> Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 970A274E132A9ACB, AWS Error
Code: RequestTimeTooSkewed, AWS Error Message: The difference between the request time and
the current time is too large., S3 Extended Request ID: 9w8a6YBxTn+WlBg96s9stxWuuP8oQ7ksZtg6++wVRHJfE2qmucrilhoEJVetJui4
>         at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:609)
>         at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:309)
>         at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
>         at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2863)
>         at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1100)
>         at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:963)
>         at com.cloud.utils.S3Utils.putDirectory(S3Utils.java:282)
>         at com.cloud.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:414)
>         at com.cloud.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:212)
>         at com.cloud.agent.Agent.processRequest(Agent.java:525)
>         at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
>         at com.cloud.utils.nio.Task.run(Task.java:83)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 
> It does not appear that the System VM image has NTP nor does it seem like a valid assumption
that an SSVM would have the connectivity to reach an NTP server.  Therefore, do we have some
other means to sync the clock on the SSVM (e.g. with the host through hypervisor extensions)?
 In my previous tests, the SSVM seemed to be syncing its time to the host which addressed
the problem.
> 
> Thanks,
> -John
> 
> On May 14, 2013, at 4:58 PM, John Burwell <jburwell@basho.com> wrote:
> 
>> Chip,
>> 
>> I am looking into the issue now.  There is a failure when the S3 upload template
command is issued.  I working to determine whether or not the cause is environmental or code.
>> 
>> Thanks,
>> -John
>> 
>> On May 14, 2013, at 4:56 PM, Chip Childers <chip.childers@sungard.com> wrote:
>> 
>>> Hi all,
>>> 
>>> We have a clear bug list (blockers and critical) for 4.1.0.  I'm going
>>> to cut a new release candidate tonight.
>>> 
>>> If there are *any* outstanding issues known, now's the time to raise
>>> them.
>>> 
>>> (I'm specifically looking for an ACK from jburwell here, since he
>>> mentioned a possible S3 feature issue.  John, if you can please give me
>>> a heads up on where you stand in the next few hours, I'd appreciate it.)
>>> 
>>> -chip
>> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message