cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Burwell (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CLOUDSTACK-2492) System VM Clock Drift
Date Tue, 14 May 2013 23:20:54 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13657643#comment-13657643
] 

John Burwell edited comment on CLOUDSTACK-2492 at 5/14/13 11:19 PM:
--------------------------------------------------------------------

My first brush expectation is that each hypervisor's system vm image will need to be modified
to implement the clock synchronization strategy/best practice for the hypervisor.  The following
are some links discussing clock sync per hypervisor:

   * Xen
      - http://wiki.xensource.com/wiki/Xen_FAQ_DomU#How_can_i_synchronize_a_dom0_clock.3F
   * KVM
      - http://docs.fedoraproject.org/en-US/Fedora/13/html/Virtualization_Guide/chap-Virtualization-KVM_guest_timing_management.html
      - http://s19n.net/articles/2011/kvm_clock.html
   * VMWare
     - http://www.vmware.com/files/pdf/techpaper/Timekeeping-In-VirtualMachines.pdf

Also, VMware tools (the typical clock sync solution) is available as open source (http://open-vm-tools.sourceforge.net/).

Finally, we will likely need to add documentation with regards to time keeping to the administration
and/or installation documentation.
                
      was (Author: jburwell):
    My first brush expectation is that each hypervisor's system vm image will need to be modified
to implement the clock synchronization strategy/best practice for the hypervisor.  The following
are some links discussing clock sync per hypervisor:

   * Xen
      - http://wiki.xensource.com/wiki/Xen_FAQ_DomU#How_can_i_synchronize_a_dom0_clock.3F
   * KVM
      - http://docs.fedoraproject.org/en-US/Fedora/13/html/Virtualization_Guide/chap-Virtualization-KVM_guest_timing_management.html
      - http://s19n.net/articles/2011/kvm_clock.html
   * VMWare
     - http://www.vmware.com/files/pdf/techpaper/Timekeeping-In-VirtualMachines.pdf

Also, VMware tools (the typical clock sync solution) is available as open source (http://open-vm-tools.sourceforge.net/).
                  
> System VM Clock Drift
> ---------------------
>
>                 Key: CLOUDSTACK-2492
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2492
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: ISO
>    Affects Versions: 4.1.0
>         Environment: devcloud/Xen
>            Reporter: John Burwell
>            Priority: Blocker
>
> Testing of S3-backed Secondary Storage has revealed that the SSVM (and likely all other
system VMs) have no provision for clock synchronization (e.g. NTP to dom0 for Xen, vmware-tools
for VMWare, etc).  In particular, the S3 protocol is sensitive to drift between clients and
S3.  As an example, the following is the stack trace caused by clock drift S3:
> 2013-05-14 06:51:55,400 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) Putting
directory /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5 in S3 bucket
jsb-cloudstack-templates.
> 2013-05-14 06:51:55,401 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) Creating
S3 client with configuration: [protocol: https, connectionTimeOut: 50000, maxErrorRetry: 3,
socketTimeout: 50000]
> 2013-05-14 06:51:55,403 DEBUG [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:)
Determining key using account id 1 and template id 5
> 2013-05-14 06:51:55,403 DEBUG [cloud.utils.S3Utils] (agentRequest-Handler-3:) Putting
file /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5/template.properties
into bucket jsb-cloudstack-templates with key template/tmpl/1/5/template.properties.
> 2013-05-14 06:51:55,578 ERROR [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:)
Failed to upload template id 5
> Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 970A274E132A9ACB, AWS Error
Code: RequestTimeTooSkewed, AWS Error Message: The difference between the request time and
the current time is too large., S3 Extended Request ID: 9w8a6YBxTn+WlBg96s9stxWuuP8oQ7ksZtg6++wVRHJfE2qmucrilhoEJVetJui4
>         at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:609)
>         at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:309)
>         at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
>         at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2863)
>         at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1100)
>         at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:963)
>         at com.cloud.utils.S3Utils.putDirectory(S3Utils.java:282)
>         at com.cloud.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:414)
>         at com.cloud.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:212)
>         at com.cloud.agent.Agent.processRequest(Agent.java:525)
>         at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
>         at com.cloud.utils.nio.Task.run(Task.java:83)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> In addition to impacting S3, this clock drift also makes log correlation between the
management server and system VMs very difficult, if not, impossible.  Finally, there are suspicions
that the clock drift could also impact operation of console proxy and virtual router VMs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message