Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Received-SPF: pass (nike.apache.org: domain of min.chen@citrix.com designates
 66.165.176.89 as permitted sender)
From: Min Chen <min.chen@citrix.com>
To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>,
	"users@cloudstack.apache.org" <users@cloudstack.apache.org>
Subject: Re: S3 Secondary Storage fails with large files
Thread-Topic: S3 Secondary Storage fails with large files
Thread-Index: AQHO1Yv1tJRgp5F1dUq6kp58H37aqpoNcnmAgACC3QD//414AA==
Date: Wed, 30 Oct 2013 17:39:15 +0000
Message-ID: <CE969145.4D457%min.chen@citrix.com>
In-Reply-To: <29537605.449.1383154147818.JavaMail.andrei@finka>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.3.6.130613
Content-Type: text/plain; charset="us-ascii"
Content-ID: <17983DC608716A41B1ACBD3FAAA1E1A7@citrix.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

So you are doing create template from volume, right? I think that it
should go through the same code used in backup snapshot, so will have the
same limit. Would you mind paste the stack trace you got when you got that
error in this case?

Thanks
-min

On 10/30/13 10:29 AM, "Andrei Mikhailovsky" <andrei@arhont.com> wrote:

>Hi,=20
>
>I've just tried creating a template from a 10GB disk image file and it
>has failed with the same error - EntityTooLarge. So, neither snapshots
>nor templates over 5GB in size are working for me. Could someone confirm
>if I am not the only one?
>
>Thanks=20
>
>Andrei=20
>----- Original Message -----
>
>From: "Min Chen" <min.chen@citrix.com>
>To: dev@cloudstack.apache.org, users@cloudstack.apache.org
>Sent: Wednesday, 30 October, 2013 4:40:48 PM
>Subject: Re: S3 Secondary Storage fails with large files
>
>For 4.2.0, you should be able to register template > 5GB. But we are
>aware=20
>of an issue you mentioned here: backup snapshot > 5GB will fail since
>that=20
>part of code is still using S3 single part upload api. This bug has been
>fixed in latest 4.2 branch, and will be available in 4.2.1 release.
>
>Thanks=20
>-min=20
>
>On 10/30/13 9:18 AM, "Andrei Mikhailovsky" <andrei@arhont.com> wrote:
>
>>Hello Guys ,=20
>>=20
>>I am using ACS 4.2.0 with KVM hypervisor. While trying to create
>>templates and snapshots from the existing vms I am getting an error
>>message / exception on the management logs:
>>=20
>>2013-10-30 13:29:07,264 DEBUG [storage.snapshot.SnapshotManagerImpl]
>>(Job-Executor-12:job-41 =3D [ 85659739-df2e-4afe-b638-2d25e804063f ])
>>Failed to create snapshot
>>com.cloud.utils.exception.CloudRuntimeException: Status Code: 400, AWS
>>Service: Amazon S3, AWS Request ID: null, AWS Error Code:
>>EntityTooLarge,=20
>>AWS Error Message: null, S3 Extended Request ID: null
>>at=20
>>com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.
>>j=20
>>ava:610)=20
>>at=20
>>com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:3
>>1=20
>>0)=20
>>at=20
>>com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164)
>>at=20
>>com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2906)
>>=20
>>at=20
>>com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:11
>>2=20
>>3)=20
>>at=20
>>com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:97
>>9=20
>>)=20
>>at com.cloud.utils.S3Utils.putFile(S3Utils.java:143)
>>at=20
>>com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.copyToS3(KVMStorageP
>>r=20
>>ocessor.java:556)
>>at=20
>>com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.copyToObjectStore(KV
>>M=20
>>StorageProcessor.java:582)
>>at=20
>>com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.backupSnapshotForObj
>>e=20
>>ctStore(KVMStorageProcessor.java:622)
>>at=20
>>com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.backupSnapshot(KVMSt
>>o=20
>>rageProcessor.java:634)
>>at=20
>>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(Sto
>>r=20
>>ageSubsystemCommandHandlerBase.java:84)
>>at=20
>>com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStora
>>g=20
>>eCommands(StorageSubsystemCommandHandlerBase.java:49)
>>at=20
>>com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest
>>(=20
>>LibvirtComputingResource.java:1286)
>>at com.cloud.agent.Agent.processRequest(Agent.java:525)
>>at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
>>at com.cloud.utils.nio.Task.run(Task.java:83)
>>at=20
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
>>:=20
>>1146)=20
>>at=20
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
>>a=20
>>:615)=20
>>at java.lang.Thread.run(Thread.java:679)
>>=20
>>at=20
>>org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot
>>(=20
>>SnapshotServiceImpl.java:280)
>>at=20
>>org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSn
>>a=20
>>pshot(XenserverSnapshotStrategy.java:138)
>>at=20
>>org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnap
>>s=20
>>hot(XenserverSnapshotStrategy.java:264)
>>at=20
>>com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManag
>>e=20
>>rImpl.java:1013)=20
>>at=20
>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interceptor
>>D=20
>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>at=20
>>org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(Volum
>>e=20
>>ServiceImpl.java:1307)
>>at=20
>>com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2
>>7=20
>>19)=20
>>at=20
>>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute
>>(=20
>>CreateSnapshotCmd.java:170)
>>at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
>>at=20
>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
>>at=20
>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>at=20
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
>>:=20
>>1146)=20
>>at=20
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
>>a=20
>>:615)=20
>>at java.lang.Thread.run(Thread.java:679)
>>2013-10-30 13:29:07,270 DEBUG [storage.volume.VolumeServiceImpl]
>>(Job-Executor-12:job-41 =3D [ 85659739-df2e-4afe-b638-2d25e804063f ]) Tak=
e
>>snapshot: 41 failed
>>com.cloud.utils.exception.CloudRuntimeException: Failed to create
>>snapshot=20
>>at=20
>>com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManag
>>e=20
>>rImpl.java:1040)=20
>>at=20
>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interceptor
>>D=20
>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>at=20
>>org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(Volum
>>e=20
>>ServiceImpl.java:1307)
>>at=20
>>com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2
>>7=20
>>19)=20
>>at=20
>>org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute
>>(=20
>>CreateSnapshotCmd.java:170)
>>at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
>>at=20
>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
>>at=20
>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>at=20
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
>>:=20
>>1146)=20
>>at=20
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
>>a=20
>>:615)=20
>>at java.lang.Thread.run(Thread.java:679)
>>Caused by: com.cloud.utils.exception.CloudRuntimeException: Status Code:
>>400, AWS Service: Amazon S3, AWS Request ID: null, AWS Error Code:
>>EntityTooLarge, AWS Error Message: null, S3 Extended Request ID: null
>>=20
>>=20
>>Does anyone know how to resolve this problem? I can upload files up to
>>5GB in size. Anything over that fails.
>>=20
>>Thanks for help=20
>>=20
>>Andrei=20
>>=20
>
>