Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 79E2110D04 for ; Thu, 13 Feb 2014 10:22:24 +0000 (UTC) Received: (qmail 15902 invoked by uid 500); 13 Feb 2014 10:22:23 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 15838 invoked by uid 500); 13 Feb 2014 10:22:22 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 15820 invoked by uid 500); 13 Feb 2014 10:22:21 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 15816 invoked by uid 99); 13 Feb 2014 10:22:21 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Feb 2014 10:22:21 +0000 Date: Thu, 13 Feb 2014 10:22:21 +0000 (UTC) From: "Daan Hoogland (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CLOUDSTACK-6024) template copy to primary storage uses a random source secstorage from any zone MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-6024?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId= =3D13900192#comment-13900192 ]=20 Daan Hoogland commented on CLOUDSTACK-6024: ------------------------------------------- code analysis of the 4.1 branch shows that the problem existed before the e= ntity manager re-factor. Kind of hoped to find the origin there:(=20 > template copy to primary storage uses a random source secstorage from any= zone > -------------------------------------------------------------------------= ----- > > Key: CLOUDSTACK-6024 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-602= 4 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the defa= ult.)=20 > Affects Versions: 4.2.1, 4.1.2, 4.3.0 > Environment: Multiple zones where the secstorage of a zone is not= accessible to hosts from the other zone. > Reporter: Joris van Lieshout > Assignee: Daan Hoogland > Priority: Blocker > > 2014-02-04 15:19:07,674 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Exec= utor-92:job-221857 =3D [ 6f2d5dbb-575e-49b9-89dd-d7567869849e ]) Checking i= f we need to prepare 1 volumes for VM[User|xxxxxx-app01] > 2014-02-04 15:19:07,693 DEBUG [storage.image.TemplateDataFactoryImpl] (Jo= b-Executor-92:job-221857 =3D [ 6f2d5dbb-575e-49b9-89dd-d7567869849e ]) temp= late 467 is already in store:117, type:Image > // store 117 is not accessible from the zone where this hypervisor lives > 2014-02-04 15:19:07,705 DEBUG [storage.datastore.PrimaryDataStoreImpl] (J= ob-Executor-92:job-221857 =3D [ 6f2d5dbb-575e-49b9-89dd-d7567869849e ]) Not= found (templateId:467poolId:208) in template_spool_ref, persisting it > 2014-02-04 15:19:07,718 DEBUG [storage.image.TemplateDataFactoryImpl] (Jo= b-Executor-92:job-221857 =3D [ 6f2d5dbb-575e-49b9-89dd-d7567869849e ]) temp= late 467 is already in store:208, type:Primary > 2014-02-04 15:19:07,722 DEBUG [storage.volume.VolumeServiceImpl] (Job-Exe= cutor-92:job-221857 =3D [ 6f2d5dbb-575e-49b9-89dd-d7567869849e ]) Found tem= plate 467-2-6c05b599-95ed-34c3-b8f0-fd9c30bac938 in storage pool 208 with V= MTemplateStoragePool id: 36433 > 2014-02-04 15:19:07,732 DEBUG [storage.volume.VolumeServiceImpl] (Job-Exe= cutor-92:job-221857 =3D [ 6f2d5dbb-575e-49b9-89dd-d7567869849e ]) Acquire l= ock on VMTemplateStoragePool 36433 with timeout 3600 seconds > 2014-02-04 15:19:07,737 INFO [storage.volume.VolumeServiceImpl] (Job-Exe= cutor-92:job-221857 =3D [ 6f2d5dbb-575e-49b9-89dd-d7567869849e ]) lock is a= cquired for VMTemplateStoragePool 36433 > 2014-02-04 15:19:07,748 DEBUG [storage.motion.AncientDataMotionStrategy] = (Job-Executor-92:job-221857 =3D [ 6f2d5dbb-575e-49b9-89dd-d7567869849e ]) c= opyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLA= TE > 2014-02-04 15:19:07,775 DEBUG [agent.manager.ClusteredAgentAttache] (Job-= Executor-92:job-221857 =3D [ 6f2d5dbb-575e-49b9-89dd-d7567869849e ]) Seq 93= -1862347354: Forwarding Seq 93-1862347354: { Cmd , MgmtId: 345052370018, v= ia: 93, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.Co= pyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"= path":"template/tmpl/2/467/c263eb76-3d72-3732-8cc6-42b0dad55c4d.vhd","origU= rl":"http://xxxxx.xxxxx.com/image/centos64x64-daily-v1b104.vhd","uuid":"ca5= e3f26-e9b6-41c8-a85b-df900be5673c","id":467,"format":"VHD","accountId":2,"c= hecksum":"604a8327bd83850ed621ace2ea84402a","hvm":true,"displayText":"cento= s template created by hans.pl from machine name centos-daily-b104","imageDa= taStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://xxxxxxxx.storage.xx= xx.xxxxxx.xxx/volumes/pool0/xxxx-xxxx-1-1","_role":"Image"}},"name":"467-2-= 6c05b599-95ed-34c3-b8f0-fd9c30bac938","hypervisorType":"XenServer"}},"destT= O":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"http://= xxxxxx.xxxxxx.com/image/centos64x64-daily-v1b104.vhd","uuid":"ca5e3f26-e9b6= -41c8-a85b-df900be5673c","id":467,"format":"VHD","accountId":2,"checksum":"= 604a8327bd83850ed621ace2ea84402a","hvm":true,"displayText":"centos template= created by hans.pl from machine name centos-daily-b104","imageDataStore":{= "org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"b290385b-466= d-3243-a939-3d242164e034","id":208,"poolType":"NetworkFilesystem","host":"x= xxxxxxx.xxxx.xxxxx.net","path":"/volumes/pool0/xxxxxx-XEN-1","port":2049}},= "name":"467-2-6c05b599-95ed-34c3-b8f0-fd9c30bac938","hypervisorType":"XenSe= rver"}},"executeInSequence":true,"wait":10800}}] } to 345052370017 > =3D=3D=3DFILE: server/src/com/cloud/storage/VolumeManagerImpl.java > public void prepare(VirtualMachineProfile vm, > DeployDestination dest) throws StorageUnavailableException, > InsufficientStorageCapacityException, ConcurrentOperationExce= ption { > if (dest =3D=3D null) { > if (s_logger.isDebugEnabled()) { > s_logger.debug("DeployDestination cannot be null, cannot = prepare Volumes for the vm: " > + vm); > } > throw new CloudRuntimeException( > "Unable to prepare Volume for vm because DeployDestin= ation is null, vm:" > + vm); > } > List vols =3D _volsDao.findUsableVolumesForInstance(vm.= getId()); > if (s_logger.isDebugEnabled()) { > s_logger.debug("Checking if we need to prepare " + vols.size(= ) > + " volumes for " + vm); > } > List tasks =3D getTasks(vols, dest.getStorageForDisks= ()); > Volume vol =3D null; > StoragePool pool =3D null; > for (VolumeTask task : tasks) { > if (task.type =3D=3D VolumeTaskType.NOP) { > pool =3D (StoragePool)dataStoreMgr.getDataStore(task.pool= .getId(), DataStoreRole.Primary); > vol =3D task.volume; > } else if (task.type =3D=3D VolumeTaskType.MIGRATE) { > pool =3D (StoragePool)dataStoreMgr.getDataStore(task.pool= .getId(), DataStoreRole.Primary); > vol =3D migrateVolume(task.volume, pool); > } else if (task.type =3D=3D VolumeTaskType.RECREATE) { > Pair result =3D recreateVolume(task.= volume, vm, dest); > pool =3D (StoragePool)dataStoreMgr.getDataStore(result.se= cond().getId(), DataStoreRole.Primary); > vol =3D result.first(); > } > DataTO volumeTO =3D volFactory.getVolume(vol.getId()).getTO()= ; > DiskTO disk =3D new DiskTO(volumeTO, vol.getDeviceId(), null,= vol.getVolumeType()); > vm.addDisk(disk); > } > } > private Pair recreateVolume(VolumeVO vol, Virtua= lMachineProfile vm, > DeployDestination dest) throws StorageUnavailableException { > VolumeVO newVol; > boolean recreate =3D _recreateSystemVmEnabled; > DataStore destPool =3D null; > if (recreate > && (dest.getStorageForDisks() =3D=3D null || dest > .getStorageForDisks().get(vol) =3D=3D null)) { > destPool =3D dataStoreMgr.getDataStore(vol.getPoolId(), DataS= toreRole.Primary); > s_logger.debug("existing pool: " + destPool.getId()); > } else { > StoragePool pool =3D dest.getStorageForDisks().get(vol); > destPool =3D dataStoreMgr.getDataStore(pool.getId(), DataStor= eRole.Primary); > } > if (vol.getState() =3D=3D Volume.State.Allocated > || vol.getState() =3D=3D Volume.State.Creating) { > newVol =3D vol; > } else { > newVol =3D switchVolume(vol, vm); > // update the volume->PrimaryDataStoreVO map since volumeId h= as > // changed > if (dest.getStorageForDisks() !=3D null > && dest.getStorageForDisks().containsKey(vol)) { > StoragePool poolWithOldVol =3D dest > .getStorageForDisks().get(vol); > dest.getStorageForDisks().put(newVol, poolWithOldVol); > dest.getStorageForDisks().remove(vol); > } > if (s_logger.isDebugEnabled()) { > s_logger.debug("Created new volume " + newVol > + " for old volume " + vol); > } > } > VolumeInfo volume =3D volFactory.getVolume(newVol.getId(), destPo= ol); > Long templateId =3D newVol.getTemplateId(); > for (int i =3D 0; i < 2; i++) { > // retry one more time in case of template reload is required= for Vmware case > AsyncCallFuture future =3D null; > if (templateId =3D=3D null) { > future =3D volService.createVolumeAsync(volume, destPool)= ; > } else { > TemplateInfo templ =3D tmplFactory.getTemplate(templateId= , DataStoreRole.Image); > future =3D volService.createVolumeFromTemplateAsync(volum= e, destPool.getId(), templ); > } > VolumeApiResult result =3D null; > try { > result =3D future.get(); > if (result.isFailed()) { > if (result.getResult().contains("request template rel= oad") && (i =3D=3D 0)) { > s_logger.debug("Retry template re-deploy for vmwa= re"); > continue; > } > else { > s_logger.debug("Unable to create " > + newVol + ":" + result.getResult()); > throw new StorageUnavailableException("Unable to = create " > + newVol + ":" + result.getResult(), dest= Pool.getId()); > } > } > newVol =3D _volsDao.findById(newVol.getId()); > break; //break out of template-redeploy retry loop > } catch (InterruptedException e) { > s_logger.error("Unable to create " + newVol, e); > throw new StorageUnavailableException("Unable to create " > + newVol + ":" + e.toString(), destPool.getId()); > } catch (ExecutionException e) { > s_logger.error("Unable to create " + newVol, e); > throw new StorageUnavailableException("Unable to create " > + newVol + ":" + e.toString(), destPool.getId()); > } > } > return new Pair(newVol, destPool); > } > =3D=3D=3DFILE: engine/storage/image/src/org/apache/cloudstack/storage/ima= ge/TemplateDataFactoryImpl.java > public TemplateInfo getTemplate(long templateId, DataStoreRole storeRole)= { > TemplateDataStoreVO tmplStore =3D templateStoreDao.findByTemplate= (templateId, storeRole); > DataStore store =3D null; > if (tmplStore !=3D null) { > store =3D this.storeMgr.getDataStore(tmplStore.getDataStoreId= (), storeRole); > } > return this.getTemplate(templateId, store); > } > =3D=3D=3DFILE: engine/storage/src/org/apache/cloudstack/storage/image/db/= TemplateDataStoreDaoImpl.java > public TemplateDataStoreVO findByTemplate(long templateId, DataStoreRole = role) { > SearchCriteria sc =3D templateRoleSearch.cre= ate(); > sc.setParameters("template_id", templateId); > sc.setParameters("store_role", role); > sc.setParameters("destroyed", false); > return findOneIncludingRemovedBy(sc); > } > =3D=3D=3DFILE: utils/src/com/cloud/utils/db/GenericDaoBase.java > protected T findOneIncludingRemovedBy(final SearchCriteria sc) { > Filter filter =3D new Filter(1); > List results =3D searchIncludingRemoved(sc, filter, null, fals= e); > assert results.size() <=3D 1 : "Didn't the limiting worked?"; > return results.size() =3D=3D 0 ? null : results.get(0); > } > =3D=3D=3DFILE: utils/src/com/cloud/utils/db/Filter.java > public Filter(long limit) { > _orderBy =3D " ORDER BY RAND() LIMIT " + limit; > } -- This message was sent by Atlassian JIRA (v6.1.5#6160)