From users-return-33883-archive-asf-public=cust-asf.ponee.io@cloudstack.apache.org Tue Jan 14 14:05:28 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 316D318061A for ; Tue, 14 Jan 2020 15:05:28 +0100 (CET) Received: (qmail 22707 invoked by uid 500); 14 Jan 2020 14:05:27 -0000 Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@cloudstack.apache.org Delivered-To: mailing list users@cloudstack.apache.org Received: (qmail 22694 invoked by uid 99); 14 Jan 2020 14:05:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jan 2020 14:05:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id AF870C0870 for ; Tue, 14 Jan 2020 14:05:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.002 X-Spam-Level: X-Spam-Status: No, score=0.002 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id eQ1UD5Tut8VX for ; Tue, 14 Jan 2020 14:05:19 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.210.52; helo=mail-ot1-f52.google.com; envelope-from=daan.hoogland@gmail.com; receiver= Received: from mail-ot1-f52.google.com (mail-ot1-f52.google.com [209.85.210.52]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 52810BE74F for ; Tue, 14 Jan 2020 14:05:19 +0000 (UTC) Received: by mail-ot1-f52.google.com with SMTP id p8so12655553oth.10 for ; Tue, 14 Jan 2020 06:05:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=HFADJ/ILEokqlF5Hih+rama9+BbFd3ypKhT9EBZrdDQ=; b=HPoE5whfFLK0/Y4+SJoossk2U5iQ/iDoG+WxtYgvQ9ZJM/4r+DfUVNeRNnMejfZeTz /fq35kcakS+nqQ+CeurmHQz1wfZIcFe7ZOTf7U0A44qV3qilfBRKVIXj93iSMISbx8sX Yi+iLWmwZ5+noQZQ4YXQrY1WrIbUxInp6zF/ZWkSzwYCLRbGUraCCPQ5R8b3pC51rxlS 4igTXZ9yJ1pfu/itTH3HcSETijbuEySzzEBiDcO5l9YeC5/xRERsGGy/nhl7EqWyOFnu 1U+yQqEg9CeBekjKxtOWYI9dAJDczOsJgolq82th6UvAYjK/D/1cg9mLM7ISM5Rhob9/ PIWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=HFADJ/ILEokqlF5Hih+rama9+BbFd3ypKhT9EBZrdDQ=; b=ZqBsiBzxi9SMEfQh9Ue/dgrxbSFK/EJe2i8+VIEJpO0pKT36WgSH6364uKk1DUe3wv b7JJL6Aw+Wf0zD38f9I6mknBt7byyRFJ8tZA+H93pV7pfJBRjHOGnerpyGPRcPaRk9cZ cfCtFd8G7has5mg7BYncIUIClFidLyYB0TURyMjljq82RVb3XgIWrAASuFhNgbIUQfOp MPr6DUoCvpW9FHCIvVaCExOYqWekasXF3K0SENZEmvembKX2wn1a4yTUo6RZGPHzMxwk NfysYD4iyBVfWly8BuparijpUy4alno5MOg5/H+KF/cWEAvESJZiKXlF961IV+a2PrZ8 Eirg== X-Gm-Message-State: APjAAAUDxO9IMDZCaXxSDOThhzOkbQ91QpF889xm1qL1U9OR4SyLnFuN qiDOsdRQRimacjrMG//HSwOfxNHqF97SqlryrEQ= X-Google-Smtp-Source: APXvYqxDGJdfxB5DuBm1TQLRI8+a0d8oX1sXntwPIHqeyFiCAMkdX2LMQLV3kwid9Lqobaq6eQjQJDI6m3er5H80mxQ= X-Received: by 2002:a9d:70cb:: with SMTP id w11mr17778856otj.230.1579010718561; Tue, 14 Jan 2020 06:05:18 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Daan Hoogland Date: Tue, 14 Jan 2020 15:05:05 +0100 Message-ID: Subject: Re: new primary storage To: Charlie Holeowsky Cc: users Content-Type: multipart/alternative; boundary="000000000000f9fa71059c1a14bd" --000000000000f9fa71059c1a14bd Content-Type: text/plain; charset="UTF-8" So Charlie, d93d3c0a-3859-4473-951d-9b5c5912c767 is actually a valid disk? does it exist on the backend nfs? and the pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c does it exist both in cloudstack and on the backend? if both are answered with yes, you probably have a permissions issue, which might be in the network. On Tue, Jan 14, 2020 at 10:21 AM Charlie Holeowsky < charlie.holeowsky@gmail.com> wrote: > Hi Daan and users, > the infrastructure is based on the Linux environment. The management > server, hosts and storage are all Ubuntu 16.04 except the new storage > server which is an Ubuntu 18.04. The hypervisor used is Qemu-kvm with NFS > to share the storage. > > We tried to add another primary storage and creating a VM that would use > it we found no problems, the statistics update and no error messages appear. > > Gere is an excerpt of the logs of the most complete agent: > > 2020-01-14 09:01:45,749 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-2:null) (logid:c3851d3a) Trying to fetch storage pool > 171e90f4-511e-3b10-9310-b9eec0094be6 from libvirt > 2020-01-14 09:01:45,752 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-2:null) (logid:c3851d3a) Asking libvirt to refresh > storage pool 171e90f4-511e-3b10-9310-b9eec0094be6 > 2020-01-14 09:01:46,641 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-4:null) (logid:c3851d3a) Trying to fetch storage pool > 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt > 2020-01-14 09:01:46,643 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-4:null) (logid:c3851d3a) Asking libvirt to refresh > storage pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c > 2020-01-14 09:05:51,529 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-1:null) (logid:2765ff88) Trying to fetch storage pool > 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt > 2020-01-14 09:05:51,532 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-1:null) (logid:2765ff88) Asking libvirt to refresh > storage pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c > 2020-01-14 09:10:47,286 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-3:null) (logid:6d27b740) Trying to fetch storage pool > 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt > 2020-01-14 09:10:47,419 WARN [cloud.agent.Agent] > (agentRequest-Handler-3:null) (logid:6d27b740) Caught: > com.cloud.utils.exception.CloudRuntimeException: Can't find > volume:d93d3c0a-3859-4473-951d-9b5c5912c767 > at > com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) > at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450) > at com.cloud.agent.Agent.processRequest(Agent.java:645) > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083) > at com.cloud.utils.nio.Task.call(Task.java:83) > at com.cloud.utils.nio.Task.call(Task.java:29) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-01-14 09:20:48,390 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-4:null) (logid:ec72387b) Trying to fetch storage pool > 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt > 2020-01-14 09:20:48,536 WARN [cloud.agent.Agent] > (agentRequest-Handler-4:null) (logid:ec72387b) Caught: > com.cloud.utils.exception.CloudRuntimeException: Can't find > volume:d93d3c0a-3859-4473-951d-9b5c5912c767 > at > com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) > at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450) > at com.cloud.agent.Agent.processRequest(Agent.java:645) > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083) > at com.cloud.utils.nio.Task.call(Task.java:83) > at com.cloud.utils.nio.Task.call(Task.java:29) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-01-14 09:25:15,259 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-5:null) (logid:1a7e082e) Trying to fetch storage pool > 9af0d1c6-85f2-3c55-94af-6ac17cb4024c from libvirt > 2020-01-14 09:25:15,261 INFO [kvm.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-5:null) (logid:1a7e082e) Asking libvirt to refresh > storage pool 9af0d1c6-85f2-3c55-94af-6ac17cb4024c > > > And here the management server log: > > 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request] > (AgentManager-Handler-2:null) (logid:) Seq 15-705657766613619075: > Processing: { Ans: , MgmtId: 220777304233416, via: 15, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: > Can't find volume:d93d3c0a-3859-4473-951d-9b5c5912c767\n\tat > com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149)\n\tat > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63)\n\tat > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52)\n\tat > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40)\n\tat > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)\n\tat > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450)\n\tat > com.cloud.agent.Agent.processRequest(Agent.java:645)\n\tat > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083)\n\tat > com.cloud.utils.nio.Task.call(Task.java:83)\n\tat > com.cloud.utils.nio.Task.call(Task.java:29)\n\tat > java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat > java.lang.Thread.run(Thread.java:748)\n","wait":0}}] } > 2020-01-14 09:21:27,105 DEBUG [c.c.a.t.Request] > (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Seq 15-705657766613619075: > Received: { Ans: , MgmtId: 220777304233416, via: 15(csdell017), Ver: v1, > Flags: 10, { Answer } } > 2020-01-14 09:21:27,105 DEBUG [c.c.a.m.AgentManagerImpl] > (StatsCollector-6:ctx-fd801d0a) (logid:ec72387b) Details from executing > class com.cloud.agent.api.GetVolumeStatsCommand: > com.cloud.utils.exception.CloudRuntimeException: Can't find > volume:d93d3c0a-3859-4473-951d-9b5c5912c767 > at > com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:149) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.getVolumeStat(LibvirtGetVolumeStatsCommandWrapper.java:63) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:52) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetVolumeStatsCommandWrapper.execute(LibvirtGetVolumeStatsCommandWrapper.java:40) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) > at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1450) > at com.cloud.agent.Agent.processRequest(Agent.java:645) > at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1083) > at com.cloud.utils.nio.Task.call(Task.java:83) > at com.cloud.utils.nio.Task.call(Task.java:29) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > > on 09/01/20 12:58, Daan Hoogland wrote: > > Charlie, I think you'll have to explain a bit more about your environment > to get an answer. what type of storage is it? Where did you migrate the VM > from and to? What types() of hypervisors are you using? Though saying **the** > agent logs suggests KVM, you are still leaving people guessing a lot. > > -- Daan --000000000000f9fa71059c1a14bd--