cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francois Gaudreault <fgaudrea...@cloudops.com>
Subject Re: LXC and SSVM/CPVM on the host
Date Tue, 29 Oct 2013 18:00:21 GMT
Filed as : https://issues.apache.org/jira/browse/CLOUDSTACK-4990

On 10/29/2013, 1:45 PM, Kelven Yang wrote:
> It looks like a LXC resource agent bug. LXC resource agent needs to
> respond VNC command correctly in order to get console work on LXC.
>
> Could you file a bug against LXC resource agent component?
>
> Kelven
>
> On 10/29/13, 7:13 AM, "Francois Gaudreault" <fgaudreault@cloudops.com>
> wrote:
>
>> Kelven,
>>
>> This is what I get in the LXC host agent log when I try to check the
>> console from CS:
>>
>> 2013-10-29 10:07:42,079 WARN  [cloud.agent.Agent]
>> (agentRequest-Handler-4:null) Caught:
>> java.lang.NullPointerException
>>      at
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(Libvirt
>> ComputingResource.java:2711)
>>      at
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(
>> LibvirtComputingResource.java:1216)
>>      at com.cloud.agent.Agent.processRequest(Agent.java:525)
>>      at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
>>      at com.cloud.utils.nio.Task.run(Task.java:83)
>>      at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>> 1145)
>>      at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>> :615)
>>      at java.lang.Thread.run(Thread.java:744)
>>
>> This is the error on the CS side:
>>
>> 2013-10-29 10:12:58,215 DEBUG [agent.manager.AgentManagerImpl]
>> (catalina-exec-16:null) Details from executing class
>> com.cloud.agent.api.GetVncPortCommand: java.lang.NullPointerException
>>      at
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(Libvirt
>> ComputingResource.java:2711)
>>      at
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(
>> LibvirtComputingResource.java:1216)
>>      at com.cloud.agent.Agent.processRequest(Agent.java:525)
>>      at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
>>      at com.cloud.utils.nio.Task.run(Task.java:83)
>>      at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>> 1145)
>>      at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>> :615)
>>      at java.lang.Thread.run(Thread.java:744)
>>
>> 2013-10-29 10:12:58,215 ERROR [cloud.servlet.ConsoleProxyServlet]
>> (catalina-exec-16:null) Unexepected exception in ConsoleProxyServlet
>> java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast
>> to com.cloud.agent.api.GetVncPortAnswer
>>      at
>> com.cloud.server.ManagementServerImpl.getVncPort(ManagementServerImpl.java
>> :2193)
>>      at
>> com.cloud.servlet.ConsoleProxyServlet.composeConsoleAccessUrl(ConsoleProxy
>> Servlet.java:381)
>>      at
>> com.cloud.servlet.ConsoleProxyServlet.handleAccessRequest(ConsoleProxyServ
>> let.java:269)
>>      at
>> com.cloud.servlet.ConsoleProxyServlet.doGet(ConsoleProxyServlet.java:171)
>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>      at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicati
>> onFilterChain.java:290)
>>      at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilter
>> Chain.java:206)
>>      at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.
>> java:233)
>>      at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.
>> java:191)
>>      at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:1
>> 27)
>>      at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:1
>> 02)
>>      at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
>>      at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.ja
>> va:109)
>>      at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298
>> )
>>      at
>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.jav
>> a:889)
>>      at
>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process
>> (Http11NioProtocol.java:721)
>>      at
>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.jav
>> a:2268)
>>      at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>> 1146)
>>      at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>> :615)
>>      at java.lang.Thread.run(Thread.java:679)
>>
>>
>> On 10/28/2013, 9:05 PM, Kelven Yang wrote:
>>> As long as the container VNC console is reachable from console proxy
>>> agent, I don't see a technical issue here. Need some log details to
>>> understand the situation. For example, what information does LXC
>>> hypervisor resource agent returns on GetVncPort command?
>>>
>>> Kelven
>>>    
>>>
>>> On 10/28/13 10:59 AM, "Francois Gaudreault" <fgaudreault@cloudops.com>
>>> wrote:
>>>
>>>> Thanks! Is the console proxy issue will be fixed at some point in the
>>>> future or this won't be possible?
>>>>
>>>> Francois
>>>>
>>>> On 10/28/2013, 1:57 PM, Phong Nguyen wrote:
>>>>> Yes, the console proxy VM is unable to connect to the LXC console. Use
>>>>> virsh to connect from the host.
>>>>>
>>>>> virsh -c lxc:/// console i-2-20-VM
>>>>>
>>>>> -Phong
>>>>>
>>>>>
>>>>> On Mon, Oct 28, 2013 at 1:22 PM, Francois Gaudreault
>>>>> <fgaudreault@cloudops.com <mailto:fgaudreault@cloudops.com>>
wrote:
>>>>>
>>>>>       Ok, I followed Chiradeep's comment here, and it now works fine.
I
>>>>>       have CentOS 6 containers running.
>>>>>
>>>>>       My only other questions would be:
>>>>>       - Is it normal that the console proxy VM cannot proxy the
>>>>>       container's console?
>>>>>       - How can I access the container console from the host? (I am on
>>>>>       CentOS) I can ssh to the container tho.
>>>>>
>>>>>       Thanks!
>>>>>
>>>>>       Francois
>>>>>
>>>>>
>>>>>       On 10/24/2013, 7:00 PM, Chiradeep Vittal wrote:
>>>>>
>>>>>           So here's what should work
>>>>>           Create zone
>>>>>           Add a KVM cluster -> add a KVM host -> wait for systemvms
to
>>>>> start
>>>>>           Add a LXC cluster -> add a LXC host
>>>>>
>>>>>           On 10/24/13 9:55 AM, "Francois Gaudreault"
>>>>>           <fgaudreault@cloudops.com <mailto:fgaudreault@cloudops.com>>
>>>>>           wrote:
>>>>>
>>>>>               If it's designed to do that, then something is wrong with
>>>>>               how CS deals
>>>>>               with it.
>>>>>
>>>>>               When I was trying to get the KVM images to work, they
>>>>> were
>>>>>               starting,
>>>>>               getting IPs, but then something was killing the VM. I
>>>>>               though for
>>>>>               sometime that libvirt was the issue, so I tried Ubuntu
>>>>>               13.10, 12.04 and
>>>>>               CentOS with the same results. I then switched the
>>>>>               hypervisor type in CS
>>>>>
>>>>>           >from LXC to KVM (rebuilt the zone), keep the same settings
>>>>> on
>>>>>           my host,
>>>>>
>>>>>               and the System VMs are running fine since then.
>>>>>
>>>>>               Anyone have time to help me troubleshoot? I mean, this
is
>>>>>               not a blocker,
>>>>>               but I can't get standalone LXC cluster to work...
>>>>>
>>>>>               Francois
>>>>>
>>>>>               On 10/24/2013, 11:53 AM, Phong Nguyen wrote:
>>>>>
>>>>>                       So we need a KVM cluster to run the VMs? (Added
>>>>>                       the author of the
>>>>>
>>>>>                   feature)
>>>>>
>>>>>                   As it was originally discussed and implemented, the
>>>>>                   decision was to
>>>>>                   use KVM as the system VM for LXC clusters instead of
>>>>>                   creating an LXC
>>>>>                   system VM. A zone with only LXC clusters will deploy
>>>>> a
>>>>>                   KVM system VM
>>>>>                   on a host running an LXC agent. Behind the scenes,
>>>>>                   this is possible
>>>>>                   because both KVM and LXC agents use libvirt for
>>>>>                   provisioning (and that
>>>>>                   the setup of an LXC agent is almost identical to KVM
>>>>>                   and perfectly
>>>>>                   capable of running KVM VMs).
>>>>>
>>>>>                   -Phong
>>>>>
>>>>>
>>>>>                   On Thu, Oct 24, 2013 at 8:57 AM, Francois Gaudreault
>>>>>                   <fgaudreault@cloudops.com
>>>>>                   <mailto:fgaudreault@cloudops.com>
>>>>>                   <mailto:fgaudreault@cloudops.com
>>>>>                   <mailto:fgaudreault@cloudops.com>>> wrote:
>>>>>
>>>>>                        If this is the case, then you should remove the
>>>>>                   ability to create
>>>>>                        LXC zones or clarify the documentation about
>>>>> that.
>>>>>
>>>>>                        According to the wiki page:
>>>>>
>>>>>                        Each of the different hypervisors currently have
>>>>>                   their own System
>>>>>                        VMs. These system VM images are used to run a
>>>>>                   console proxy,
>>>>>                        secondary storage, and router VMs.
>>>>>
>>>>>                        We discussed the possibility of creating System
>>>>>                   VMs for LXC. There
>>>>>                        was concern with the complexity and potential
>>>>>                   issues involving
>>>>>                        iptables for the router inside an LXC container.
>>>>>                   As an
>>>>>                        intermediate solution we are going to use KVM
>>>>>                   System VMs inside
>>>>>                        the LXC Cluster.
>>>>>
>>>>>                        So we need a KVM cluster to run the VMs? (Added
>>>>>                   the author of the
>>>>>                        feature)
>>>>>
>>>>>                        Francois
>>>>>
>>>>>                        On 10/22/2013, 1:24 AM, Chiradeep Vittal wrote:
>>>>>
>>>>>                            As far as I understand, in an LXC scenario,
>>>>>                   the system vms are
>>>>>                            expected to
>>>>>                            run on real hypervisors.
>>>>>                            You can always use the QuickCloud way to not
>>>>>                   use system vms at
>>>>>                            all.
>>>>>
>>>>>                            On 10/21/13 1:45 PM, "Francois Gaudreault"
>>>>>                            <fgaudreault@cloudops.com
>>>>>                   <mailto:fgaudreault@cloudops.com>
>>>>>                   <mailto:fgaudreault@cloudops.com
>>>>>                   <mailto:fgaudreault@cloudops.com>>>
>>>>>                            wrote:
>>>>>
>>>>>                                Ok I think we have to look at this
>>>>>                   further. I'll stop
>>>>>                                hijacking other
>>>>>                                threads.
>>>>>
>>>>>                                I am trying to get the SSVM/CPVM to run
>>>>>                   on a LXC host. The
>>>>>                                SSVM/CPVM
>>>>>                                starts, get IPs, but then CloudStack
>>>>> kill
>>>>>                   them for some
>>>>>                                reason. Yes, I
>>>>>                                use the 4.2 images :
>>>>>
>>>>>                                2013-10-21 16:19:21,605 DEBUG
>>>>>                   [agent.manager.AgentManagerImpl]
>>>>>                                (AgentManager-Handler-9:null) SeqA
>>>>> 73--1:
>>>>>                   Processing Seq
>>>>>                                73--1:  { Cmd ,
>>>>>                                MgmtId: -1, via: 73, Ver: v1, Flags:
>>>>> 111,
>>>>>                
>>>>>                
>>>>>
>>>>> [{"com.cloud.agent.api.ShutdownCommand":{"reason":"sig.kill","wait":0}}
>>>>> ]
>>>>>                                }
>>>>>                                2013-10-21 16:19:21,605 INFO
>>>>>                   [agent.manager.AgentManagerImpl]
>>>>>                                (AgentManager-Handler-9:null) Host 73
>>>>> has
>>>>>                   informed us that
>>>>>                                it is
>>>>>                                shutting down with reason sig.kill and
>>>>>                   detail null
>>>>>                                2013-10-21 16:19:21,606 INFO
>>>>>                   [agent.manager.AgentManagerImpl]
>>>>>                                (AgentTaskPool-11:null) Host 73 is
>>>>>                   disconnecting with event
>>>>>                                ShutdownRequested
>>>>>                                2013-10-21 16:19:21,609 DEBUG
>>>>>                   [agent.manager.AgentManagerImpl]
>>>>>                                (AgentTaskPool-11:null) The next status
>>>>>                   of agent 73is
>>>>>                                Disconnected,
>>>>>                                current status is Up
>>>>>                                2013-10-21 16:19:21,609 DEBUG
>>>>>                   [agent.manager.AgentManagerImpl]
>>>>>                                (AgentTaskPool-11:null) Deregistering
>>>>>                   link for 73 with
>>>>>                                state Disconnected
>>>>>                                2013-10-21 16:19:21,609 DEBUG
>>>>>                   [agent.manager.AgentManagerImpl]
>>>>>                                (AgentTaskPool-11:null) Remove Agent :
>>>>> 73
>>>>>                                2013-10-21 16:19:21,609 DEBUG
>>>>>                                [agent.manager.ConnectedAgentAttache]
>>>>>                                (AgentTaskPool-11:null) Processing
>>>>>                   Disconnect.
>>>>>
>>>>>                                I transferred the host to KVM, and now
>>>>>                   the same SSVM/CPVM
>>>>>                                images are
>>>>>                                running fine for the last 30min ( so I
>>>>>                   assume it works
>>>>>                                fine...).
>>>>>                                Something seems to be wrong with the LXC
>>>>>                   side :S
>>>>>
>>>>>                                Anyone wants to invest some time to
>>>>>                   troubleshoot? I'll
>>>>>                                open a ticket also.
>>>>>
>>>>>                                --
>>>>>                                Francois Gaudreault
>>>>>                                Architecte de Solution Cloud | Cloud
>>>>>                   Solutions Architect
>>>>>                   fgaudreault@cloudops.com
>>>>>                   <mailto:fgaudreault@cloudops.com>
>>>>>                   <mailto:fgaudreault@cloudops.com
>>>>>                   <mailto:fgaudreault@cloudops.com>>
>>>>>                   514-629-6775 <tel:514-629-6775> <tel:514-629-6775
>>>>>                   <tel:514-629-6775>>
>>>>>                                - - -
>>>>>                                CloudOps
>>>>>                                420 rue Guy
>>>>>                                Montréal QC  H3J 1S6
>>>>>                   www.cloudops.com <http://www.cloudops.com>
>>>>>                   <http://www.cloudops.com>
>>>>>                                @CloudOps_
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                        --
>>>>>                        Francois Gaudreault
>>>>>                        Architecte de Solution Cloud | Cloud Solutions
>>>>>                   Architect
>>>>>                   fgaudreault@cloudops.com
>>>>>                   <mailto:fgaudreault@cloudops.com>
>>>>>                   <mailto:fgaudreault@cloudops.com
>>>>>                   <mailto:fgaudreault@cloudops.com>>
>>>>>                   514-629-6775 <tel:514-629-6775> <tel:514-629-6775
>>>>>                   <tel:514-629-6775>>
>>>>>                        - - -
>>>>>                        CloudOps
>>>>>                        420 rue Guy
>>>>>                        Montréal QC  H3J 1S6
>>>>>                   www.cloudops.com <http://www.cloudops.com>
>>>>>                   <http://www.cloudops.com>
>>>>>                        @CloudOps_
>>>>>
>>>>>
>>>>>
>>>>>               --
>>>>>               Francois Gaudreault
>>>>>               Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>               fgaudreault@cloudops.com
>>>>> <mailto:fgaudreault@cloudops.com>
>>>>>               514-629-6775 <tel:514-629-6775>
>>>>>               - - -
>>>>>               CloudOps
>>>>>               420 rue Guy
>>>>>               Montréal QC  H3J 1S6
>>>>>               www.cloudops.com <http://www.cloudops.com>
>>>>>               @CloudOps_
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>       --
>>>>>       Francois Gaudreault
>>>>>       Architecte de Solution Cloud | Cloud Solutions Architect
>>>>>       fgaudreault@cloudops.com <mailto:fgaudreault@cloudops.com>
>>>>>       514-629-6775 <tel:514-629-6775>
>>>>>       - - -
>>>>>       CloudOps
>>>>>       420 rue Guy
>>>>>       Montréal QC  H3J 1S6
>>>>>       www.cloudops.com <http://www.cloudops.com>
>>>>>       @CloudOps_
>>>>>
>>>>>
>>>> -- 
>>>> Francois Gaudreault
>>>> Architecte de Solution Cloud | Cloud Solutions Architect
>>>> fgaudreault@cloudops.com
>>>> 514-629-6775
>>>> - - -
>>>> CloudOps
>>>> 420 rue Guy
>>>> Montréal QC  H3J 1S6
>>>> www.cloudops.com
>>>> @CloudOps_
>>>>
>>
>> -- 
>> Francois Gaudreault
>> Architecte de Solution Cloud | Cloud Solutions Architect
>> fgaudreault@cloudops.com
>> 514-629-6775
>> - - -
>> CloudOps
>> 420 rue Guy
>> Montréal QC  H3J 1S6
>> www.cloudops.com
>> @CloudOps_
>>
>
>


-- 
Francois Gaudreault
Architecte de Solution Cloud | Cloud Solutions Architect
fgaudreault@cloudops.com
514-629-6775
- - -
CloudOps
420 rue Guy
Montréal QC  H3J 1S6
www.cloudops.com
@CloudOps_


Mime
View raw message