cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Kudryavtsev <kudryavtsev...@bw-sw.com>
Subject Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
Date Sat, 11 Nov 2017 15:59:21 GMT
Hello, I implemented some hotfixes for 4.10 to work

https://github.com/apache/cloudstack/pull/2319 - to master (load
br_netfilter module)
https://github.com/apache/cloudstack/pull/2320 - to 4.10 which fixes SG
failures related to ipv6.


2017-11-11 15:51 GMT+07:00 Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.com>:

> Following up with previous question. I managed to make it work by removing
> all and heading to ubuntu 14.04 hypervisor host.
>
> Also, what I found more:
>
> 1. when setup databases (management server) if custom port is specified,
> databases themself is not created. If create manually, import scripts work
> fine.
> 2. UI: unable to download ISO to __all__ zones. Have to specify certain
> zone, else UI gives an error.
> 3. Ubuntu doesn't load module *br_netfilter* but
>
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>
> uses it and nothing good as a result:
>
> 2017-11-11 15:38:29,241 - sysctl -w net.bridge.bridge-nf-call-arptables=1
> 2017-11-11 15:38:29,244 - sysctl -w net.bridge.bridge-nf-call-iptables=1
> 2017-11-11 15:38:29,247 - sysctl -w net.bridge.bridge-nf-call-ip6tables=1
>
> adding br_netfilter to /etc/modules fixes it. I suppose it's necessary to
> add
>
> in script something like: modprobe br_netfilter (or smarter thing).
>
> But It doesn't work completely, actually, security groups are unfunctional:
>
> ==> /var/log/cloudstack/agent/agent.log <==
> 2017-11-11 15:40:41,326 WARN  [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) (logid:eab9a328) Exception:
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> add_network_rules --vmname i-2-7-VM --vmid 7 --vmip 176.120.28.4 --vmip6
> null --sig d60255deb618b7be9f477eed10d65234 --seq 4 --vmmac
> 1e:00:6f:00:01:01 --vif vnet8 --brname cloudbr0 --nicsecips 0: --rules
> I:icmp:-1:-1:0.0.0.0/0,NEXT;I:tcp:1:65535:0.0.0.0/0,NEXT;I:
> udp:1:65535:0.0.0.0/0,NEXT;E:icmp:-1:-1:0.0.0.0/0,NEXT;E:
> tcp:1:65535:0.0.0.0/0,NEXT;E:udp:1:65535:0.0.0.0/0,NEXT;
> java.lang.NullPointerException
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
> at com.cloud.utils.script.Script.execute(Script.java:214)
> at com.cloud.utils.script.Script.execute(Script.java:182)
> at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> addNetworkRules(LibvirtComputingResource.java:3429)
> at com.cloud.hypervisor.kvm.resource.wrapper.
> LibvirtSecurityGroupRulesCommandWrapper.execute(
> LibvirtSecurityGroupRulesCommandWrapper.java:57)
> at com.cloud.hypervisor.kvm.resource.wrapper.
> LibvirtSecurityGroupRulesCommandWrapper.execute(
> LibvirtSecurityGroupRulesCommandWrapper.java:36)
> at com.cloud.hypervisor.kvm.resource.wrapper.
> LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
> at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.
> executeRequest(LibvirtComputingResource.java:1369)
> at com.cloud.agent.Agent.processRequest(Agent.java:525)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:833)
> at com.cloud.utils.nio.Task.call(Task.java:83)
> at com.cloud.utils.nio.Task.call(Task.java:29)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2017-11-11 15:40:41,327 WARN  [resource.wrapper.
> LibvirtSecurityGroupRulesCommandWrapper] (agentRequest-Handler-2:null)
> (logid:eab9a328) Failed to program network rules for vm i-2-7-VM
>
> So, no rules are actually created. Script doesn't call... I suppose may be
> quotes are required because shell interprets ';' as command separator. I
> suppose that optimization introduced in 4.10, because in 4.9 SGs work like
> a charm...
>
>
> 2017-11-11 3:15 GMT+07:00 Paul Angus <paul.angus@shapeblue.com>:
>
>> Ivan,
>>
>> Can you paste a larger section of unfiltered logs.  There would always be
>> a message explaining why the mgmt. server thought that a VR should be shut
>> down
>>
>>
>>
>> Kind regards,
>>
>> Paul Angus
>>
>> paul.angus@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>> -----Original Message-----
>> From: Simon Weller [mailto:sweller@ena.com.INVALID]
>> Sent: 10 November 2017 18:39
>> To: dev@cloudstack.apache.org
>> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>>
>> What VR template image are you using?
>>
>>
>> ________________________________
>> From: Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.com>
>> Sent: Friday, November 10, 2017 11:59 AM
>> To: dev@cloudstack.apache.org
>> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>>
>> Hi. No, regular NFS. VR starts great, but stopped by ms, other system vms
>> are working. I even added to communication script on compute node "sleep
>> 3600" before ssh, so response to management is delayed, I logged so to VR,
>> all interfaces are up, iptables rules are OK.
>>
>> So agent rolls vr good, but stops it by management order with no obvious
>> reason.
>>
>> 11 нояб. 2017 г. 0:54 пользователь "Simon Weller" <sweller@ena.com.invalid
>> >
>> написал:
>>
>> > Is the storage ceph?
>> >
>> >
>> > ________________________________
>> > From: Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.com>
>> > Sent: Friday, November 10, 2017 11:52 AM
>> > To: dev@cloudstack.apache.org
>> > Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>> >
>> > Hi, I did, and it does the things right, I even added "tee" to ssh
>> > 3922 communication script to out vr response to additional log and it
>> > only receives VR version line and sends all info (the same from
>> > pastebin) to ACS and receives "stop" order.
>> >
>> > I'll try to provide additional info, but ad you can see, management
>> > receives proper response and sends stop next op. It looks very freaky
>> > without any notification...
>> >
>> > 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"
>> > <sweller@ena.com.invalid
>> > >
>> > написал:
>> >
>> > > Ivan,
>> > >
>> > >
>> > > Can you put the host agents into debug mode? Hopefully that will
>> > > provide more information.
>> > >
>> > >
>> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+deb
>> > > ug
>> KVM agent debug - Apache Cloudstack - Apache Software ...<
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug>
>> cwiki.apache.org
>> Steps to debug the KVM agent from eclipse: In KVM agent edit
>> '/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_socket,address=8787
>> ...
>>
>>
>>
>> > >
>> > >
>> > > - Si
>> > >
>> > > ________________________________
>> > > From: Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.com>
>> > > Sent: Friday, November 10, 2017 11:34 AM
>> > > To: dev@cloudstack.apache.org
>> > > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>> > >
>> > > Hello, Devs.
>> > >
>> > > I experience VR Start Problem in the fresh ACS 4.10 deployment
>> > >
>> > > Intersting place of logs is here: https://pastebin.com/iBXRBA5N
>> [https://pastebin.com/i/facebook.png]<https://pastebin.com/iBXRBA5N>
>>
>> 2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com<https://pastebin.
>> com/iBXRBA5N>
>> pastebin.com
>>
>>
>>
>> > >
>> > > Basically, the situation looks like:
>> > >
>> > > 1. Management Server tries to launch VR 2. It gets from Agent proper
>> > > VR response with VR details 3. It sends StopCommand without
>> > > explanation.
>> > >
>> > > I'm trying to figure out what happens inside, but the codebase is
>> > > huge
>> > and
>> > > still no positive results. Please, let me know if you have any ideas
>> > which
>> > > could help me finding the reason. Thanks a lot.
>> > >
>> > > --
>> > > With best regards, Ivan Kudryavtsev
>> > > Bitworks Software, Ltd.
>> > > Cell: +7-923-414-1515
>> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
>> > >
>> >
>>
>>
>
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>
>
>


-- 
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message