cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "venkata swamybabu budumuru (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CLOUDSTACK-2761) [VMware] [VPC] Failed to create PF/StaticNAT rules on VPC
Date Thu, 30 May 2013 11:12:21 GMT
venkata swamybabu budumuru created CLOUDSTACK-2761:
------------------------------------------------------

             Summary: [VMware] [VPC] Failed to create PF/StaticNAT rules on VPC
                 Key: CLOUDSTACK-2761
                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2761
             Project: CloudStack
          Issue Type: Bug
      Security Level: Public (Anyone can view this level - this is the default.)
          Components: Network Controller
    Affects Versions: 4.2.0
         Environment: commit # 8d1189c2ae87216bc1c4a1443f75e9a8629abdc2
            Reporter: venkata swamybabu budumuru
            Priority: Blocker
             Fix For: 4.2.0


Steps to reproduce:

1. Have latest CloudStack build with at least 1 advanced zone
2. Have at least 1 VMware cluster with 1 host.
3. Create a VPC with at least 1 Tier
4. deploy a VM
5. apply an ACL to allow all 
6. acquire at least 1 IP
7. create PF/staticNAT on the above created IP to the VM created in step (4) 

Observations:

(i) It failed to create staticNAT with the following error


2013-05-30 08:05:20,203 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-38:null) Seq
1-1416495168: Response Received:
2013-05-30 08:05:20,206 DEBUG [agent.transport.Request] (DirectAgent-38:null) Seq 1-1416495168:
Processing:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 0, [{"routing.IpAssocAnswer":{"results":["10.147.44.63
- success"],"result":true,"wait":0}}] }
2013-05-30 08:05:20,207 DEBUG [agent.transport.Request] (catalina-exec-19:null) Seq 1-1416495168:
Received:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 0, { IpAssocAnswer } }
2013-05-30 08:05:20,211 INFO  [cloud.network.NetworkManagerImpl] (catalina-exec-19:null) Let
VpcVirtualRouter handle StaticNat in network 204
2013-05-30 08:05:20,223 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (catalina-exec-19:null)
Applying static nat rules in network Ntwk[204|Guest|11]
2013-05-30 08:05:20,288 DEBUG [agent.transport.Request] (catalina-exec-19:null) Seq 1-1416495169:
Sending  { Cmd , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 100001, [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
}
2013-05-30 08:05:20,288 DEBUG [agent.transport.Request] (catalina-exec-19:null) Seq 1-1416495169:
Executing:  { Cmd , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 100001, [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
}
2013-05-30 08:05:20,289 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-16:null) Seq
1-1416495169: Executing request
2013-05-30 08:05:20,290 INFO  [vmware.resource.VmwareResource] (DirectAgent-16:10.147.40.12)
Executing resource SetFirewallRuleCommand: {"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}
2013-05-30 08:05:20,290 DEBUG [vmware.resource.VmwareResource] (DirectAgent-16:10.147.40.12)
Use router's private IP for SSH control. IP : 10.147.40.62
2013-05-30 08:05:21,746 ERROR [utils.ssh.SshHelper] (DirectAgent-16:10.147.40.12) SSH execution
of command /root/firewall.sh  -A  -l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G  has an error status
code in return. result output: Bad argument `10.147.44.63'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `10.147.44.63'
Try `iptables -h' or 'iptables --help' for more information.
iptables v1.4.14: option "--set-mark" requires an argument
Try `iptables -h' or 'iptables --help' for more information.
iptables: No chain/target/match by that name.
Bad argument `10.147.44.63'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `eth0'
Try `iptables -h' or 'iptables --help' for more information.
Bad argument `10.147.44.63'
Try `iptables -h' or 'iptables --help' for more information.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
Bad argument `10.147.44.63'
Try `iptables -h' or 'iptables --help' for more information.

2013-05-30 08:05:21,790 DEBUG [vmware.resource.VmwareResource] (DirectAgent-16:10.147.40.12)
Executing script on domain router 10.147.40.62: /root/firewall.sh  -A  -l 10.147.44.63 -r
10.0.1.188 -d 0:0 -G

(ii) After changing the firewall.sh as mentioned below, it went fine.

get_dev_list() {
  ip link show | grep -e eth[2-9] | awk -F ":" '{print $2}'
  ip link show | grep -e eth1[0-9] | awk -F ":" '{print $2}'
}

Changed the above with the following to include eth1 device as well.

get_dev_list() {
  ip link show | grep -e eth[1-9] | awk -F ":" '{print $2}'
  ip link show | grep -e eth1[0-9] | awk -F ":" '{print $2}'
}
 
2013-05-30 08:32:52,492 INFO  [cloud.network.NetworkManagerImpl] (catalina-exec-3:null) Let
VpcVirtualRouter handle StaticNat in network 204
2013-05-30 08:32:52,506 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (catalina-exec-3:null)
Applying static nat rules in network Ntwk[204|Guest|11]
2013-05-30 08:32:52,523 DEBUG [agent.transport.Request] (catalina-exec-3:null) Seq 1-1416495239:
Sending  { Cmd , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 100001, [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
}
2013-05-30 08:32:52,524 DEBUG [agent.transport.Request] (catalina-exec-3:null) Seq 1-1416495239:
Executing:  { Cmd , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 100001, [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
}
2013-05-30 08:32:52,524 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-12:null) Seq
1-1416495239: Executing request
2013-05-30 08:32:52,525 INFO  [vmware.resource.VmwareResource] (DirectAgent-12:10.147.40.12)
Executing resource SetFirewallRuleCommand: {"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}
2013-05-30 08:32:52,529 DEBUG [vmware.resource.VmwareResource] (DirectAgent-12:10.147.40.12)
Use router's private IP for SSH control. IP : 10.147.40.62
2013-05-30 08:32:53,937 DEBUG [vmware.resource.VmwareResource] (DirectAgent-12:10.147.40.12)
Executing script on domain router 10.147.40.62: /root/firewall.sh  -A  -l 10.147.44.63 -r
10.0.1.188 -d 0:0 -G
2013-05-30 08:32:53,938 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-12:null) Seq
1-1416495239: Response Received:
2013-05-30 08:32:53,938 DEBUG [agent.transport.Request] (DirectAgent-12:null) Seq 1-1416495239:
Processing:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 0, [{"routing.SetStaticNatRulesAnswer":{"results":[null],"result":true,"wait":0}}]
}


(iii) Tried to access the staticNat rule but, it failed because, it created a firewall rule
with DROP policy which is not required in case of staticNAt. Ideally instead of firewall.sh,
it should call "vlc_staticnat.sh" script to configure staticnat.

Here is the snippet of iptable output.


root@r-3-VM:~# iptables -L -nv -t mangle
Chain PREROUTING (policy ACCEPT 3060 packets, 346K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    1    60 CONNMARK   all  --  eth1   *       0.0.0.0/0            0.0.0.0/0            state
NEW CONNMARK set 0x1
 2360 3256K FIREWALL_10.147.44.63  all  --  *      *       0.0.0.0/0            10.147.44.63
       
 1388 77740 CONNMARK   all  --  eth2   *       0.0.0.0/0            0.0.0.0/0            state
RELATED,ESTABLISHED CONNMARK restore
   27  1620 ACL_OUTBOUND_eth2  all  --  eth2   *       10.0.1.0/24         !10.0.1.1     
       state NEW
    0     0 MARK       all  --  eth1   *       0.0.0.0/0            10.147.44.63         state
NEW MARK set 0x1
    0     0 CONNMARK   all  --  eth1   *       0.0.0.0/0            10.147.44.63         state
NEW CONNMARK save
    0     0 MARK       all  --  eth0   *       10.0.1.188           0.0.0.0/0            state
NEW MARK set 0x1
    0     0 CONNMARK   all  --  eth0   *       10.0.1.188           0.0.0.0/0            state
NEW CONNMARK save

Chain INPUT (policy ACCEPT 3048 packets, 345K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 21 packets, 10108 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 3774 3335K VPN_STATS_eth1  all  --  *      *       0.0.0.0/0            0.0.0.0/0       
   

Chain OUTPUT (policy ACCEPT 2402 packets, 384K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CHECKSUM   udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp
dpt:68 CHECKSUM fill

Chain POSTROUTING (policy ACCEPT 2423 packets, 395K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain ACL_OUTBOUND_eth2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   27  1620 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FIREWALL_10.147.44.63 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 2359 3256K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state
RELATED,ESTABLISHED
    1    60 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain VPN_STATS_eth1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0            all  --  *      eth1    0.0.0.0/0            0.0.0.0/0            mark
match 0x525
    0     0            all  --  eth1   *       0.0.0.0/0            0.0.0.0/0            mark
match 0x524

(iv) Tried to configure PF rule but that as well failed but with a different error.

here is the snippet from mgmt server log.


2013-05-30 11:30:35,264 DEBUG [vmware.resource.VmwareResource] (DirectAgent-495:10.147.40.12)
Use router's private IP for SSH control. IP : 10.147.40.62
2013-05-30 11:30:36,548 ERROR [utils.ssh.SshHelper] (DirectAgent-495:10.147.40.12) SSH execution
of command /opt/cloud/bin/vpc_portforwarding.sh  -A -P tcp -l 10.147.44.64 -p 22-22 -r 10.0.1.188
-d 22-22 has an error status code in return. result output:
2013-05-30 11:30:36,555 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-495:null) Seq
1-1416495677: Response Received:
2013-05-30 11:30:36,556 DEBUG [agent.transport.Request] (DirectAgent-495:null) Seq 1-1416495677:
Processing:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 0, [{"routing.SetPortForwardingRulesAnswer":{"results":["Failed"],"result":false,"wait":0}}]
}
2013-05-30 11:30:36,556 DEBUG [agent.transport.Request] (Job-Executor-26:job-23) Seq 1-1416495677:
Received:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 0, { SetPortForwardingRulesAnswer
} }
2013-05-30 11:30:36,556 WARN  [network.rules.RulesManagerImpl] (Job-Executor-26:job-23) Failed
to apply port forwarding rules for ip due to
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable:
Unable to apply firewall rules on router
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3739)
        at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:3567)
        at com.cloud.network.element.VirtualRouterElement.applyPFRules(VirtualRouterElement.java:787)
        at com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:565)
        at com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:2913)
        at com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:509)
        at com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRules(RulesManagerImpl.java:889)
        at com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRules(RulesManagerImpl.java:1072)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
        at org.apache.cloudstack.api.command.user.firewall.CreatePortForwardingRuleCmd.execute(CreatePortForwardingRuleCmd.java:184)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
        at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:679)

(v) Tried the following iptables command manually and found an error with syntax

root@r-3-VM:~# sudo iptables -t nat -A PREROUTING --proto tcp -d 10.147.44.64 --destination-port
22-22 -j DNAT --to-destination 10.0.1.188:22-22
iptables v1.4.14: invalid port/service `22-22' specified
Try `iptables -h' or 'iptables --help' for more information.

Attaching all the mgmt server logs to the bug.

Here is the system vm template version that I used for testing.

root@r-3-VM:~# cat /etc/cloudstack-release
Cloudstack Release 4.2.0 Tue Apr 16 04:09:58 UTC 2013

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message