cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jayapal Reddy (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (CLOUDSTACK-2761) [VMware] [VPC] Failed to create PF/StaticNAT rules on VPC
Date Mon, 03 Jun 2013 10:02:19 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jayapal Reddy reassigned CLOUDSTACK-2761:
-----------------------------------------

    Assignee: Jayapal Reddy
    
> [VMware] [VPC] Failed to create PF/StaticNAT rules on VPC
> ---------------------------------------------------------
>
>                 Key: CLOUDSTACK-2761
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2761
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Network Controller
>    Affects Versions: 4.2.0
>         Environment: commit # 8d1189c2ae87216bc1c4a1443f75e9a8629abdc2
>            Reporter: venkata swamybabu budumuru
>            Assignee: Jayapal Reddy
>            Priority: Blocker
>             Fix For: 4.2.0
>
>         Attachments: logs.tgz
>
>
> Steps to reproduce:
> 1. Have latest CloudStack build with at least 1 advanced zone
> 2. Have at least 1 VMware cluster with 1 host.
> 3. Create a VPC with at least 1 Tier
> 4. deploy a VM
> 5. apply an ACL to allow all 
> 6. acquire at least 1 IP
> 7. create PF/staticNAT on the above created IP to the VM created in step (4) 
> Observations:
> (i) It failed to create staticNAT with the following error
> 2013-05-30 08:05:20,203 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-38:null)
Seq 1-1416495168: Response Received:
> 2013-05-30 08:05:20,206 DEBUG [agent.transport.Request] (DirectAgent-38:null) Seq 1-1416495168:
Processing:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 0, [{"routing.IpAssocAnswer":{"results":["10.147.44.63
- success"],"result":true,"wait":0}}] }
> 2013-05-30 08:05:20,207 DEBUG [agent.transport.Request] (catalina-exec-19:null) Seq 1-1416495168:
Received:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 0, { IpAssocAnswer } }
> 2013-05-30 08:05:20,211 INFO  [cloud.network.NetworkManagerImpl] (catalina-exec-19:null)
Let VpcVirtualRouter handle StaticNat in network 204
> 2013-05-30 08:05:20,223 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (catalina-exec-19:null)
Applying static nat rules in network Ntwk[204|Guest|11]
> 2013-05-30 08:05:20,288 DEBUG [agent.transport.Request] (catalina-exec-19:null) Seq 1-1416495169:
Sending  { Cmd , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 100001, [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
}
> 2013-05-30 08:05:20,288 DEBUG [agent.transport.Request] (catalina-exec-19:null) Seq 1-1416495169:
Executing:  { Cmd , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 100001, [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
}
> 2013-05-30 08:05:20,289 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-16:null)
Seq 1-1416495169: Executing request
> 2013-05-30 08:05:20,290 INFO  [vmware.resource.VmwareResource] (DirectAgent-16:10.147.40.12)
Executing resource SetFirewallRuleCommand: {"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}
> 2013-05-30 08:05:20,290 DEBUG [vmware.resource.VmwareResource] (DirectAgent-16:10.147.40.12)
Use router's private IP for SSH control. IP : 10.147.40.62
> 2013-05-30 08:05:21,746 ERROR [utils.ssh.SshHelper] (DirectAgent-16:10.147.40.12) SSH
execution of command /root/firewall.sh  -A  -l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G  has an
error status code in return. result output: Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> iptables v1.4.14: option "--set-mark" requires an argument
> Try `iptables -h' or 'iptables --help' for more information.
> iptables: No chain/target/match by that name.
> Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> Bad argument `eth0'
> Try `iptables -h' or 'iptables --help' for more information.
> Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> iptables: No chain/target/match by that name.
> iptables: No chain/target/match by that name.
> Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> 2013-05-30 08:05:21,790 DEBUG [vmware.resource.VmwareResource] (DirectAgent-16:10.147.40.12)
Executing script on domain router 10.147.40.62: /root/firewall.sh  -A  -l 10.147.44.63 -r
10.0.1.188 -d 0:0 -G
> (ii) After changing the firewall.sh as mentioned below, it went fine.
> get_dev_list() {
>   ip link show | grep -e eth[2-9] | awk -F ":" '{print $2}'
>   ip link show | grep -e eth1[0-9] | awk -F ":" '{print $2}'
> }
> Changed the above with the following to include eth1 device as well.
> get_dev_list() {
>   ip link show | grep -e eth[1-9] | awk -F ":" '{print $2}'
>   ip link show | grep -e eth1[0-9] | awk -F ":" '{print $2}'
> }
>  
> 2013-05-30 08:32:52,492 INFO  [cloud.network.NetworkManagerImpl] (catalina-exec-3:null)
Let VpcVirtualRouter handle StaticNat in network 204
> 2013-05-30 08:32:52,506 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (catalina-exec-3:null)
Applying static nat rules in network Ntwk[204|Guest|11]
> 2013-05-30 08:32:52,523 DEBUG [agent.transport.Request] (catalina-exec-3:null) Seq 1-1416495239:
Sending  { Cmd , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 100001, [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
}
> 2013-05-30 08:32:52,524 DEBUG [agent.transport.Request] (catalina-exec-3:null) Seq 1-1416495239:
Executing:  { Cmd , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 100001, [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
}
> 2013-05-30 08:32:52,524 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-12:null)
Seq 1-1416495239: Executing request
> 2013-05-30 08:32:52,525 INFO  [vmware.resource.VmwareResource] (DirectAgent-12:10.147.40.12)
Executing resource SetFirewallRuleCommand: {"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}
> 2013-05-30 08:32:52,529 DEBUG [vmware.resource.VmwareResource] (DirectAgent-12:10.147.40.12)
Use router's private IP for SSH control. IP : 10.147.40.62
> 2013-05-30 08:32:53,937 DEBUG [vmware.resource.VmwareResource] (DirectAgent-12:10.147.40.12)
Executing script on domain router 10.147.40.62: /root/firewall.sh  -A  -l 10.147.44.63 -r
10.0.1.188 -d 0:0 -G
> 2013-05-30 08:32:53,938 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-12:null)
Seq 1-1416495239: Response Received:
> 2013-05-30 08:32:53,938 DEBUG [agent.transport.Request] (DirectAgent-12:null) Seq 1-1416495239:
Processing:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 0, [{"routing.SetStaticNatRulesAnswer":{"results":[null],"result":true,"wait":0}}]
}
> (iii) Tried to access the staticNat rule but, it failed because, it created a firewall
rule with DROP policy which is not required in case of staticNAt. Ideally instead of firewall.sh,
it should call "vlc_staticnat.sh" script to configure staticnat.
> Here is the snippet of iptable output.
> root@r-3-VM:~# iptables -L -nv -t mangle
> Chain PREROUTING (policy ACCEPT 3060 packets, 346K bytes)
>  pkts bytes target     prot opt in     out     source               destination     
   
>     1    60 CONNMARK   all  --  eth1   *       0.0.0.0/0            0.0.0.0/0       
    state NEW CONNMARK set 0x1
>  2360 3256K FIREWALL_10.147.44.63  all  --  *      *       0.0.0.0/0            10.147.44.63
       
>  1388 77740 CONNMARK   all  --  eth2   *       0.0.0.0/0            0.0.0.0/0       
    state RELATED,ESTABLISHED CONNMARK restore
>    27  1620 ACL_OUTBOUND_eth2  all  --  eth2   *       10.0.1.0/24         !10.0.1.1
            state NEW
>     0     0 MARK       all  --  eth1   *       0.0.0.0/0            10.147.44.63    
    state NEW MARK set 0x1
>     0     0 CONNMARK   all  --  eth1   *       0.0.0.0/0            10.147.44.63    
    state NEW CONNMARK save
>     0     0 MARK       all  --  eth0   *       10.0.1.188           0.0.0.0/0       
    state NEW MARK set 0x1
>     0     0 CONNMARK   all  --  eth0   *       10.0.1.188           0.0.0.0/0       
    state NEW CONNMARK save
> Chain INPUT (policy ACCEPT 3048 packets, 345K bytes)
>  pkts bytes target     prot opt in     out     source               destination     
   
> Chain FORWARD (policy ACCEPT 21 packets, 10108 bytes)
>  pkts bytes target     prot opt in     out     source               destination     
   
>  3774 3335K VPN_STATS_eth1  all  --  *      *       0.0.0.0/0            0.0.0.0/0  
        
> Chain OUTPUT (policy ACCEPT 2402 packets, 384K bytes)
>  pkts bytes target     prot opt in     out     source               destination     
   
>     0     0 CHECKSUM   udp  --  *      *       0.0.0.0/0            0.0.0.0/0       
    udp dpt:68 CHECKSUM fill
> Chain POSTROUTING (policy ACCEPT 2423 packets, 395K bytes)
>  pkts bytes target     prot opt in     out     source               destination     
   
> Chain ACL_OUTBOUND_eth2 (1 references)
>  pkts bytes target     prot opt in     out     source               destination     
   
>    27  1620 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0       
   
>     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0       
   
> Chain FIREWALL_10.147.44.63 (1 references)
>  pkts bytes target     prot opt in     out     source               destination     
   
>  2359 3256K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0       
    state RELATED,ESTABLISHED
>     1    60 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0       
   
> Chain VPN_STATS_eth1 (1 references)
>  pkts bytes target     prot opt in     out     source               destination     
   
>     0     0            all  --  *      eth1    0.0.0.0/0            0.0.0.0/0       
    mark match 0x525
>     0     0            all  --  eth1   *       0.0.0.0/0            0.0.0.0/0       
    mark match 0x524
> (iv) Tried to configure PF rule but that as well failed but with a different error.
> here is the snippet from mgmt server log.
> 2013-05-30 11:30:35,264 DEBUG [vmware.resource.VmwareResource] (DirectAgent-495:10.147.40.12)
Use router's private IP for SSH control. IP : 10.147.40.62
> 2013-05-30 11:30:36,548 ERROR [utils.ssh.SshHelper] (DirectAgent-495:10.147.40.12) SSH
execution of command /opt/cloud/bin/vpc_portforwarding.sh  -A -P tcp -l 10.147.44.64 -p 22-22
-r 10.0.1.188 -d 22-22 has an error status code in return. result output:
> 2013-05-30 11:30:36,555 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-495:null)
Seq 1-1416495677: Response Received:
> 2013-05-30 11:30:36,556 DEBUG [agent.transport.Request] (DirectAgent-495:null) Seq 1-1416495677:
Processing:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 0, [{"routing.SetPortForwardingRulesAnswer":{"results":["Failed"],"result":false,"wait":0}}]
}
> 2013-05-30 11:30:36,556 DEBUG [agent.transport.Request] (Job-Executor-26:job-23) Seq
1-1416495677: Received:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: v1, Flags: 0, { SetPortForwardingRulesAnswer
} }
> 2013-05-30 11:30:36,556 WARN  [network.rules.RulesManagerImpl] (Job-Executor-26:job-23)
Failed to apply port forwarding rules for ip due to
> com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable:
Unable to apply firewall rules on router
>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3739)
>         at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:3567)
>         at com.cloud.network.element.VirtualRouterElement.applyPFRules(VirtualRouterElement.java:787)
>         at com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:565)
>         at com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:2913)
>         at com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:509)
>         at com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRules(RulesManagerImpl.java:889)
>         at com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRules(RulesManagerImpl.java:1072)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>         at org.apache.cloudstack.api.command.user.firewall.CreatePortForwardingRuleCmd.execute(CreatePortForwardingRuleCmd.java:184)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
>         at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:679)
> (v) Tried the following iptables command manually and found an error with syntax
> root@r-3-VM:~# sudo iptables -t nat -A PREROUTING --proto tcp -d 10.147.44.64 --destination-port
22-22 -j DNAT --to-destination 10.0.1.188:22-22
> iptables v1.4.14: invalid port/service `22-22' specified
> Try `iptables -h' or 'iptables --help' for more information.
> Attaching all the mgmt server logs to the bug.
> Here is the system vm template version that I used for testing.
> root@r-3-VM:~# cat /etc/cloudstack-release
> Cloudstack Release 4.2.0 Tue Apr 16 04:09:58 UTC 2013

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message