cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jayapal Reddy Uradi <jayapalreddy.ur...@citrix.com>
Subject Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
Date Mon, 20 Jul 2015 09:29:58 GMT
Hi Sonali,

In our lab setup this issue is not reproduced.
I am trying to understand how NETWORK_STATS is missed in VR. This can be missed if the VR
is rebooted outside the cloudstack. If this is not the case some thing might removed the rule.

Can you please recollect to understand how this chain is removed.

Thanks,
Jayapal




On 06-Jul-2015, at 1:44 PM, Sonali Jadhav <sonali@servercentralen.se> wrote:

> Hi,
> 
> I had updated acs from 4.4.2 to 4.5.1 three days back. I successfully executed cmd "
nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1
&"  to update VRs.
> 
> I also cross checked from ACS, version of each VR. It was 4.5.
> 
> But I was seeing these errors in management logs from few VRs. so like you said I observed
in those VR, NETWORK_STATS chain was missing. I just rebooted them and then I was able to
see that chain in iptables. So I rebooted all those VRs with error logs. And now it seems
to be ok.  
> 
> /Sonali
> 
> -----Original Message-----
> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com] 
> Sent: Monday, July 6, 2015 12:45 PM
> To: <users@cloudstack.apache.org>
> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
> 
> Hi Sonali,
> 
> Can you give me some context on how this issue is seen. I want to understand when these
chains got removed in VR.
> 
> Thanks,
> Jayapal
> 
> On 06-Jul-2015, at 12:08 PM, Sonali Jadhav <sonali@servercentralen.se> wrote:
> 
>> Sorry, error log was not for all VRs but some. 
>> 
>> I don't see NETWORK_STATS chains for those. So, I just rebooted, and then there it
was. I am suspecting VR was not updated ? weird but I was able to see version 4.5 of all VRs
in ACS. 
>> I'll keep watch if error comes again.
>> 
>> Thanks for help.
>> 
>> /Sonali
>> 
>> -----Original Message-----
>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com] 
>> Sent: Monday, July 6, 2015 11:00 AM
>> To: <users@cloudstack.apache.org>
>> Subject: Re: error logs for all VRs after upgrading ACS from 4.4.2 to 4.5.1
>> 
>> 
>> It seems VR does not have the usage related iptables chains.
>> Please check the 'iptables -L -nv' for NETWORK_STATS chains.
>> These rules supposed to be configured on VR start.
>> 
>> Thanks,
>> Jayapal
>> 
>> 
>> 
>> On 06-Jul-2015, at 10:48 AM, Sonali Jadhav <sonali@servercentralen.se<mailto:sonali@servercentralen.se>>
>> wrote:
>> 
>> Hi,
>> 
>> After I upgraded ACS from 4.4.2 to 4.5.1, I am seeing following error in management
server logs, for each and every VR,
>> 
>> 
>> 2015-07-06 07:08:20,721 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d)
Begin cleanup expired async-jobs
>> 2015-07-06 07:08:20,733 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-415ba06d)
End cleanup expired async-jobs
>> 2015-07-06 07:08:20,811 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d)
External devices stats collector is running...
>> 2015-07-06 07:08:20,833 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d)
Network 204 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,835 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d)
Network 219 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,838 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d)
Network 220 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,846 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d)
Network 211 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,851 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d)
Network 212 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,853 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d)
Network 213 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,855 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d)
Network 222 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,858 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d)
Network 223 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,884 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d)
Network 215 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,887 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-91d5dc5d)
Found 10 running routers.
>> 2015-07-06 07:08:20,893 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d)
Network 216 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,894 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f)
Found 11 routers to update status.
>> 2015-07-06 07:08:20,898 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-50b5b42f)
Found 0 networks to update RvR status.
>> 2015-07-06 07:08:20,899 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-8a22f14d)
Network 217 is not configured for external networking, so skipping usage check.
>> 2015-07-06 07:08:20,910 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018)
Seq 5-6790865288121300428: Executing request
>> 2015-07-06 07:08:20,915 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-284:ctx-0e4c6018)
Executing command in VR: /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.2.197 -g
>> 2015-07-06 07:08:21,041 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6)
Snapshot scheduler.poll is being called at 2015-07-06 05:08:21 GMT
>> 2015-07-06 07:08:21,043 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-60440aa6)
Got 0 snapshots to be executed at 2015-07-06 05:08:21 GMT
>> 2015-07-06 07:08:21,337 WARN  [c.c.h.x.r.XenServer56Resource] (DirectAgent-284:ctx-0e4c6018)
Failed to get network usage stats due to
>> java.lang.NumberFormatException: For input string: "iptables"
>>              at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>>              at java.lang.Long.parseLong(Long.java:441)
>>              at java.lang.Long.<init>(Long.java:702)
>>              at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
>>              at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
>>              at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
>>              at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
>>              at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
>>              at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
>>              at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>              at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>              at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>              at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>              at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>              at java.lang.Thread.run(Thread.java:744)
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-284:ctx-0e4c6018)
Seq 5-6790865288121300428: Response Received:
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (DirectAgent-284:ctx-0e4c6018) Seq
5-6790865288121300428: Processing:  { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags:
10, [{"com.cloud.agent.api.NetworkUsageAnswer":{"result":false,"details":"Exception: java.lang.NumberFormatException\nMessage:
For input string: \"iptables\"\nStack: java.lang.NumberFormatException: For input string:
\"iptables\"\n\tat java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n\tat
java.lang.Long.parseLong(Long.java:441)\n\tat java.lang.Long.<init>(Long.java:702)\n\tat
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)\n\tat
com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)\n\tat
com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)\n\tat
com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)\n\tat
com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)\n\tat
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)\n\tat
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)\n\tat
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat
java.lang.Thread.run(Thread.java:744)\n","wait":0}}] }
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.t.Request] (RouterMonitor-1:ctx-91d5dc5d) Seq
5-6790865288121300428: Received:  { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags:
10, { NetworkUsageAnswer } }
>> 2015-07-06 07:08:21,339 DEBUG [c.c.a.m.AgentManagerImpl] (RouterMonitor-1:ctx-91d5dc5d)
Details from executing class com.cloud.agent.api.NetworkUsageCommand: Exception: java.lang.NumberFormatException
>> Message: For input string: "iptables"
>> Stack: java.lang.NumberFormatException: For input string: "iptables"
>>              at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>>              at java.lang.Long.parseLong(Long.java:441)
>>              at java.lang.Long.<init>(Long.java:702)
>>              at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.getNetworkStats(CitrixResourceBase.java:2225)
>>              at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.execute(XenServer56Resource.java:204)
>>              at com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:62)
>>              at com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87)
>>              at com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65)
>>              at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:302)
>>              at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>              at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>              at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>              at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>              at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>              at java.lang.Thread.run(Thread.java:744)
>> 
>> what exactly is this error?
>> 
>> /sonali
>> 
> 


Mime
View raw message