cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "prashant kumar mishra (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-5714) [upgrade]failed to destroy CPVM in upgraded setup ;Unable to send due to java.lang.ArrayIndexOutOfBoundsException: 0
Date Thu, 02 Jan 2014 10:37:51 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13860116#comment-13860116
] 

prashant kumar mishra commented on CLOUDSTACK-5714:
---------------------------------------------------

wanted to narrow down problem " why blank console" did some basic trouble shooting 

Basic troubleshooting result:
-----------------------------------
1- ps aux | grep -v grep | grep com.cloud.agent.AgentShell

root      3651  0.1  4.8 1004140 50236 ?       Sl   09:09   0:06 java -Djavax.net.ssl.trustStore=./certs/realhostip.keystore
-Dlog.home=/var/log/cloud/ -mx804m -cp ./:./conf:activation-1.1.jar:antisamy-1.4.3.jar:aopalliance-1.0.jar:apache-log4j-extras-1.1.jar:aspectjweaver-1.7.0.jar:aws-java-sdk-1.3.22.jar:axis-1.4.jar:axis-jaxrpc-1.4.jar:batik-css-1.7.jar:batik-ext-1.7.jar:batik-util-1.7.jar:bcprov-jdk16-1.46.jar:bsh-core-2.0b4.jar:cglib-nodep-2.2.2.jar:cloud-agent-4.3.0-SNAPSHOT.jar:cloud-api-4.3.0-SNAPSHOT.jar:cloud-console-proxy-4.3.0-SNAPSHOT.jar:cloud-core-4.3.0-SNAPSHOT.jar:cloud-engine-api-4.3.0-SNAPSHOT.jar:cloud-engine-components-api-4.3.0-SNAPSHOT.jar:cloud-engine-orchestration-4.3.0-SNAPSHOT.jar:cloud-engine-schema-4.3.0-SNAPSHOT.jar:cloud-engine-storage-4.3.0-SNAPSHOT.jar:cloud-framework-cluster-4.3.0-SNAPSHOT.jar:cloud-framework-config-4.3.0-SNAPSHOT.jar:cloud-framework-db-4.3.0-SNAPSHOT.jar:cloud-framework-events-4.3.0-SNAPSHOT.jar:cloud-framework-ipc-4.3.0-SNAPSHOT.jar:cloud-framework-jobs-4.3.0-SNAPSHOT.jar:cloud-framework-managed-context-4.3.0-SNAPSHOT.jar:cloud-framework-rest-4.3.0-SNAPSHOT.jar:cloud-plugin-hypervisor-vmware-4.3.0-SNAPSHOT.jar:cloud-secondary-storage-4.3.0-SNAPSHOT.jar:cloud-server-4.3.0-SNAPSHOT.jar:cloud-utils-4.3.0-SNAPSHOT.jar:cloud-vmware-base-4.3.0-SNAPSHOT.jar:cloudstack-service-console-proxy-rdpclient-4.3.0-SNAPSHOT.jar:commons-beanutils-core-1.7.0.jar:commons-codec-1.6.jar:commons-collections-3.2.jar:commons-configuration-1.8.jar:commons-daemon-1.0.10.jar:commons-dbcp-1.4.jar:commons-fileupload-1.2.jar:commons-httpclient-3.1.jar:commons-io-1.4.jar:commons-lang-2.6.jar:commons-logging-1.1.1.jar:commons-net-3.3.jar:commons-pool-1.6.jar:cxf-bundle-jaxrs-2.7.0.jar:dom4j-1.6.1.jar:ehcache-core-2.6.6.jar:ejb-api-3.0.jar:esapi-2.0.1.jar:geronimo-javamail_1.4_spec-1.7.1.jar:geronimo-servlet_3.0_spec-1.0.jar:gson-1.7.1.jar:guava-14.0-rc1.jar:httpclient-4.2.1.jar:httpcore-4.2.1.jar:jackson-annotations-2.1.1.jar:jackson-core-2.1.1.jar:jackson-core-asl-1.8.9.jar:jackson-databind-2.1.1.jar:jackson-jaxrs-json-provider-2.1.1.jar:jackson-mapper-asl-1.8.9.jar:jackson-module-jaxb-annotations-2.1.1.jar:jasypt-1.9.0.jar:java-ipv6-0.10.jar:javassist-3.12.1.GA.jar:javax.inject-1.jar:javax.persistence-2.0.0.jar:javax.ws.rs-api-2.0-m10.jar:jaxen-1.1-beta-8.jar:jdom-1.0.jar:jetty-continuation-8.1.7.v20120910.jar:jetty-http-8.1.7.v20120910.jar:jetty-io-8.1.7.v20120910.jar:jetty-security-8.1.7.v20120910.jar:jetty-util-8.1.7.v20120910.jar:jsch-0.1.42.jar:jstl-1.2.jar:log4j-1.2.16.jar:mail-1.4.jar:mysql-connector-java-5.1.21.jar:nekohtml-1.9.12.jar:reflections-0.9.8.jar:slf4j-api-1.7.5.jar:slf4j-log4j12-1.7.5.jar:spring-aop-3.2.4.RELEASE.jar:spring-beans-3.2.4.RELEASE.jar:spring-context-3.2.4.RELEASE.jar:spring-core-3.2.4.RELEASE.jar:spring-expression-3.2.4.RELEASE.jar:spring-web-3.2.4.RELEASE.jar:stax2-api-3.1.1.jar:tomcat-embed-core-7.0.30.jar:trilead-ssh2-build213-svnkit-1.3-patch.jar:vmware-vim25-5.1.jar:woodstox-core-asl-4.1.4.jar:wsdl4j-1.4.jar:xalan-2.7.0.jar:xercesImpl-2.6.2.jar:xmlschema-core-2.0.3.jar:xom-1.1.jar:xpp3_min-1.1.4c.jar:xstream-1.3.1.jar
com.cloud.agent.AgentShell root=UUID debian-installer=en_US console=hvc0 template=domP type=consoleproxy
host=10.147.38.206 port=8250 name=v-6-VM premium=true zone=1 pod=1 guid=Proxy.6 proxy_vm=6
disable_rp_filter=true eth2ip=10.147.41.221 eth2mask=255.255.254.0 gateway=10.147.40.1 eth0ip=169.254.0.145
eth0mask=255.255.0.0 eth1ip=10.147.41.211 eth1mask=255.255.254.0 mgmtcidr=10.147.38.0/24 localgw=10.147.40.1
internaldns1=10.140.50.6 dns1=10.140.50.6

2-root@v-6-VM:~# netstat -na | grep 8250
tcp        0      0 10.147.41.211:58525     10.147.38.206:8250      ESTABLISHED

3-root@v-6-VM:~# netstat -na | grep 443
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN

4-was able to telnet from my local machine 
telnet 10.147.41.221




> [upgrade]failed to destroy CPVM in upgraded setup ;Unable to send due to java.lang.ArrayIndexOutOfBoundsException:
0
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-5714
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5714
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>    Affects Versions: 4.3.0
>         Environment: hyp:xen6.2
>            Reporter: prashant kumar mishra
>            Priority: Critical
>             Fix For: 4.3.0
>
>         Attachments: Logs_Db.rar, screenshot-1.jpg
>
>
> STEPS TO REPRODUCE
> =====================
> 1-prepare a CS setup with ACS4.2+xen server
> 2-register 64 bit template and upgrade it to 4.3
> 3-run the
> " nohup cloudstack-sysvmadm -d <mysqlserverIP/localhost> -u cloud -p password -s
> sysvm.log 2>&1 &"
> EXPECTED
> ==========
> system vm should be successfully stopped and started
> ACTUAL
> ======
> CPVM failed to start
> Tried to destroy CPVM manually 
> ================================
>  ran in this "Fail to destroy system vm" but actually cpvm got destroyed ,after new CPVM
came up , whenever i am trying to access console of any vm i am getting blank console (please
check the screenshot).
> LOG
> =======
> [root@rh63 CloudPlatform-4.3-1011-rhel6.3]#  nohup cloudstack-sysvmadm -d 10.147.38.206
-u cloud -p password -s > sysvm.log 2>&1 &
> [1] 28994
> [root@rh63 CloudPlatform-4.3-1011-rhel6.3]# tail -f sysvm.log
> nohup: ignoring input
> Stopping and starting 1 secondary storage vm(s)...
> Done stopping and starting secondary storage vm(s)
> Stopping and starting 1 console proxy vm(s)...
> ERROR: Failed to start console proxy vm with id 2
> Done stopping and starting console proxy vm(s) .
> LOG
> ====
> 014-01-02 09:27:54,583 DEBUG [c.c.a.t.Request] (Job-Executor-18:ctx-48b44e7f ctx-4c2c4411)
Seq 1-1441792111: Executing:  { Cmd , MgmtId: 7145449848920, via: 1(Rack1Pod1Host7), Ver:
v1, Flags: 100011, [] }
> 2014-01-02 09:27:54,583 WARN  [c.c.a.m.AgentAttache] (Job-Executor-18:ctx-48b44e7f ctx-4c2c4411)
Seq 1-1441792111: Unable to send due to
> java.lang.ArrayIndexOutOfBoundsException: 0
>         at com.cloud.agent.manager.DirectAgentAttache.send(DirectAgentAttache.java:100)
>         at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:374)
>         at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:398)
>         at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:394)
>         at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:920)
>         at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:497)
>         at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:448)
>         at com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:437)
>         at com.cloud.consoleproxy.ConsoleProxyManagerImpl.destroyProxy(ConsoleProxyManagerImpl.java:1162)
>         at com.cloud.server.ManagementServerImpl.destroyConsoleProxy(ManagementServerImpl.java:2155)
>         at com.cloud.server.ManagementServerImpl.destroySystemVM(ManagementServerImpl.java:3220)
>         at com.cloud.server.ManagementServerImpl.destroySystemVM(ManagementServerImpl.java:609)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy193.destroySystemVM(Unknown Source)
>         at org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd.execute(DestroySystemVmCmd.java:97)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
>         at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
>         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2014-01-02 09:27:54,589 DEBUG [c.c.a.m.AgentAttache] (Job-Executor-18:ctx-48b44e7f ctx-4c2c4411)
Seq 1-1441792111: Cancelling.
> 2014-01-02 09:27:54,589 WARN  [c.c.c.ConsoleProxyManagerImpl] (Job-Executor-18:ctx-48b44e7f
ctx-4c2c4411) Unable to expunge VM[ConsoleProxy|v-2-VM]
> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host
1: Problem due to other exception 0
>         at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:389)
>         at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:398)
>         at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:394)
>         at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:920)
>         at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:497)
>         at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:448)
>         at com.cloud.vm.VirtualMachineManagerImpl.expunge(VirtualMachineManagerImpl.java:437)
>         at com.cloud.consoleproxy.ConsoleProxyManagerImpl.destroyProxy(ConsoleProxyManagerImpl.java:1162)
>         at com.cloud.server.ManagementServerImpl.destroyConsoleProxy(ManagementServerImpl.java:2155)
>         at com.cloud.server.ManagementServerImpl.destroySystemVM(ManagementServerImpl.java:3220)
>         at com.cloud.server.ManagementServerImpl.destroySystemVM(ManagementServerImpl.java:609)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>         at $Proxy193.destroySystemVM(Unknown Source)
>         at org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd.execute(DestroySystemVmCmd.java:97)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
>         at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
>         at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> 2014-01-02 09:27:54,618 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-18:ctx-48b44e7f)
Complete async job-32, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Fail
to destroy system vm"}
> 2014-01-02 09:27:54,634 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-18:ctx-48b44e7f)
Done executing org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd for job-32
> 2014-01-02 09:27:54,645 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-18:ctx-48b44e7f)
Remove job-32 from job monitoring
> 2014-01-02 09:27:54,988 DEBUG [c.c.a.ApiServlet] (catalina-exec-9:ctx-fca8b213) ===START===
 10.252.192.25 -- GET  command=queryAsyncJobResult&jobId=43b57242-16d0-4882-aa78-6c6372c94d89&response=json&sessionkey=0fl8cR8CIDSU78AdEsEbO8fUUk8%3D&_=1388653670612



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message