cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Stevens <williamstev...@gmail.com>
Subject Re: Null Pointer listing public ips for a project
Date Thu, 13 Apr 2017 21:49:50 GMT
Yes. I believe this is the issue we are facing as well. The query you
supplied returned 3 rows, so I think that is where the problem stems from.

We have seen some periodic issues with the deletion of a project not
actually cascading the delete of all of the resources associated with it.
We have monitoring setup on some resources to help detect zombies, but we
had not set one up on IPs.

We have not been able to isolate when/why we sometimes get the zombies when
deleting projects. We may just delete the resources owned by the project
first before deleting the project, but that is not ideal.

Thanks for the quick answer on the Wei.

On Apr 13, 2017 4:29 PM, "Wei ZHOU" <ustcweizhou@gmail.com> wrote:

> Hi Will,
>
> In our case, the account removal is terminated because the last host of a
> vm is removed from platform, so NPE was thrown.
> The NPE error is fixed by commit 4312d884629d4a9e7716da4c869433cdaa8181b8
> We were running with cloudstack 4.2.1 before, so we fixed it by similar
> change but the resources are not cleaned yet.
>
>
> -Wei
>
>
>
> 2017-04-13 22:06 GMT+02:00 Will Stevens <wstevens@cloudops.com>:
>
> > Hey Wei,
> > That sounds about right.  I will have a look on our side and see what I
> can
> > come up with.
> >
> > Thanks,
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Thu, Apr 13, 2017 at 3:48 PM, Wei ZHOU <ustcweizhou@gmail.com> wrote:
> >
> > > Hi Will,
> > >
> > > I checked our platforms. We also have this issue. After investigation,
> I
> > > found a public ip was not released in an account removal in 2015 which
> > > should not happen.
> > >
> > > If you execute the following query, you should get empty result. If
> not,
> > > then something went wrong in the past.
> > > select * from user_ip_address uia join account on uia.account_id=
> > > account.id
> > > where account.removed is not null;
> > >
> > > -Wei
> > >
> > >
> > > 2017-04-13 18:36 GMT+02:00 Will Stevens <wstevens@cloudops.com>:
> > >
> > > > The problem with that is that we are using the 'projectid=-1' to
> search
> > > > across all projects.
> > > >
> > > > I am not sure how a project gets into that state, but must be
> possible
> > > > through the API since that is our interface with ACS.
> > > >
> > > > We should be more defensive in this api call though because an end
> user
> > > api
> > > > call should never be able to throw a null pointer error.
> > > >
> > > > My guess is that we should do a null check on the 'project' here
> before
> > > we
> > > > try to use it.
> > > > https://github.com/apache/cloudstack/blob/master/server/
> > > src/com/cloud/api/
> > > > ApiResponseHelper.java#L2303
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Thu, Apr 13, 2017 at 12:17 PM, Rohit Yadav <
> > rohit.yadav@shapeblue.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Will,
> > > > >
> > > > >
> > > > > I tested this against a local KVM-based 4.9.2.0 environment and I
> > could
> > > > > not reproduce the error. I see this in api logs:
> > > > >
> > > > >
> > > > > ==> apilog.log <==
> > > > > 2017-04-13 18:16:30,340 INFO  [a.c.c.a.ApiServer]
> > > > > (catalina-exec-5:ctx-2b0966b8 ctx-2760e856) (logid:7439bdc4)
> > (userId=2
> > > > > accountId=2 sessionId=6183303731812D5EC32B1B37445AEC60)
> 192.168.1.58
> > > --
> > > > > GET signatureversion=3&sessionkey=H6E-LUsWpVY6WT_mJzz7qjGAdxg&
> > > > > command=listPublicIpAddresses&projectid=-1&expires=2017-04-
> > > > > 13T16%3A26%3A30%2B0000&response=json&listall=true 200 {"
> > > > > listpublicipaddressesresponse":{}}
> > > > >
> > > > > I think proper way to fix this would be to check that project
> object
> > is
> > > > > not null before calling any methods on it.
> > > > >
> > > > >
> > > > > Regards.
> > > > >
> > > > > ________________________________
> > > > > From: Will Stevens <swill@apache.org>
> > > > > Sent: 13 April 2017 21:38:34
> > > > > To: dev@cloudstack.apache.org
> > > > > Subject: Null Pointer listing public ips for a project
> > > > >
> > > > > Hey All,
> > > > > Looking for the best way to resolve this issue.
> > > > >
> > > > > My line numbers in the stack trace will not match the line numbers
> I
> > > > > reference because I am running 4.7 and I am linking the master
> files.
> > > > From
> > > > > what I can tell by reviewing the code, this has not been fixed in
> > > master.
> > > > > So here goes.
> > > > >
> > > > > When making a call to listPublicIpAddresses [1] with 'listall=true'
> > and
> > > > > 'projectid=-1', I get the following stacktrace.
> > > > >
> > > > > java.lang.NullPointerException
> > > > > at
> > > > > com.cloud.api.ApiResponseHelper.populateOwner(
> > > > ApiResponseHelper.java:2258)
> > > > > at
> > > > > com.cloud.api.ApiResponseHelper.createIPAddressResponse(
> > > > > ApiResponseHelper.java:718)
> > > > > at
> > > > > org.apache.cloudstack.api.command.admin.address.
> > > > > ListPublicIpAddressesCmdByAdmin.execute(
> > ListPublicIpAddressesCmdByAdmi
> > > > > n.java:44)
> > > > > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> > > > > at com.cloud.api.ApiServer.queueCommand(ApiServer.java:698)
> > > > > at com.cloud.api.ApiServer.handleRequest(ApiServer.java:529)
> > > > > at com.cloud.api.ApiServlet.processRequestInContext(
> > > ApiServlet.java:297)
> > > > > at com.cloud.api.ApiServlet$1.run(ApiServlet.java:127)
> > > > > at
> > > > > org.apache.cloudstack.managed.context.impl.
> > > DefaultManagedContext$1.call(
> > > > > DefaultManagedContext.java:56)
> > > > > at
> > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.
> > > > > callWithContext(DefaultManagedContext.java:103)
> > > > > at
> > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.
> > > > > runWithContext(DefaultManagedContext.java:53)
> > > > > at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:124)
> > > > > at com.cloud.api.ApiServlet.doGet(ApiServlet.java:86)
> > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> > > > > at
> > > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> > > > > ApplicationFilterChain.java:290)
> > > > > at
> > > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(
> > > > > ApplicationFilterChain.java:206)
> > > > > at
> > > > > org.apache.catalina.core.StandardWrapperValve.invoke(
> > > > > StandardWrapperValve.java:233)
> > > > > at
> > > > > org.apache.catalina.core.StandardContextValve.invoke(
> > > > > StandardContextValve.java:191)
> > > > > at
> > > > > org.apache.catalina.core.StandardHostValve.invoke(
> > > > > StandardHostValve.java:127)
> > > > > at
> > > > > org.apache.catalina.valves.ErrorReportValve.invoke(
> > > > > ErrorReportValve.java:102)
> > > > > at org.apache.catalina.valves.AccessLogValve.invoke(
> > > > > AccessLogValve.java:555)
> > > > > at
> > > > > org.apache.catalina.core.StandardEngineValve.invoke(
> > > > > StandardEngineValve.java:109)
> > > > > at
> > > > > org.apache.catalina.connector.CoyoteAdapter.service(
> > > > > CoyoteAdapter.java:299)
> > > > > at
> > > > > org.apache.coyote.http11.Http11NioProcessor.process(
> > > > > Http11NioProcessor.java:889)
> > > > > at
> > > > > org.apache.coyote.http11.Http11NioProtocol$
> Http11ConnectionHandler.
> > > > > process(Http11NioProtocol.java:732)
> > > > > at
> > > > > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.
> > > > > run(NioEndpoint.java:2274)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > ThreadPoolExecutor.java:1145)
> > > > > at
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > ThreadPoolExecutor.java:615)
> > > > > at java.lang.Thread.run(Thread.java:745)
> > > > >
> > > > >
> > > > > The stack trace comes from this code [2]:
> > > > > Project project = ApiDBUtils.findProjectByProjectAccountId(
> > > > > account.getId());
> > > > > response.setProjectId(project.getUuid());  // <-- Null Pointer
> > > > > response.setProjectName(project.getName());
> > > > >
> > > > > In this case, the flow is coming from [3], which comes from [4].
> > > > >
> > > > > What would be the right way to fix this issue?
> > > > >
> > > > > Thanks...
> > > > >
> > > > > [1]
> > > > > https://cloudstack.apache.org/api/apidocs-4.7/root_admin/
> > > > > listPublicIpAddresses.html
> > > > > [2]
> > > > > https://github.com/apache/cloudstack/blob/master/server/
> > > > src/com/cloud/api/
> > > > > ApiResponseHelper.java#L2303
> > > > > [3]
> > > > > https://github.com/apache/cloudstack/blob/master/server/
> > > > src/com/cloud/api/
> > > > > ApiResponseHelper.java#L761
> > > > > [4]
> > > > > https://github.com/apache/cloudstack/blob/master/api/
> > > > > src/org/apache/cloudstack/api/command/admin/address/
> > > > > ListPublicIpAddressesCmdByAdmin.java#L44
> > > > >
> > > > > *Will Stevens*
> > > > >
> > > > > rohit.yadav@shapeblue.com
> > > > > www.shapeblue.com
> > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > @shapeblue
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message