incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kelven Yang <kelven.y...@citrix.com>
Subject Re: [DISCUSS] Fix Bloat: Getting rid of autoscanning and using xml for spring DI
Date Fri, 22 Feb 2013 03:33:44 GMT
I did heap analysis today. It is quite certain that the problem is not
related with component auto-scanning, it is related with the memory leak
introduced by Spring AOP. I'm now working on a solution to it.

Kelven

On 2/21/13 3:02 PM, "Kelven Yang" <kelven.yang@citrix.com> wrote:

>Rohit,
>
>I'll checkout out your work about using XML instead of @Component.
>However, I'm not sure if the memory leak issue is related with auto-scan
>at all. I'm now doing a heap analysis and will update with the result
>
>Kelven
>
>
>
>On 2/21/13 7:02 AM, "Rohit Yadav" <bhaisaab@apache.org> wrote:
>
>>Okay Chip, posted a comment on the issue.
>>
>>Hi Kelven, you want to check my fix. Thanks.
>>
>>Regards.
>>
>>On Thu, Feb 21, 2013 at 8:17 PM, Chip Childers
>><chip.childers@sungard.com> wrote:
>>>
>>> Can you coordinate with Kelvin on this?  He's got CLOUDSTACK-1276, and
>>> had previously mentioned this exact same thing (using the xml file vs.
>>> annotations).
>>>
>>> On Thu, Feb 21, 2013 at 07:26:42PM +0530, Rohit Yadav wrote:
>>>> Just to add, we will have separate xmls for every kind of profile, I
>>>> can make every plugin have its own spring xml which is copied and
>>>> injected to the web.xml comments, suggestions, flames?
>>>>
>>>> Possible future experiments:
>>>> - Spring xmls artefacts based
>>>> - Have a way to fix AOP, don't add pointcuts on all classes, methods
>>>> - Make DI framework replaceable in CloudStack, I would go for Guice ;)
>>>>
>>>> Regards.
>>>>
>>>> On Thu, Feb 21, 2013 at 7:22 PM, Rohit Yadav <bhaisaab@apache.org>
>>>>wrote:
>>>> > I'm experimenting with removing all @Components and moving beans
>>>>into
>>>> > an xml that could be read by spring to do DI.
>>>> > This decreases the load time and
>>>> >
>>>> > Errors:
>>>> > - Abstract classes annotated as spring components
>>>> > - Annotations for project which are not included in either build
>>>> > profiles: usage, simulator, oss (default) and nonoss
>>>> > - Some project don't even build or are not part of any build
>>>> > artefacts, looks like the @Component annotation was done in a hurry
>>>>:)
>>>> > - Interfaces being annotated as spring components
>>>> > - Prasanna, Abhi and I spoke with Spring gurus Vijay V and Vaibhav
>>>>on
>>>> > how to fix this bloat, so in applicationContext.xml we do some aop
>>>> > stuff to have transaction stuff we get from @DB annotation and this
>>>> > needs fixing as it's applying the following for all classes and
>>>> > methods (dude bloat, run! :)
>>>> >
>>>> >   <aop:config proxy-target-class="true">
>>>> >     <aop:aspect id="dbContextBuilder"
>>>>ref="transactionContextBuilder">
>>>> >         <aop:pointcut id="captureAnyMethod"
>>>> >             expression="execution(* *(..))"
>>>> >         />
>>>> >
>>>> > Methodology and results:
>>>> > I wrote a python program that would go through the whole source code
>>>> > and fix all java files (except test, awsapi) who have @Component and
>>>> > create bean rules in an xml file. Next I had to manually fix few
>>>> > things (see error above) and I got the mgmt server running, the load
>>>> > time had improved a lot (little faster now) after disabling
>>>> > autoscanning and just using the xml generated. I see the benefit of
>>>> > just using the xml and not having spring as a build time dependency
>>>> > (we're using @Component in our classes so :).
>>>> >
>>>> > This was an experiment mostly because a lot of folks are having
>>>>spring
>>>> > memory issues, especially Prasanna's devcloud-ci cannot run and
>>>>check
>>>> > our commits as memory kills dom0. So, if no one objects may I go
>>>>ahead
>>>> > and commit it? For me the jvm took about 650MB real memory on osx.
>>>> >
>>>> > Right now I made it for all build profiles, nonoss being the
>>>>superset,
>>>> > my experiment can be checked out from here:
>>>> > https://github.com/bhaisaab/cloudstack/tree/spring-annotation-to-xml
>>>> >
>>>> > Regards.
>>>> >
>>>> > The python program, in case anyone wants to replicate or is curious:
>>>> > (defaulter is just a file containing result of: file . | grep java$
>>>>>
>>>> > defaulter)
>>>> > f = open('defaulter', 'r')
>>>> > files = f.read().split('\n')[:-1]
>>>> > files = filter(lambda x: not x.startswith('./awsapi') and
>>>> > x.find('/test/') == -1, files)
>>>> > xmldata = ""
>>>> > bean = """  <bean id="%s" class="%s"/>\n"""
>>>> > xml = """<!--
>>>> >   Licensed to the Apache Software Foundation (ASF) under one
>>>> >   or more contributor license agreements. See the NOTICE file
>>>> >   distributed with this work for additional information
>>>> >   regarding copyright ownership. The ASF licenses this file
>>>> >   to you under the Apache License, Version 2.0 (the
>>>> >   "License"); you may not use this file except in compliance
>>>> >   with the License. You may obtain a copy of the License at
>>>> >
>>>> >   http://www.apache.org/licenses/LICENSE-2.0
>>>> >
>>>> >   Unless required by applicable law or agreed to in writing,
>>>> >   software distributed under the License is distributed on an
>>>> >   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
>>>> >   KIND, either express or implied. See the License for the
>>>> >   specific language governing permissions and limitations
>>>> >   under the License.
>>>> > -->
>>>> > <beans xmlns="http://www.springframework.org/schema/beans"
>>>> >   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>> >   xmlns:context="http://www.springframework.org/schema/context"
>>>> >   xmlns:tx="http://www.springframework.org/schema/tx"
>>>> >   xmlns:aop="http://www.springframework.org/schema/aop"
>>>> >   xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>> >
>>>> > http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
>>>> >                       http://www.springframework.org/schema/tx
>>>> >             
>>>>http://www.springframework.org/schema/tx/spring-tx-3.0.xsd
>>>> >                       http://www.springframework.org/schema/aop
>>>> >
>>>> > http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
>>>> >                       http://www.springframework.org/schema/context
>>>> >
>>>> > 
>>>>http://www.springframework.org/schema/context/spring-context-3.0.xsd">
>>>> >
>>>> > %s
>>>> > </beans>
>>>> > """
>>>> > misses = []
>>>> > for file in files:
>>>> >     name = file.split('/')[-1].split('.')[0]
>>>> >     name = name[0].lower() + name[1:]
>>>> >     pkgname = ""
>>>> >     idx = 0
>>>> >     paths = file.split('/')[1:]
>>>> >     paths[-1] = paths[-1].split('.')[0]
>>>> >     for str in paths:
>>>> >         if str == 'org' or str == 'com':
>>>> >             pkgname = '.'.join(paths[idx:])
>>>> >         idx += 1
>>>> >     f = open(file, 'r')
>>>> >     data = f.read()
>>>> >     f.close()
>>>> >     idx = data.find('@Component')
>>>> >     if idx == -1:
>>>> >         misses.append(file)
>>>> >         continue
>>>> >     string = '@Component'
>>>> >     if data[len(string)+idx] == '(':
>>>> >       endidx = data.find(')', len(string)+idx)
>>>> >       name =
>>>>data[len(string)+idx+1:endidx].split('=')[-1].replace('"', '')
>>>> >       name = name.replace("'", '')
>>>> >       data = data[:idx] + data[endidx+2:]
>>>> >     else:
>>>> >       data = data.replace(string, '')
>>>> >     data = data.replace('import
>>>> > org.springframework.stereotype.Component;\n', '')
>>>> >     f = open(file, 'w')
>>>> >     f.write(data)
>>>> >     f.close()
>>>> >     print name, pkgname
>>>> >     xmldata += bean % (name, pkgname)
>>>> >     xmldata += "\n"
>>>> >
>>>> > xml = xml % xmldata
>>>> > f = open('./client/tomcatconf/defaultComponentContext.xml.in', 'w')
>>>> > f.write(xml)
>>>> > f.close()
>>>> >
>>>> > f = open('misses', 'w')
>>>> > f.write('\n'.join(misses))
>>>> > f.close()
>>>>
>


Mime
View raw message