jackrabbit-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Morrell Jacobs <mjac...@maned.com>
Subject Re: Intermittent Slow Down
Date Mon, 17 Dec 2012 22:57:58 GMT
Thanks Ian, will try.

I added the -server flag earlier today so I'll see if it helps.  I've used the JMX tools (VisualVM)
and running GC manually seems to drop back down to a base level (or close).

When tomcat / web app is idle, the heap usage seems to be a saw tooth pattern, which isn't
a surprise.  The test takes a while to run so have to wait and see if -server helps.

I'm using JVM 6.

Thanks!


On Dec 17, 2012, at 5:16 PM, Ian Boston wrote:

> Hi Morrell,
> Sounds suspiciously like garbage collection activity. Rather than
> attaching a profiler which may impact heap usage, try attaching a JMX
> console which should be less impact (eg JConsole) and looking at the
> graph of Garbage Collection activity during the slow down.
> 
> If you see it taking up a significant amount of time and CPU, try
> changing the type of garbage collector[1]. IIRC the default without
> the -server flag is a serial GC. Increasing heap will lengthen the
> time between these slowdowns but won't eliminate them.
> 
> Also, you may be able to recreate the slowdown by forcing a full GC
> operation from JConsole. Obviously if a full GC operation doesn't
> leave the JVM with significantly more free heap, then you need to up
> the limit to prevent the JVM spending all its time trying to free
> more.
> 
> You didn't say which JVM you are using, the link below is for 6.
> Adjust appropriately.
> 
> HTH
> Ian
> 
> 
> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#available_collectors
> 
> On 18 December 2012 05:14, Morrell Jacobs <mjacobs@maned.com> wrote:
>> We're running MySQL 5.5.19 for Linux
>> 
>> We had the blobs in the database because of clustering.  Since we've moved to single
stack, I suppose we could pull them out - not a long term solution, but we could try and see
if it helps.
>> 
>> I'll have our MySQL experts review the settings, the restructuring may be accounting
for some of the slow down.  If we were seeing delays from MySQL, I wouldn't expect Tomcat's
CPU usage to be pegged though - I would think all the threads would be idle, waiting for a
response.  Does that sound right to you?
>> 
>> Thanks
>> Morrell
>> 
>> 
>> On Dec 17, 2012, at 12:48 PM, Ron Wheeler wrote:
>> 
>>> I would play with the JVM memory configuration.
>>> It still looks very small.
>>> Try doubling it and see if it changes anything.
>>> If not, you can look elsewhere and if it does, try making it larger until it
stops improving things.
>>> 
>>> What version of MySQL are you running?
>>> Have you tuned your MySQL?
>>> You might want to look at how MySQL is going to treat these objects and what
configuration parameters are recommended when you have large blobs.
>>> Have you thought about getting your binary content out of the database. Databases
are not very good at 100Mb blobs.
>>> Links to 100Mb blobs/files work much better.
>>> 
>>> Is there a compelling reason to put the binary data inside a database?
>>> Anytime MySQL has to restructure its indexes or physical storage, you are going
to see a pause in the action.
>>> 
>>> Ron
>>> 
>>> On 17/12/2012 12:04 PM, Morrell Jacobs wrote:
>>>> Hello all,
>>>> 
>>>> We're currently experience a problem where JackRabbit function will occasionally
slow down; according to our QA, the slow down comes in waves: good performance for a while
(hours) then bad for a while (minutes / hours).  The slow down is on the order of minutes
to perform create or modify an object (creation involves creating a 2-3 levels of nodes, and
setting properties).  Normally these operations take 100 - 300 ms, but during the slow down
they can take as much 5 minutes.
>>>> 
>>>> The slow down does not seem to be connected to the load on the server: we've
seen it perform well, when many people are connecting, then slow down with only one user.
 I suspect the slow down is related to indexing, but that's just a guess on my part - I'm
looking for something that would happen at some interval.
>>>> 
>>>> 
>>>> Environment:
>>>> * JackRabbit is built into war running in Tomcat (was 7.0.25, recently upgraded
to 7.0.34)
>>>> * Tomcat was running with default configs but recently up'd memory - -Xms128M
-Xmx512M -XX:MaxPermSize=256M
>>>> * Repository XML is at end of email
>>>> * JackRabbit was originally configured to be clustered, but is now running
as just a single instance.
>>>> 
>>>> 
>>>> Data:
>>>> * Our data is structured similar to a file system, except any object can
be both a file and contain other objects (files); each object has various metadata (some in
properties, some in child nodes), a file and child objects
>>>> * We don't use full text searching and don't want file (binary) contents
indexed; the only way I was able to prevent the search index from examining the files (some
are large 100+ M) was to create my own node types for files.  My node types are identical
nt:file and nt:content, except that they are named prd:file and prd:content; the unrecognized
name seems to prevent indexing from touch them.
>>>> 
>>>> 
>>>> When I've attempted to use profiling tools, it appears that during the slow
down the CPU is pegged; memory usage is also at or close to the limit.  In depth profiling
tools (NetBeans) get overwhelmed during the slow down.
>>>> 
>>>> We're going thru a variety of tests, but any advice the community can provide
will be greatly appreciated.
>>>> 
>>>> Here's the repository.xml:
>>>> <Repository>
>>>>    <FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem">
>>>>        <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>        <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&amp;createDatabaseIfNotExist=true"/>
>>>>        <param name="schema" value="mysql"/>
>>>>        <param name="schemaObjectPrefix" value="rep_"/>
>>>>        <param name="user" value="${pm.database.user}"/>
>>>>        <param name="password" value="${pm.database.pwd}"/>
>>>>    </FileSystem>
>>>> 
>>>>    <!--
>>>>        security configuration
>>>>    -->
>>>>    <Security appName="Jackrabbit">
>>>>        <!--
>>>>            security manager:
>>>>            class: FQN of class implementing the JackrabbitSecurityManager
interface
>>>>        -->
>>>>        <SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager"
workspaceName="default">
>>>>            <!--
>>>>                workspace access:
>>>>                class: FQN of class implementing the WorkspaceAccessManager
interface
>>>>            -->
>>>>            <!-- <WorkspaceAccessManager class="..."/> -->
>>>>            <!-- <param name="config" value="${rep.home}/security.xml"/>
-->
>>>>        </SecurityManager>
>>>>        <!--
>>>>            access manager:
>>>>            class: FQN of class implementing the AccessManager interface
>>>>        -->
>>>>        <AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager">
>>>>            <!-- <param name="config" value="${rep.home}/access.xml"/>
-->
>>>>        </AccessManager>
>>>>                 <LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule">
>>>>            <!--
>>>>                anonymous user name ('anonymous' is the default value)
>>>>            -->
>>>>            <param name="anonymousId" value="anonymous"/>
>>>>            <!--
>>>>                administrator user id (default value if param is missing is
'admin')
>>>>            -->
>>>>            <param name="adminId" value="admin"/>
>>>>        </LoginModule>
>>>>    </Security>
>>>>         <!--
>>>>        location of workspaces root directory and name of default workspace
>>>>    -->
>>>>    <Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>
>>>>         <!--
>>>>        workspace configuration template:
>>>>        used to create the initial workspace if there's no workspace yet
>>>>    -->
>>>> <!--    <Workspace name="default">-->
>>>>    <Workspace name="default">
>>>>        <!--
>>>>            virtual file system of the workspace:
>>>>            class: FQN of class implementing the FileSystem interface
>>>>        -->
>>>>        <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>            <param name="path" value="${wsp.home}"/>
>>>>        </FileSystem>
>>>>        <!--
>>>>            persistence manager of the workspace:
>>>>            class: FQN of class implementing the PersistenceManager interface
>>>>        -->
>>>>        <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>            <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>            <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/>
>>>>            <param name="user" value="${pm.database.user}"/>
>>>>            <param name="password" value="${pm.database.pwd}"/>
>>>>            <param name="schemaObjectPrefix" value="Product_"/>
>>>>            <param name="schema" value="mysql"/>
>>>>        </PersistenceManager>
>>>>        <!--
>>>>            Search index and the file system it uses.
>>>>            class: FQN of class implementing the QueryHandler interface
>>>>        -->
>>>>        <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
>>>>            <param name="path" value="${wsp.home}/index"/>
>>>>            <!--<param name="textFilterClasses"
>>>>                value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/>
>>>>            <param name="extractorPoolSize" value="2"/>
>>>>            <param name="supportHighlighting" value="true"/> -->
>>>>        </SearchIndex>
>>>>    </Workspace>
>>>>         <!--
>>>>        Configures the versioning
>>>>    -->
>>>>    <Versioning rootPath="${rep.home}/version">
>>>>        <!--
>>>>            Configures the filesystem to use for versioning for the respective
>>>>            persistence manager
>>>>        -->
>>>>        <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>            <param name="path" value="${rep.home}/version"/>
>>>>        </FileSystem>
>>>>        <!--
>>>>            Configures the persistence manager to be used for persisting version
state.
>>>>            Please note that the current versioning implementation is based
on
>>>>            a 'normal' persistence manager, but this could change in future
>>>>            implementations.
>>>>        -->
>>>>        <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>            <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>            <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/>
>>>>            <param name="user" value="${pm.database.user}"/>
>>>>            <param name="password" value="${pm.database.pwd}"/>
>>>>            <param name="schemaObjectPrefix" value="version_"/>
>>>>            <param name="schema" value="mysql"/>
>>>>        </PersistenceManager>
>>>>    </Versioning>
>>>>         <!--
>>>>        Configures the Data Store for large binary objects.
>>>>    -->
>>>>    <DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
>>>>        <param name="path" value="${datastore.location.path}"/>
>>>>        <param name="minRecordLength" value="100"/>
>>>>    </DataStore>
>>>>    <!--
>>>>    <Cluster id="${cluster.id}" syncDelay="2000">
>>>>        <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
>>>>            <param name="driver" value="com.mysql.jdbc.Driver" />
>>>>            <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&amp;createDatabaseIfNotExist=true"/>
>>>>            <param name="schemaObjectPrefix" value="journal_"/>
>>>>            <param name="databaseType" value="mysql"/>
>>>>            <param name="user" value="${pm.database.user}"/>
>>>>            <param name="password" value="${pm.database.pwd}"/>
>>>>        </Journal>
>>>>    </Cluster>
>>>>    -->
>>>> 
>>>> </Repository>
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Morrell Jacobs
>>>> Chief Software Architect
>>>> MEI
>>>> 610 Old York Road, Suite 250
>>>> Jenkintown, PA 19046
>>>> Phone: 215-886-5662, ext. 252
>>>> Fax: 215-886-5681
>>>> http://www.maned.com
>>>> E-mail: mjacobs@maned.com
>>>> AOL IM: MorrellMEI
>>>> 
>>>> Have you seen Nervous Pixel, MEI's creative services division?
>>>> www.nervouspixel.com
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> Ron Wheeler
>>> President
>>> Artifact Software Inc
>>> email: rwheeler@artifact-software.com
>>> skype: ronaldmwheeler
>>> phone: 866-970-2435, ext 102
>>> 
>> 


Mime
View raw message