jackrabbit-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ron Wheeler <rwhee...@artifact-software.com>
Subject Re: Intermittent Slow Down
Date Tue, 18 Dec 2012 00:04:16 GMT
Did increasing the memory affect the situation?
That is probably the easiest stick to poke the beast with.

If is has no effect, it is less likely to be a garbage collection issue.


Ron

On 17/12/2012 6:34 PM, Ian Boston wrote:
> On 18 December 2012 09:57, Morrell Jacobs <mjacobs@maned.com> wrote:
>> Thanks Ian, will try.
>>
>> I added the -server flag earlier today so I'll see if it helps.  I've used the JMX
tools (VisualVM) and running GC manually seems to drop back down to a base level (or close).
>>
>> When tomcat / web app is idle, the heap usage seems to be a saw tooth pattern, which
isn't a surprise.  The test takes a while to run so have to wait and see if -server helps.
>>
>> I'm using JVM 6.
> I think -server will switch to the ParallelGC. If you still have
> problems read the section on  Ergonomics [1], and if the adjustments
> there make no difference think about switching to the Concurrent GC,
> however if you have no free cores for long periods the Concurrent GC
> may not get enough time to free tenured space which will lead to a JVM
> pause.  [2]
>
> VisualVM should show you which pool is being exhausted and give you a
> clue to the root cause of a GC related pause to the application
> threads.
>
> (if you wanted to get forensic you could turn on GC logging leave it
> for 24h and then graph the output in a spreadsheet.).
>
> HTH
> Ian
>
>
> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#par_gc.ergonomics
>
> 2 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#cms.concurrent_mode_failure
>
>> Thanks!
>>
>>
>> On Dec 17, 2012, at 5:16 PM, Ian Boston wrote:
>>
>>> Hi Morrell,
>>> Sounds suspiciously like garbage collection activity. Rather than
>>> attaching a profiler which may impact heap usage, try attaching a JMX
>>> console which should be less impact (eg JConsole) and looking at the
>>> graph of Garbage Collection activity during the slow down.
>>>
>>> If you see it taking up a significant amount of time and CPU, try
>>> changing the type of garbage collector[1]. IIRC the default without
>>> the -server flag is a serial GC. Increasing heap will lengthen the
>>> time between these slowdowns but won't eliminate them.
>>>
>>> Also, you may be able to recreate the slowdown by forcing a full GC
>>> operation from JConsole. Obviously if a full GC operation doesn't
>>> leave the JVM with significantly more free heap, then you need to up
>>> the limit to prevent the JVM spending all its time trying to free
>>> more.
>>>
>>> You didn't say which JVM you are using, the link below is for 6.
>>> Adjust appropriately.
>>>
>>> HTH
>>> Ian
>>>
>>>
>>> 1 http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#available_collectors
>>>
>>> On 18 December 2012 05:14, Morrell Jacobs <mjacobs@maned.com> wrote:
>>>> We're running MySQL 5.5.19 for Linux
>>>>
>>>> We had the blobs in the database because of clustering.  Since we've moved
to single stack, I suppose we could pull them out - not a long term solution, but we could
try and see if it helps.
>>>>
>>>> I'll have our MySQL experts review the settings, the restructuring may be
accounting for some of the slow down.  If we were seeing delays from MySQL, I wouldn't expect
Tomcat's CPU usage to be pegged though - I would think all the threads would be idle, waiting
for a response.  Does that sound right to you?
>>>>
>>>> Thanks
>>>> Morrell
>>>>
>>>>
>>>> On Dec 17, 2012, at 12:48 PM, Ron Wheeler wrote:
>>>>
>>>>> I would play with the JVM memory configuration.
>>>>> It still looks very small.
>>>>> Try doubling it and see if it changes anything.
>>>>> If not, you can look elsewhere and if it does, try making it larger until
it stops improving things.
>>>>>
>>>>> What version of MySQL are you running?
>>>>> Have you tuned your MySQL?
>>>>> You might want to look at how MySQL is going to treat these objects and
what configuration parameters are recommended when you have large blobs.
>>>>> Have you thought about getting your binary content out of the database.
Databases are not very good at 100Mb blobs.
>>>>> Links to 100Mb blobs/files work much better.
>>>>>
>>>>> Is there a compelling reason to put the binary data inside a database?
>>>>> Anytime MySQL has to restructure its indexes or physical storage, you
are going to see a pause in the action.
>>>>>
>>>>> Ron
>>>>>
>>>>> On 17/12/2012 12:04 PM, Morrell Jacobs wrote:
>>>>>> Hello all,
>>>>>>
>>>>>> We're currently experience a problem where JackRabbit function will
occasionally slow down; according to our QA, the slow down comes in waves: good performance
for a while (hours) then bad for a while (minutes / hours).  The slow down is on the order
of minutes to perform create or modify an object (creation involves creating a 2-3 levels
of nodes, and setting properties).  Normally these operations take 100 - 300 ms, but during
the slow down they can take as much 5 minutes.
>>>>>>
>>>>>> The slow down does not seem to be connected to the load on the server:
we've seen it perform well, when many people are connecting, then slow down with only one
user.  I suspect the slow down is related to indexing, but that's just a guess on my part
- I'm looking for something that would happen at some interval.
>>>>>>
>>>>>>
>>>>>> Environment:
>>>>>> * JackRabbit is built into war running in Tomcat (was 7.0.25, recently
upgraded to 7.0.34)
>>>>>> * Tomcat was running with default configs but recently up'd memory
- -Xms128M -Xmx512M -XX:MaxPermSize=256M
>>>>>> * Repository XML is at end of email
>>>>>> * JackRabbit was originally configured to be clustered, but is now
running as just a single instance.
>>>>>>
>>>>>>
>>>>>> Data:
>>>>>> * Our data is structured similar to a file system, except any object
can be both a file and contain other objects (files); each object has various metadata (some
in properties, some in child nodes), a file and child objects
>>>>>> * We don't use full text searching and don't want file (binary) contents
indexed; the only way I was able to prevent the search index from examining the files (some
are large 100+ M) was to create my own node types for files.  My node types are identical
nt:file and nt:content, except that they are named prd:file and prd:content; the unrecognized
name seems to prevent indexing from touch them.
>>>>>>
>>>>>>
>>>>>> When I've attempted to use profiling tools, it appears that during
the slow down the CPU is pegged; memory usage is also at or close to the limit.  In depth
profiling tools (NetBeans) get overwhelmed during the slow down.
>>>>>>
>>>>>> We're going thru a variety of tests, but any advice the community
can provide will be greatly appreciated.
>>>>>>
>>>>>> Here's the repository.xml:
>>>>>> <Repository>
>>>>>>     <FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem">
>>>>>>         <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>>         <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&amp;createDatabaseIfNotExist=true"/>
>>>>>>         <param name="schema" value="mysql"/>
>>>>>>         <param name="schemaObjectPrefix" value="rep_"/>
>>>>>>         <param name="user" value="${pm.database.user}"/>
>>>>>>         <param name="password" value="${pm.database.pwd}"/>
>>>>>>     </FileSystem>
>>>>>>
>>>>>>     <!--
>>>>>>         security configuration
>>>>>>     -->
>>>>>>     <Security appName="Jackrabbit">
>>>>>>         <!--
>>>>>>             security manager:
>>>>>>             class: FQN of class implementing the JackrabbitSecurityManager
interface
>>>>>>         -->
>>>>>>         <SecurityManager class="org.apache.jackrabbit.core.DefaultSecurityManager"
workspaceName="default">
>>>>>>             <!--
>>>>>>                 workspace access:
>>>>>>                 class: FQN of class implementing the WorkspaceAccessManager
interface
>>>>>>             -->
>>>>>>             <!-- <WorkspaceAccessManager class="..."/> -->
>>>>>>             <!-- <param name="config" value="${rep.home}/security.xml"/>
-->
>>>>>>         </SecurityManager>
>>>>>>         <!--
>>>>>>             access manager:
>>>>>>             class: FQN of class implementing the AccessManager interface
>>>>>>         -->
>>>>>>         <AccessManager class="org.apache.jackrabbit.core.security.DefaultAccessManager">
>>>>>>             <!-- <param name="config" value="${rep.home}/access.xml"/>
-->
>>>>>>         </AccessManager>
>>>>>>                  <LoginModule class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule">
>>>>>>             <!--
>>>>>>                 anonymous user name ('anonymous' is the default value)
>>>>>>             -->
>>>>>>             <param name="anonymousId" value="anonymous"/>
>>>>>>             <!--
>>>>>>                 administrator user id (default value if param is
missing is 'admin')
>>>>>>             -->
>>>>>>             <param name="adminId" value="admin"/>
>>>>>>         </LoginModule>
>>>>>>     </Security>
>>>>>>          <!--
>>>>>>         location of workspaces root directory and name of default
workspace
>>>>>>     -->
>>>>>>     <Workspaces rootPath="${rep.home}/workspaces" defaultWorkspace="default"/>
>>>>>>          <!--
>>>>>>         workspace configuration template:
>>>>>>         used to create the initial workspace if there's no workspace
yet
>>>>>>     -->
>>>>>> <!--    <Workspace name="default">-->
>>>>>>     <Workspace name="default">
>>>>>>         <!--
>>>>>>             virtual file system of the workspace:
>>>>>>             class: FQN of class implementing the FileSystem interface
>>>>>>         -->
>>>>>>         <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>>>             <param name="path" value="${wsp.home}"/>
>>>>>>         </FileSystem>
>>>>>>         <!--
>>>>>>             persistence manager of the workspace:
>>>>>>             class: FQN of class implementing the PersistenceManager
interface
>>>>>>         -->
>>>>>>         <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>>>             <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>>             <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/>
>>>>>>             <param name="user" value="${pm.database.user}"/>
>>>>>>             <param name="password" value="${pm.database.pwd}"/>
>>>>>>             <param name="schemaObjectPrefix" value="Product_"/>
>>>>>>             <param name="schema" value="mysql"/>
>>>>>>         </PersistenceManager>
>>>>>>         <!--
>>>>>>             Search index and the file system it uses.
>>>>>>             class: FQN of class implementing the QueryHandler interface
>>>>>>         -->
>>>>>>         <SearchIndex class="org.apache.jackrabbit.core.query.lucene.SearchIndex">
>>>>>>             <param name="path" value="${wsp.home}/index"/>
>>>>>>             <!--<param name="textFilterClasses"
>>>>>>                 value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/>
>>>>>>             <param name="extractorPoolSize" value="2"/>
>>>>>>             <param name="supportHighlighting" value="true"/>
-->
>>>>>>         </SearchIndex>
>>>>>>     </Workspace>
>>>>>>          <!--
>>>>>>         Configures the versioning
>>>>>>     -->
>>>>>>     <Versioning rootPath="${rep.home}/version">
>>>>>>         <!--
>>>>>>             Configures the filesystem to use for versioning for the
respective
>>>>>>             persistence manager
>>>>>>         -->
>>>>>>         <FileSystem class="org.apache.jackrabbit.core.fs.local.LocalFileSystem">
>>>>>>             <param name="path" value="${rep.home}/version"/>
>>>>>>         </FileSystem>
>>>>>>         <!--
>>>>>>             Configures the persistence manager to be used for persisting
version state.
>>>>>>             Please note that the current versioning implementation
is based on
>>>>>>             a 'normal' persistence manager, but this could change
in future
>>>>>>             implementations.
>>>>>>         -->
>>>>>>         <PersistenceManager class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager">
>>>>>>             <param name="driver" value="com.mysql.jdbc.Driver"/>
>>>>>>             <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/>
>>>>>>             <param name="user" value="${pm.database.user}"/>
>>>>>>             <param name="password" value="${pm.database.pwd}"/>
>>>>>>             <param name="schemaObjectPrefix" value="version_"/>
>>>>>>             <param name="schema" value="mysql"/>
>>>>>>         </PersistenceManager>
>>>>>>     </Versioning>
>>>>>>          <!--
>>>>>>         Configures the Data Store for large binary objects.
>>>>>>     -->
>>>>>>     <DataStore class="org.apache.jackrabbit.core.data.FileDataStore">
>>>>>>         <param name="path" value="${datastore.location.path}"/>
>>>>>>         <param name="minRecordLength" value="100"/>
>>>>>>     </DataStore>
>>>>>>     <!--
>>>>>>     <Cluster id="${cluster.id}" syncDelay="2000">
>>>>>>         <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal">
>>>>>>             <param name="driver" value="com.mysql.jdbc.Driver"
/>
>>>>>>             <param name="url" value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&amp;createDatabaseIfNotExist=true"/>
>>>>>>             <param name="schemaObjectPrefix" value="journal_"/>
>>>>>>             <param name="databaseType" value="mysql"/>
>>>>>>             <param name="user" value="${pm.database.user}"/>
>>>>>>             <param name="password" value="${pm.database.pwd}"/>
>>>>>>         </Journal>
>>>>>>     </Cluster>
>>>>>>     -->
>>>>>>
>>>>>> </Repository>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Morrell Jacobs
>>>>>> Chief Software Architect
>>>>>> MEI
>>>>>> 610 Old York Road, Suite 250
>>>>>> Jenkintown, PA 19046
>>>>>> Phone: 215-886-5662, ext. 252
>>>>>> Fax: 215-886-5681
>>>>>> http://www.maned.com
>>>>>> E-mail: mjacobs@maned.com
>>>>>> AOL IM: MorrellMEI
>>>>>>
>>>>>> Have you seen Nervous Pixel, MEI's creative services division?
>>>>>> www.nervouspixel.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Ron Wheeler
>>>>> President
>>>>> Artifact Software Inc
>>>>> email: rwheeler@artifact-software.com
>>>>> skype: ronaldmwheeler
>>>>> phone: 866-970-2435, ext 102
>>>>>


-- 
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102


Mime
View raw message