manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gustavo Beneitez <gustavo.benei...@gmail.com>
Subject Re: High performance issues with Elastic Connector
Date Mon, 17 Sep 2018 13:44:32 GMT
also, here are the top instances for each object, after a few navigation
they seem somehow retained by the connector

Class Name
Objects   Shallow
Heap    Retained Heap
-----------------------------------------------------------------------------------------------------------
char[]                                                            75.233
197.316.528  >= 197.316.528
byte[]                                                            7.842
9.589.952    >= 9.589.952
java.lang.String                                                  73.108
1.754.592   >= 65.437.048
java.util.HashMap$Node                                            40.294
1.289.408    >= 3.626.760
java.util.concurrent.ConcurrentHashMap$Node                       23.259
744.288    >= 2.563.680
java.util.HashMap$Node[]                                          4.140
598.560    >= 4.222.672
int[]                                                             4.993
573.072      >= 573.072
java.lang.Object[]                                                4.299
335.232    >= 3.012.440
java.lang.reflect.Method                                          3.628
319.264      >= 445.016
java.util.HashMap                                                 6.074
291.552    >= 4.481.752
java.util.Hashtable$Entry                                         6.423
205.536      >= 738.760
java.util.concurrent.ConcurrentHashMap$Node[]                     461
202.608    >= 2.760.224
java.util.regex.Pattern                                           2.546
183.312      >= 642.104
java.lang.Object                                                  10.149
162.384      >= 162.384
java.lang.String[]                                                1.768
120.888      >= 556.048
java.lang.Integer                                                 7.269
116.304      >= 116.928
com.mysql.jdbc.ConnectionPropertiesImpl$BooleanConnectionProperty 1.666
106.624      >= 106.632
java.util.ArrayList                                               3.479
83.496    >= 2.185.496
java.util.TreeMap$Entry                                           1.931
77.240      >= 294.136
org.apache.tika.mime.MimeType                                     1.540
73.920      >= 192.576
java.lang.Long                                                    3.061
73.464       >= 73.504
java.util.LinkedList$Node                                         3.052
73.248       >= 82.152
java.util.Hashtable$Entry[]                                       317
70.144
>= 829.168
java.util.LinkedHashMap$Entry                                     1.726
69.040      >= 132.560
org.apache.catalina.loader.ResourceEntry                          2.819
67.656       >= 67.656
java.lang.Class                                                   5.735
62.712    >= 6.118.936
java.util.regex.Pattern$Slice                                     2.434
58.416      >= 141.216
java.util.regex.Pattern$Start                                     2.431
58.344       >= 58.344
org.apache.tika.mime.MediaType                                    1.818
58.176      >= 171.440
java.util.concurrent.ConcurrentHashMap                            899 57.536
>= 2.814.184
org.apache.manifoldcf.crawler.interfaces.DocumentDescription      1.303
52.120      >= 190.520
sun.security.util.DerValue                                        1.541
49.312      >= 212.248
sun.security.util.DerInputBuffer                                  1.541
49.312      >= 125.928
java.net.URL                                                      736
47.104
>= 138.832
java.util.LinkedList                                              1.452
46.464      >= 128.624
org.apache.tomcat.util.modeler.AttributeInfo                      845
40.560
>= 109.608
-----------------------------------------------------------------------------------------------------------


El lun., 17 sept. 2018 a las 15:32, Gustavo Beneitez (<
gustavo.beneitez@gmail.com>) escribió:

> you are right, in fact there is only one instance that manages more than
> 200 MB, with Top Consumers
> Biggest Top-Level Dominator Class Loaders
> LabelNumber of ObjectsUsed Heap SizeRetained Heap SizeRetained Heap, %
> java.net.FactoryURLClassLoader @ 0xed503040 2.437 89.384 212.569.400
> 89,81%
> org.apache.catalina.loader.ParallelWebappClassLoader @ 0xecc5f4a0 3.212
> 121.744 11.547.920 4,88%
> <system class loader> 21.740 938.712 9.264.456 3,91%
> java.net.URLClassLoader @ 0xecc27b48
>
>
> 519 18.784 2.762.008 1,17%
>
>    - Total: 4 entries
>
> 27.908 1.168.624 236.143.784
>
>
> I am trying to gather further info but eclipse MAT is sometimes a bit
> confuse
>
>
> El lun., 17 sept. 2018 a las 14:18, Karl Wright (<daddywri@gmail.com>)
> escribió:
>
>> How many instances does MAT show for ElasticSearchConnection?
>>
>> A "leak suspect" is the class whose number of instances is the largest.
>> It
>> doesn't mean there's an actual memory leak.
>>
>> Karl
>>
>>
>> On Mon, Sep 17, 2018 at 7:49 AM Gustavo Beneitez <
>> gustavo.beneitez@gmail.com>
>> wrote:
>>
>> > Hi Karl,
>> >
>> > I reviewed the UI and there are a total of
>> > ElasticSearch 100
>> > Seems not so many connections...
>> >
>> >
>> >
>> > El lun., 17 sept. 2018 a las 12:56, Karl Wright (<daddywri@gmail.com>)
>> > escribió:
>> >
>> > > Hi Gustavo,
>> > >
>> > > Connector instances are pooled in general, so once a connector
>> instance
>> > is
>> > > created it is kept around.  The number of instances will max out at
>> the
>> > > number of connections you allow in the UI when defining the
>> connection.
>> > > How many did you allocate?
>> > >
>> > > Karl
>> > >
>> > >
>> > > On Mon, Sep 17, 2018 at 6:23 AM Gustavo Beneitez <
>> > > gustavo.beneitez@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi there,
>> > > >
>> > > > I was indexing in production several intranet websites and I
>> realized
>> > the
>> > > > performance was becoming poor (no tomcat restart for two weeks).
>> > > >
>> > > > After a few thoughts about what  is going on I decided to create a
>> dump
>> > > > file and review it with eclipse MAT and I observed that the class
>> > > >
>> > > > Leak Suspects
>> > > >
>> > >
>> >
>> *org.apache.manifoldcf.agents.output.elasticsearch.ElasticSearchConnection*
>> > > >
>> > > > had a lot of instances and was growing up and never released. Do you
>> > have
>> > > > any issue related to Elastic? Maybe there is something related to
>> our
>> > > > version (2.4.6) of Elastic.
>> > > >
>> > > > I also have attached the MAT report. We have all components "by
>> > default",
>> > > > i.e. neither no customization of any component nor any new one
>> created.
>> > > >
>> > > > Thanks in advance.
>> > > >
>> > >
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message