ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anil <anilk...@gmail.com>
Subject Re: High heap on ignite client
Date Sat, 10 Jun 2017 13:18:23 GMT
I understand from the code that there is no cursor from h2 db (or ignite
embed h2 db) internally and all mapper response consolidated at reducer. It
means when exporting large number of records, all data is in memory.

             if (send(nodes,
                    oldStyle ?
                        new GridQueryRequest(qryReqId,
                            r.pageSize,
                            space,
                            mapQrys,
                            topVer,
                            extraSpaces(space, qry.spaces()),
                            null,
                            timeoutMillis) :
                        new GridH2QueryRequest()
                            .requestId(qryReqId)
                            .topologyVersion(topVer)
                            .pageSize(r.pageSize)
                            .caches(qry.caches())
                            .tables(distributedJoins ? qry.tables() : null)
                            .partitions(convert(partsMap))
                            .queries(mapQrys)
                            .flags(flags)
                            .timeout(timeoutMillis),
                    oldStyle && partsMap != null ? new
ExplicitPartitionsSpecializer(partsMap) : null,
                    false)) {

                    awaitAllReplies(r, nodes, cancel);

*// once the responses from all nodes for the query received.. proceed
further ?*

          if (!retry) {
                    if (skipMergeTbl) {
                        List<List<?>> res = new ArrayList<>();

                        // Simple UNION ALL can have multiple indexes.
                        for (GridMergeIndex idx : r.idxs) {
                            Cursor cur = idx.findInStream(null, null);

                            while (cur.next()) {
                                Row row = cur.get();

                                int cols = row.getColumnCount();

                                List<Object> resRow = new ArrayList<>(cols);

                                for (int c = 0; c < cols; c++)
                                    resRow.add(row.getValue(c).getObject());

                                res.add(resRow);
                            }
                        }

                        resIter = res.iterator();
                    }else {
                      // incase of split query scenario
                    }

         }

      return new GridQueryCacheObjectsIterator(resIter, cctx, keepPortable);


Query cursor is iterator which does column value mapping per page. But
still all records of query are still in memory. correct?

Please correct me if I am wrong. thanks.


Thanks


On 10 June 2017 at 15:53, Anil <anilklce@gmail.com> wrote:

>
> jvm parameters used -
>
> -Xmx6144m -XX:NewSize=512m -XX:+UseTLAB -XX:+UseG1GC
> -XX:MaxGCPauseMillis=500 -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
> -Xloggc:C:/Anil/dumps/gc-client.log -XX:+HeapDumpOnOutOfMemoryError
> -XX:+PrintGCCause -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy
> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpAfterFullGC
> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+AlwaysPreTouch
> -XX:+PrintFlagsFinal -XX:HeapDumpPath=C:/Anil/dumps/heapdump-client.hprof
>
> Thanks.
>
> On 10 June 2017 at 15:06, Anil <anilklce@gmail.com> wrote:
>
>> HI,
>>
>> I have implemented export feature of ignite data using JDBC Interator
>>
>> ResultSet rs = statement.executeQuery();
>>
>> while (rs.next()){
>> // do operations
>>
>> }
>>
>> and fetch size is 200.
>>
>> when i run export operation twice for 4 L records whole 6B is filled up
>> and never getting released.
>>
>> Initially i thought that operations transforting result set to file
>> causing the memory full. But not.
>>
>> I just did follwoing and still the memory is growing and not getting
>> released
>>
>> while (rs.next()){
>>  // nothing
>> }
>>
>> num     #instances         #bytes  class name
>> ----------------------------------------------
>>    1:      55072353     2408335272  [C
>>    2:      54923606     1318166544  java.lang.String
>>    3:        779006      746187792  [B
>>    4:        903548      304746304  [Ljava.lang.Object;
>>    5:        773348      259844928  net.juniper.cs.entity.InstallBase
>>    6:       4745694      113896656  java.lang.Long
>>    7:       1111692       44467680  sun.nio.cs.UTF_8$Decoder
>>    8:        773348       30933920  org.apache.ignite.internal.bi
>> nary.BinaryObjectImpl
>>    9:        895627       21495048  java.util.ArrayList
>>   10:         12427       16517632  [I
>>
>>
>> Not sure why string objects are getting increased.
>>
>> Could you please help in understanding the issue ?
>>
>> Thanks
>>
>
>

Mime
View raw message