ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Fedotov <alexander.fedot...@gmail.com>
Subject Re: High heap on ignite client
Date Mon, 19 Jun 2017 11:46:40 GMT
Hi Anil.

Could you please also share C:/Anil/ignite-client.xml ? As well, it would
be useful if you took JFR reports for the case with allocation profiling
enabled.
Just to clarify, by 4L do you mean 4 million entries?

Kind regards,
Alex.

On Mon, Jun 19, 2017 at 10:15 AM, Alexander Fedotov <
alexander.fedotoff@gmail.com> wrote:

> Thanks. I'll take a look and let you know about any findings.
>
> Kind regards,
> Alex
>
> 18 июня 2017 г. 3:33 PM пользователь "Anil" <anilklce@gmail.com>
написал:
>
> Hi Alex,
>
> test program repository - https://github.com/adasari/test-ignite-jdbc.git
>
> please let us if you have any suggestions/questions. thanks.
>
> Thanks
>
> On 15 June 2017 at 10:58, Anil <anilklce@gmail.com> wrote:
>
>> Sure. thanks
>>
>> On 14 June 2017 at 19:51, afedotov <alexander.fedotoff@gmail.com> wrote:
>>
>>> Hi, Anil.
>>>
>>> Could you please share your full code (class/method) you are using to
>>> read data.
>>>
>>> Kind regards,
>>> Alex
>>>
>>> 12 июня 2017 г. 4:07 PM пользователь "Anil [via Apache Ignite
Users]" <[hidden
>>> email] <http:///user/SendEmail.jtp?type=node&node=13706&i=0>>
написал:
>>>
>>>> Do you have any advice on implementing large records export from ignite
>>>> ?
>>>>
>>>> I could not use ScanQuery right as my whole application built around
>>>> Jdbc driver and writing complex queries in scan query is very difficult.
>>>>
>>>> Thanks
>>>>
>>>> On 10 June 2017 at 18:48, Anil <[hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=0>> wrote:
>>>>
>>>>> I understand from the code that there is no cursor from h2 db (or
>>>>> ignite embed h2 db) internally and all mapper response consolidated at
>>>>> reducer. It means when exporting large number of records, all data is
in
>>>>> memory.
>>>>>
>>>>>              if (send(nodes,
>>>>>                     oldStyle ?
>>>>>                         new GridQueryRequest(qryReqId,
>>>>>                             r.pageSize,
>>>>>                             space,
>>>>>                             mapQrys,
>>>>>                             topVer,
>>>>>                             extraSpaces(space, qry.spaces()),
>>>>>                             null,
>>>>>                             timeoutMillis) :
>>>>>                         new GridH2QueryRequest()
>>>>>                             .requestId(qryReqId)
>>>>>                             .topologyVersion(topVer)
>>>>>                             .pageSize(r.pageSize)
>>>>>                             .caches(qry.caches())
>>>>>                             .tables(distributedJoins ? qry.tables() :
>>>>> null)
>>>>>                             .partitions(convert(partsMap))
>>>>>                             .queries(mapQrys)
>>>>>                             .flags(flags)
>>>>>                             .timeout(timeoutMillis),
>>>>>                     oldStyle && partsMap != null ? new
>>>>> ExplicitPartitionsSpecializer(partsMap) : null,
>>>>>                     false)) {
>>>>>
>>>>>                     awaitAllReplies(r, nodes, cancel);
>>>>>
>>>>> *// once the responses from all nodes for the query received.. proceed
>>>>> further ?*
>>>>>
>>>>>           if (!retry) {
>>>>>                     if (skipMergeTbl) {
>>>>>                         List<List<?>> res = new ArrayList<>();
>>>>>
>>>>>                         // Simple UNION ALL can have multiple indexes.
>>>>>                         for (GridMergeIndex idx : r.idxs) {
>>>>>                             Cursor cur = idx.findInStream(null, null);
>>>>>
>>>>>                             while (cur.next()) {
>>>>>                                 Row row = cur.get();
>>>>>
>>>>>                                 int cols = row.getColumnCount();
>>>>>
>>>>>                                 List<Object> resRow = new
>>>>> ArrayList<>(cols);
>>>>>
>>>>>                                 for (int c = 0; c < cols; c++)
>>>>>                                     resRow.add(row.getValue(c).get
>>>>> Object());
>>>>>
>>>>>                                 res.add(resRow);
>>>>>                             }
>>>>>                         }
>>>>>
>>>>>                         resIter = res.iterator();
>>>>>                     }else {
>>>>>                       // incase of split query scenario
>>>>>                     }
>>>>>
>>>>>          }
>>>>>
>>>>>       return new GridQueryCacheObjectsIterator(resIter, cctx,
>>>>> keepPortable);
>>>>>
>>>>>
>>>>> Query cursor is iterator which does column value mapping per page. But
>>>>> still all records of query are still in memory. correct?
>>>>>
>>>>> Please correct me if I am wrong. thanks.
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> On 10 June 2017 at 15:53, Anil <[hidden email]
>>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=1>>
wrote:
>>>>>
>>>>>>
>>>>>> jvm parameters used -
>>>>>>
>>>>>> -Xmx6144m -XX:NewSize=512m -XX:+UseTLAB -XX:+UseG1GC
>>>>>> -XX:MaxGCPauseMillis=500 -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
>>>>>> -Xloggc:C:/Anil/dumps/gc-client.log -XX:+HeapDumpOnOutOfMemoryError
>>>>>> -XX:+PrintGCCause -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy
>>>>>> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpAfterFullGC
>>>>>> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+AlwaysPreTouch
>>>>>> -XX:+PrintFlagsFinal -XX:HeapDumpPath=C:/Anil/dumps
>>>>>> /heapdump-client.hprof
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On 10 June 2017 at 15:06, Anil <[hidden email]
>>>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=2>>
wrote:
>>>>>>
>>>>>>> HI,
>>>>>>>
>>>>>>> I have implemented export feature of ignite data using JDBC
>>>>>>> Interator
>>>>>>>
>>>>>>> ResultSet rs = statement.executeQuery();
>>>>>>>
>>>>>>> while (rs.next()){
>>>>>>> // do operations
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> and fetch size is 200.
>>>>>>>
>>>>>>> when i run export operation twice for 4 L records whole 6B is
filled
>>>>>>> up and never getting released.
>>>>>>>
>>>>>>> Initially i thought that operations transforting result set to
file
>>>>>>> causing the memory full. But not.
>>>>>>>
>>>>>>> I just did follwoing and still the memory is growing and not
getting
>>>>>>> released
>>>>>>>
>>>>>>> while (rs.next()){
>>>>>>>  // nothing
>>>>>>> }
>>>>>>>
>>>>>>> num     #instances         #bytes  class name
>>>>>>> ----------------------------------------------
>>>>>>>    1:      55072353     2408335272  [C
>>>>>>>    2:      54923606     1318166544  java.lang.String
>>>>>>>    3:        779006      746187792  [B
>>>>>>>    4:        903548      304746304  [Ljava.lang.Object;
>>>>>>>    5:        773348      259844928  net.juniper.cs.entity.Install
>>>>>>> Base
>>>>>>>    6:       4745694      113896656  java.lang.Long
>>>>>>>    7:       1111692       44467680  sun.nio.cs.UTF_8$Decoder
>>>>>>>    8:        773348       30933920  org.apache.ignite.internal.bi
>>>>>>> nary.BinaryObjectImpl
>>>>>>>    9:        895627       21495048  java.util.ArrayList
>>>>>>>   10:         12427       16517632  [I
>>>>>>>
>>>>>>>
>>>>>>> Not sure why string objects are getting increased.
>>>>>>>
>>>>>>> Could you please help in understanding the issue ?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> If you reply to this email, your message will be added to the
>>>> discussion below:
>>>> http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-
>>>> ignite-client-tp13594p13626.html
>>>> To start a new topic under Apache Ignite Users, email [hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=13706&i=1>
>>>> To unsubscribe from Apache Ignite Users, click here.
>>>> NAML
>>>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>
>>> ------------------------------
>>> View this message in context: Re: High heap on ignite client
>>> <http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client-tp13594p13706.html>
>>> Sent from the Apache Ignite Users mailing list archive
>>> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>>>
>>
>>
>
>

Mime
View raw message