ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From afedotov <alexander.fedot...@gmail.com>
Subject Re: High heap on ignite client
Date Wed, 14 Jun 2017 14:21:05 GMT
Hi, Anil.

Could you please share your full code (class/method) you are using to read
data.

Kind regards,
Alex

12 июня 2017 г. 4:07 PM пользователь "Anil [via Apache Ignite Users]" <
ml+s70518n13626h40@n6.nabble.com> написал:

> Do you have any advice on implementing large records export from ignite ?
>
> I could not use ScanQuery right as my whole application built around Jdbc
> driver and writing complex queries in scan query is very difficult.
>
> Thanks
>
> On 10 June 2017 at 18:48, Anil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=13626&i=0>> wrote:
>
>> I understand from the code that there is no cursor from h2 db (or ignite
>> embed h2 db) internally and all mapper response consolidated at reducer. It
>> means when exporting large number of records, all data is in memory.
>>
>>              if (send(nodes,
>>                     oldStyle ?
>>                         new GridQueryRequest(qryReqId,
>>                             r.pageSize,
>>                             space,
>>                             mapQrys,
>>                             topVer,
>>                             extraSpaces(space, qry.spaces()),
>>                             null,
>>                             timeoutMillis) :
>>                         new GridH2QueryRequest()
>>                             .requestId(qryReqId)
>>                             .topologyVersion(topVer)
>>                             .pageSize(r.pageSize)
>>                             .caches(qry.caches())
>>                             .tables(distributedJoins ? qry.tables() :
>> null)
>>                             .partitions(convert(partsMap))
>>                             .queries(mapQrys)
>>                             .flags(flags)
>>                             .timeout(timeoutMillis),
>>                     oldStyle && partsMap != null ? new
>> ExplicitPartitionsSpecializer(partsMap) : null,
>>                     false)) {
>>
>>                     awaitAllReplies(r, nodes, cancel);
>>
>> *// once the responses from all nodes for the query received.. proceed
>> further ?*
>>
>>           if (!retry) {
>>                     if (skipMergeTbl) {
>>                         List<List<?>> res = new ArrayList<>();
>>
>>                         // Simple UNION ALL can have multiple indexes.
>>                         for (GridMergeIndex idx : r.idxs) {
>>                             Cursor cur = idx.findInStream(null, null);
>>
>>                             while (cur.next()) {
>>                                 Row row = cur.get();
>>
>>                                 int cols = row.getColumnCount();
>>
>>                                 List<Object> resRow = new
>> ArrayList<>(cols);
>>
>>                                 for (int c = 0; c < cols; c++)
>>                                     resRow.add(row.getValue(c).get
>> Object());
>>
>>                                 res.add(resRow);
>>                             }
>>                         }
>>
>>                         resIter = res.iterator();
>>                     }else {
>>                       // incase of split query scenario
>>                     }
>>
>>          }
>>
>>       return new GridQueryCacheObjectsIterator(resIter, cctx,
>> keepPortable);
>>
>>
>> Query cursor is iterator which does column value mapping per page. But
>> still all records of query are still in memory. correct?
>>
>> Please correct me if I am wrong. thanks.
>>
>>
>> Thanks
>>
>>
>> On 10 June 2017 at 15:53, Anil <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=13626&i=1>> wrote:
>>
>>>
>>> jvm parameters used -
>>>
>>> -Xmx6144m -XX:NewSize=512m -XX:+UseTLAB -XX:+UseG1GC
>>> -XX:MaxGCPauseMillis=500 -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
>>> -Xloggc:C:/Anil/dumps/gc-client.log -XX:+HeapDumpOnOutOfMemoryError
>>> -XX:+PrintGCCause -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy
>>> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpAfterFullGC
>>> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+AlwaysPreTouch
>>> -XX:+PrintFlagsFinal -XX:HeapDumpPath=C:/Anil/dumps
>>> /heapdump-client.hprof
>>>
>>> Thanks.
>>>
>>> On 10 June 2017 at 15:06, Anil <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=2>> wrote:
>>>
>>>> HI,
>>>>
>>>> I have implemented export feature of ignite data using JDBC Interator
>>>>
>>>> ResultSet rs = statement.executeQuery();
>>>>
>>>> while (rs.next()){
>>>> // do operations
>>>>
>>>> }
>>>>
>>>> and fetch size is 200.
>>>>
>>>> when i run export operation twice for 4 L records whole 6B is filled up
>>>> and never getting released.
>>>>
>>>> Initially i thought that operations transforting result set to file
>>>> causing the memory full. But not.
>>>>
>>>> I just did follwoing and still the memory is growing and not getting
>>>> released
>>>>
>>>> while (rs.next()){
>>>>  // nothing
>>>> }
>>>>
>>>> num     #instances         #bytes  class name
>>>> ----------------------------------------------
>>>>    1:      55072353     2408335272  [C
>>>>    2:      54923606     1318166544  java.lang.String
>>>>    3:        779006      746187792  [B
>>>>    4:        903548      304746304  [Ljava.lang.Object;
>>>>    5:        773348      259844928  net.juniper.cs.entity.InstallBase
>>>>    6:       4745694      113896656  java.lang.Long
>>>>    7:       1111692       44467680  sun.nio.cs.UTF_8$Decoder
>>>>    8:        773348       30933920  org.apache.ignite.internal.bi
>>>> nary.BinaryObjectImpl
>>>>    9:        895627       21495048  java.util.ArrayList
>>>>   10:         12427       16517632  [I
>>>>
>>>>
>>>> Not sure why string objects are getting increased.
>>>>
>>>> Could you please help in understanding the issue ?
>>>>
>>>> Thanks
>>>>
>>>
>>>
>>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client-
> tp13594p13626.html
> To start a new topic under Apache Ignite Users, email
> ml+s70518n1h65@n6.nabble.com
> To unsubscribe from Apache Ignite Users, click here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=YWxleGFuZGVyLmZlZG90b2ZmQGdtYWlsLmNvbXwxfC0xMzYxNTU0NTg=>
> .
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client-tp13594p13706.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Mime
View raw message