apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bright Chen <bri...@datatorrent.com>
Subject Suggestion on optimise kryo Output
Date Tue, 10 Jan 2017 00:10:51 GMT
Hi,

The kryo Output has some limitation

   - The size of the data is limited. kryo write data to a fixed buffer, it
   will throw the overflow exception if the data exceed the size
   - The Output.toBytes() will copy the data to temporary buffer and
   output, it will decrease the performance and introduce garbage collection.

When I was tuning Spillable Data structure and Manage State. I create a
mechanism to share and reuse the memory to avoid above problem. And it can
be reused in core serialization with small change. Please see jira:
https://issues.apache.org/jira/browse/APEXMALHAR-2190


Any suggestion or comments, please put in jira:
https://issues.apache.org/jira/browse/APEXCORE-606


thanks

Bright

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message