Mailing-List: contact dev-help@apex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.apache.org
Date: Mon, 26 Sep 2016 21:38:20 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: dev@apex.incubator.apache.org
Message-ID: <JIRA.12997422.1471302720000.667404.1474925900651@Atlassian.JIRA>
In-Reply-To: <JIRA.12997422.1471302720000@Atlassian.JIRA>
References: <JIRA.12997422.1471302720000@Atlassian.JIRA> <JIRA.12997422.1471302720367@arcas>
Subject: [jira] [Commented] (APEXMALHAR-2190) Use reusable buffer to serial
 spillable data structure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Mon, 26 Sep 2016 21:38:26 -0000


    [ https://issues.apache.org/jira/browse/APEXMALHAR-2190?page=3Dcom.atla=
ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=
=3D15524245#comment-15524245 ]=20

ASF GitHub Bot commented on APEXMALHAR-2190:
--------------------------------------------

GitHub user brightchen reopened a pull request:

    https://github.com/apache/apex-malhar/pull/404

    APEXMALHAR-2190 #resolve #comment Use reusable buffer to serial spill=
=E2=80=A6

    =E2=80=A6able data structure

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/brightchen/apex-malhar APEXMALHAR-2190-PR

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/apex-malhar/pull/404.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #404
   =20
----
commit 13e8b60811501f8bdbb75b4112c9a74676ec3b3c
Author: brightchen <bright@datatorrent.com>
Date:   2016-08-16T00:46:27Z

    APEXMALHAR-2190 #resolve #comment Use reusable buffer to serial spillab=
le data structure

----


> Use reusable buffer to serial spillable data structure
> ------------------------------------------------------
>
>                 Key: APEXMALHAR-2190
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-219=
0
>             Project: Apache Apex Malhar
>          Issue Type: Task
>            Reporter: bright chen
>            Assignee: bright chen
>   Original Estimate: 240h
>  Remaining Estimate: 240h
>
> Spillable Data Structure created lots of temporary memory to serial data =
lot of of memory copy( see SliceUtils.concatenate(byte[], byte[]). Which us=
ed up memory very quickly. See APEXMALHAR-2182.
> Use a shared memory to avoid allocate temporary memory and memory copy
> some basic ideas
> - SerToLVBuffer interface provides a method serTo(T object, LengthValueBu=
ffer buffer): instead of create a memory and then return the serialized dat=
a, this method let the caller pass in the buffer. So different objects or o=
bject with embed objects can share the same LengthValueBuffer
> - LengthValueBuffer: It is a buffer which manage the memory as length and=
 value(which is the generic format of serialized data). which provide lengt=
h placeholder mechanism to avoid temporary memory and data copy when the le=
ngth can be know after data serialized
> - memory management classes: includes interface ByteStream and it's imple=
mentations: Block, FixedBlock, BlocksStream. Which provides a mechanism to =
dynamic allocate and manage memory. Which basically provides following func=
tion. I tried other some other stream mechamism such as ByteArrayInputStrea=
m, but it can meet 3rd criteria, and don't have good performance(50% loss)=
=20
>   - dynamic allocate memory
>   - reset memory for reuse
>   - BlocksStream make sure the output slices will not be changed when nee=
d extra memory; Block can change the reference of output slices buffer is d=
ata was moved due to reallocate of memory(BlocksStream is better solution).
>   - WindowableBlocksStream extends from BlocksStream and provides functio=
n to reset memory window by window instead of reset all memory. It provides=
 certain amount of cache( as bytes ) in memory


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)