Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: "Kirschnick, Johannes" <johannes.kirschnick@tu-berlin.de>
To: "dev@flink.apache.org" <dev@flink.apache.org>
Subject: Operating on Serialized Data
Thread-Topic: Operating on Serialized Data
Thread-Index: AQHQUBjKoBm7XHbYT0KjnW412Gx76g==
Date: Tue, 24 Feb 2015 10:13:45 +0000
Message-ID: <1424772824901.49508@tu-berlin.de>
Accept-Language: de-DE, en-US
Content-Language: de-DE
Content-Type: multipart/alternative;
	boundary="_000_142477282490149508tuberlinde_"
MIME-Version: 1.0

--_000_142477282490149508tuberlinde_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi list,


I have a general question on as to whether it's possible to significantly s=
peed up the processing by cutting down on the serialization costs during it=
erations.


The basic setup that I have are a couple of vectors that are repeatedly mut=
ated (added & multiplied) as part of an iterative run within a reducer.

A vector is basically "just" an array of doubles - all of the same size.


I noticed during simple profiling that roughly 50% of the execution time is=
 spent on serializing the data in using the com.esotericsoftware.kryo.seria=
lizers.DefaultArraySerializers in Kryo.


I know that any custom operation would would varant custom processing, but =
given the serialization contributes such a large amount of processing time =
to the overall runtime it might very well be worthwhile


Is that currently exposed in any fashion to the user code, or are there som=
e hooks I could look into?


Thanks

Johannes

--_000_142477282490149508tuberlinde_--