Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <-1984445410349459728@unknownmsgid>
References: <CAHvM9d2DKo31DYRDQbthGma1AXM--zHYhZf4-j5mZ6wYuw6eRA@mail.gmail.com>
 <CAJ484BgstF2oBPhtAsm+eRgC-Sn1V3cn-wzrkh2-1_dnP6GQOg@mail.gmail.com>
 <CAJhKGzoqfqx4-WA-DhCX_V_p3MtKP71Jr3b8cqDCJxTvXPG=Xw@mail.gmail.com>
 <5d63056d-194f-a5b1-1485-fab7444db2a6@datatorrent.com> <-1984445410349459728@unknownmsgid>
From: Bhupesh Chawda <bhupesh@datatorrent.com>
Date: Tue, 17 May 2016 11:34:23 -0700
Message-ID: <CAHvM9d3h2M4xXCNb+xL6X7exjSFVOPUim62hGW9AtFn3Lh-zcQ@mail.gmail.com>
Subject: Re: Serialization in Apex
To: dev@apex.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a113d0f049a05b905330dfe58
archived-at: Tue, 17 May 2016 18:34:54 -0000

--001a113d0f049a05b905330dfe58
Content-Type: text/plain; charset=UTF-8

As Ram ans Sandesh pointed out, we do have @Bind and @DefaultSerializer
annotations. However, these are tightly coupled with the field in question
and do require modifying external code. Additionally it may also break
other systems, if we are binding it to a JavaSerializer and perhaps there
are systems which have other means of serializing the field.

My point was more to do with user having to worry about what serializer to
use and how to serialize objects.
For example, I liked the approach that Storm takes by falling back to Java
serialization automatically in case the target class does not have a
default constructor.

Of course, we can explore type based serialization. But this email was more
about the usability aspect; to handle classes not having default
constructors in general, not just POJO tuples.

~Bhupesh


On Tue, May 17, 2016 at 9:53 AM, Pramod Immaneni <pramod@datatorrent.com>
wrote:

> Can we do a test where we hard code a codec for a POJO and compare
> performance against kryo. Thereafter we can dynamically compose a
> codec via pojoutils and inject it.
>
> Thanks
>
> > On May 17, 2016, at 8:16 AM, Vlad Rozov <v.rozov@datatorrent.com> wrote:
> >
> > +1 for type based serialization. Tuples in most cases are flat
> records/pojo and it should be possible programmatically construct a codec
> that will significantly outperform Kryo. It should also reduce amount of
> data passed over the wire. I started to look in that direction as well as
> Kryo serialization is one of bottlenecks that limits Apex throughput when
> operators are deployed into different containers including NODE_LOCAL case.
> >
> > Thank you,
> > Vlad
> >
> >> On 5/17/16 07:13, Sandesh Hegde wrote:
> >> If it is possible to serialize, platform should do it automatically, it
> >> reduces the tribal knowledge requirement to use the platform. Couples of
> >> month back, I also sent out the similar email.
> >>
> >> Type based serialization may improve the performance.
> >>
> >>> On Tue, May 17, 2016, 6:06 AM Munagala Ramanath <ram@datatorrent.com>
> wrote:
> >>>
> >>> Traditionally, we've recommended using
> >>> "@DefaultSerializer(JavaSerializer.class)" or
> >>> "@FieldSerializer.Bind(CustomSerializer.class)" as outlined at
> >>>
> >>>
> http://docs.datatorrent.com/troubleshooting/#application-throwing-following-kryo-exception
> >>>
> >>> Can you describe why those approaches are not adequate ?
> >>>
> >>> Ram
> >>>
> >>> On Mon, May 16, 2016 at 11:46 PM, Bhupesh Chawda <
> bhupesh@datatorrent.com>
> >>> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> While working on the integration of Apex with Apache Samoa, I am
> coming
> >>>> across some scenarios where I have to add default constructors in some
> >>>> external classes to make them Kryo serializable. Although this should
> be
> >>>> okay, we would like to avoid modifying external classes as far as
> >>> possible.
> >>>> Some other streaming engines have taken different approaches towards
> >>>> serialization.
> >>>>
> >>>> I looked at Flink and Storm serialization mechanisms.
> >>>>
> >>>> Storm has a fall back mechanism on Java serialization. It does use
> Kryo
> >>> for
> >>>> serialization due to performance. But, if the class is not
> serializable
> >>>> using Kryo, then it will try to serialize it using Java
> serialization. If
> >>>> even then it cannot serialize, then it throws an error. [1]
> >>>>
> >>>> Flink has its own serialization stack where it uses a serializer
> based on
> >>>> the type information known about the data. [2]
> >>>>
> >>>> What does the community think about the current state of
> serialization in
> >>>> Apex. Is there a need to explore some approaches which could avoid
> >>>> serialization issues such as the one described above? Are there any
> other
> >>>> approaches one could use?
> >>>>
> >>>> 1.
> >>>
> http://storm.apache.org/releases/current/Serialization.html#java-serialization
> >>>> 2.
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/Type+System,+Type+Extraction,+Serialization
> >>>>
> >>>> ~Bhupesh
> >
>

--001a113d0f049a05b905330dfe58--