apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Farkas <...@datatorrent.com>
Subject Re: Kryo misuse in definePartitions method
Date Tue, 26 Jan 2016 00:59:20 GMT
+1 for util function

Thanks,
Tim

On Mon, Jan 25, 2016 at 4:52 PM, Siyuan Hua <siyuan@datatorrent.com> wrote:

> Sandesh and I found an issue in almost all the operators that use Kryo to
> clone operator itself  in definePartition method.  They all use default
> Kryo instance without setting context classloader from the thread that
> execute the code. It would fail in local mode if it tries to load class
> that only in app package classpath.
>
> When I'm trying to fix it in all the places, I found repeated pattern in
> definePartitions. And it's very error-prone since no one actually did it in
> right way until we found the issue.  I'm thinking to make a Util functin to
> create instance from operator itself and also hide the Kryo dependency from
> the operator.  Do you have any better suggestions to do this?
>
> Thanks!
>
> Siyuan
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message