apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shubham Pathak <shub...@datatorrent.com>
Subject Re: Kryo misuse in definePartitions method
Date Tue, 26 Jan 2016 15:13:40 GMT
Could we have  some more details as to what exactly is the issue and why is
it failing in local mode ?
Also, i couldn't get why only local mode is mentioned specifically ? Does
it not affect when operator runs on the cluster ?

Thanks,
Shubham

On Tue, Jan 26, 2016 at 7:40 PM, Pramod Immaneni <pramod@datatorrent.com>
wrote:

> Siyuan can you explain the statement some more "It would fail in local mode
> if it tries to load class that only in app package classpath."
>
> Thanks
>
> On Mon, Jan 25, 2016 at 4:52 PM, Siyuan Hua <siyuan@datatorrent.com>
> wrote:
>
> > Sandesh and I found an issue in almost all the operators that use Kryo to
> > clone operator itself  in definePartition method.  They all use default
> > Kryo instance without setting context classloader from the thread that
> > execute the code. It would fail in local mode if it tries to load class
> > that only in app package classpath.
> >
> > When I'm trying to fix it in all the places, I found repeated pattern in
> > definePartitions. And it's very error-prone since no one actually did it
> in
> > right way until we found the issue.  I'm thinking to make a Util functin
> to
> > create instance from operator itself and also hide the Kryo dependency
> from
> > the operator.  Do you have any better suggestions to do this?
> >
> > Thanks!
> >
> > Siyuan
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message