apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramod Immaneni <pra...@datatorrent.com>
Subject Re: Kryo misuse in definePartitions method
Date Tue, 26 Jan 2016 17:41:29 GMT
That is a good explanation, can you add it to the JIRA as well.

Thanks

On Tue, Jan 26, 2016 at 9:39 AM, Siyuan Hua <siyuan@datatorrent.com> wrote:

> @Pramod
>
> My understanding of local mode is the application is run in the same
> process as dtcli, so the default classloader can only load the classes in
> dtcli classpath.
> When it tries to launch the application that has other dependencies, in
> StramAppLauncher.loadDependencies() method, it tries to create a new
> ClassLoader to include those libs and set it as context classloader.
> So if the operator needs to load classes in it's own dependencies, it needs
> to get that context classloader.
>
> On Tue, Jan 26, 2016 at 8:46 AM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>
> > The JIRA does not explain why it happens. Can you add the explanation. It
> > might be useful if similar situations arise in future in other scenarios.
> >
> > On Tue, Jan 26, 2016 at 7:17 AM, Sandesh Hegde <sandesh@datatorrent.com>
> > wrote:
> >
> > > Here is the jira for the issue
> > > https://issues.apache.org/jira/browse/APEXMALHAR-1984
> > >
> > > It affects only local mode not on the cluster. I was trying it on
> > > DataTorrent Sandbox, in that machine local mode failed and passed on
> > yarn.
> > >
> > > On Tue, Jan 26, 2016 at 7:13 AM Shubham Pathak <
> shubham@datatorrent.com>
> > > wrote:
> > >
> > > > Could we have  some more details as to what exactly is the issue and
> > why
> > > is
> > > > it failing in local mode ?
> > > > Also, i couldn't get why only local mode is mentioned specifically ?
> > Does
> > > > it not affect when operator runs on the cluster ?
> > > >
> > > > Thanks,
> > > > Shubham
> > > >
> > > > On Tue, Jan 26, 2016 at 7:40 PM, Pramod Immaneni <
> > pramod@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > > > Siyuan can you explain the statement some more "It would fail in
> > local
> > > > mode
> > > > > if it tries to load class that only in app package classpath."
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Mon, Jan 25, 2016 at 4:52 PM, Siyuan Hua <
> siyuan@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Sandesh and I found an issue in almost all the operators that
use
> > > Kryo
> > > > to
> > > > > > clone operator itself  in definePartition method.  They all
use
> > > default
> > > > > > Kryo instance without setting context classloader from the thread
> > > that
> > > > > > execute the code. It would fail in local mode if it tries to
load
> > > class
> > > > > > that only in app package classpath.
> > > > > >
> > > > > > When I'm trying to fix it in all the places, I found repeated
> > pattern
> > > > in
> > > > > > definePartitions. And it's very error-prone since no one actually
> > did
> > > > it
> > > > > in
> > > > > > right way until we found the issue.  I'm thinking to make a
Util
> > > > functin
> > > > > to
> > > > > > create instance from operator itself and also hide the Kryo
> > > dependency
> > > > > from
> > > > > > the operator.  Do you have any better suggestions to do this?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Siyuan
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message