apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siyuan Hua <siy...@datatorrent.com>
Subject Re: Kryo misuse in definePartitions method
Date Tue, 26 Jan 2016 17:39:45 GMT
@Pramod

My understanding of local mode is the application is run in the same
process as dtcli, so the default classloader can only load the classes in
dtcli classpath.
When it tries to launch the application that has other dependencies, in
StramAppLauncher.loadDependencies() method, it tries to create a new
ClassLoader to include those libs and set it as context classloader.
So if the operator needs to load classes in it's own dependencies, it needs
to get that context classloader.

On Tue, Jan 26, 2016 at 8:46 AM, Pramod Immaneni <pramod@datatorrent.com>
wrote:

> The JIRA does not explain why it happens. Can you add the explanation. It
> might be useful if similar situations arise in future in other scenarios.
>
> On Tue, Jan 26, 2016 at 7:17 AM, Sandesh Hegde <sandesh@datatorrent.com>
> wrote:
>
> > Here is the jira for the issue
> > https://issues.apache.org/jira/browse/APEXMALHAR-1984
> >
> > It affects only local mode not on the cluster. I was trying it on
> > DataTorrent Sandbox, in that machine local mode failed and passed on
> yarn.
> >
> > On Tue, Jan 26, 2016 at 7:13 AM Shubham Pathak <shubham@datatorrent.com>
> > wrote:
> >
> > > Could we have  some more details as to what exactly is the issue and
> why
> > is
> > > it failing in local mode ?
> > > Also, i couldn't get why only local mode is mentioned specifically ?
> Does
> > > it not affect when operator runs on the cluster ?
> > >
> > > Thanks,
> > > Shubham
> > >
> > > On Tue, Jan 26, 2016 at 7:40 PM, Pramod Immaneni <
> pramod@datatorrent.com
> > >
> > > wrote:
> > >
> > > > Siyuan can you explain the statement some more "It would fail in
> local
> > > mode
> > > > if it tries to load class that only in app package classpath."
> > > >
> > > > Thanks
> > > >
> > > > On Mon, Jan 25, 2016 at 4:52 PM, Siyuan Hua <siyuan@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Sandesh and I found an issue in almost all the operators that use
> > Kryo
> > > to
> > > > > clone operator itself  in definePartition method.  They all use
> > default
> > > > > Kryo instance without setting context classloader from the thread
> > that
> > > > > execute the code. It would fail in local mode if it tries to load
> > class
> > > > > that only in app package classpath.
> > > > >
> > > > > When I'm trying to fix it in all the places, I found repeated
> pattern
> > > in
> > > > > definePartitions. And it's very error-prone since no one actually
> did
> > > it
> > > > in
> > > > > right way until we found the issue.  I'm thinking to make a Util
> > > functin
> > > > to
> > > > > create instance from operator itself and also hide the Kryo
> > dependency
> > > > from
> > > > > the operator.  Do you have any better suggestions to do this?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Siyuan
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message