apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandesh Hegde <sand...@datatorrent.com>
Subject Re: Kryo misuse in definePartitions method
Date Tue, 26 Jan 2016 18:58:44 GMT
Siyuan,

This needs to be fixed in local mode. App runs successfully on Yarn, but
fails in local mode.

Thoughts ?

Thanks
Sandesh

On Tue, Jan 26, 2016 at 9:55 AM Shubham Pathak <shubham@datatorrent.com>
wrote:

> Thanks Siyuan for the explanation !
>
> On Tue, Jan 26, 2016 at 11:11 PM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>
> > That is a good explanation, can you add it to the JIRA as well.
> >
> > Thanks
> >
> > On Tue, Jan 26, 2016 at 9:39 AM, Siyuan Hua <siyuan@datatorrent.com>
> > wrote:
> >
> > > @Pramod
> > >
> > > My understanding of local mode is the application is run in the same
> > > process as dtcli, so the default classloader can only load the classes
> in
> > > dtcli classpath.
> > > When it tries to launch the application that has other dependencies, in
> > > StramAppLauncher.loadDependencies() method, it tries to create a new
> > > ClassLoader to include those libs and set it as context classloader.
> > > So if the operator needs to load classes in it's own dependencies, it
> > needs
> > > to get that context classloader.
> > >
> > > On Tue, Jan 26, 2016 at 8:46 AM, Pramod Immaneni <
> pramod@datatorrent.com
> > >
> > > wrote:
> > >
> > > > The JIRA does not explain why it happens. Can you add the
> explanation.
> > It
> > > > might be useful if similar situations arise in future in other
> > scenarios.
> > > >
> > > > On Tue, Jan 26, 2016 at 7:17 AM, Sandesh Hegde <
> > sandesh@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Here is the jira for the issue
> > > > > https://issues.apache.org/jira/browse/APEXMALHAR-1984
> > > > >
> > > > > It affects only local mode not on the cluster. I was trying it on
> > > > > DataTorrent Sandbox, in that machine local mode failed and passed
> on
> > > > yarn.
> > > > >
> > > > > On Tue, Jan 26, 2016 at 7:13 AM Shubham Pathak <
> > > shubham@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Could we have  some more details as to what exactly is the issue
> > and
> > > > why
> > > > > is
> > > > > > it failing in local mode ?
> > > > > > Also, i couldn't get why only local mode is mentioned
> specifically
> > ?
> > > > Does
> > > > > > it not affect when operator runs on the cluster ?
> > > > > >
> > > > > > Thanks,
> > > > > > Shubham
> > > > > >
> > > > > > On Tue, Jan 26, 2016 at 7:40 PM, Pramod Immaneni <
> > > > pramod@datatorrent.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Siyuan can you explain the statement some more "It would
fail
> in
> > > > local
> > > > > > mode
> > > > > > > if it tries to load class that only in app package classpath."
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Mon, Jan 25, 2016 at 4:52 PM, Siyuan Hua <
> > > siyuan@datatorrent.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Sandesh and I found an issue in almost all the operators
that
> > use
> > > > > Kryo
> > > > > > to
> > > > > > > > clone operator itself  in definePartition method.
 They all
> use
> > > > > default
> > > > > > > > Kryo instance without setting context classloader
from the
> > thread
> > > > > that
> > > > > > > > execute the code. It would fail in local mode if it
tries to
> > load
> > > > > class
> > > > > > > > that only in app package classpath.
> > > > > > > >
> > > > > > > > When I'm trying to fix it in all the places, I found
repeated
> > > > pattern
> > > > > > in
> > > > > > > > definePartitions. And it's very error-prone since
no one
> > actually
> > > > did
> > > > > > it
> > > > > > > in
> > > > > > > > right way until we found the issue.  I'm thinking
to make a
> > Util
> > > > > > functin
> > > > > > > to
> > > > > > > > create instance from operator itself and also hide
the Kryo
> > > > > dependency
> > > > > > > from
> > > > > > > > the operator.  Do you have any better suggestions
to do this?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > Siyuan
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message