apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siyuan Hua <siy...@datatorrent.com>
Subject Re: Kryo misuse in definePartitions method
Date Tue, 26 Jan 2016 21:10:10 GMT
Fix in pull request, please review
https://github.com/apache/incubator-apex-malhar/pull/178

On Tue, Jan 26, 2016 at 10:58 AM, Sandesh Hegde <sandesh@datatorrent.com>
wrote:

> Siyuan,
>
> This needs to be fixed in local mode. App runs successfully on Yarn, but
> fails in local mode.
>
> Thoughts ?
>
> Thanks
> Sandesh
>
> On Tue, Jan 26, 2016 at 9:55 AM Shubham Pathak <shubham@datatorrent.com>
> wrote:
>
> > Thanks Siyuan for the explanation !
> >
> > On Tue, Jan 26, 2016 at 11:11 PM, Pramod Immaneni <
> pramod@datatorrent.com>
> > wrote:
> >
> > > That is a good explanation, can you add it to the JIRA as well.
> > >
> > > Thanks
> > >
> > > On Tue, Jan 26, 2016 at 9:39 AM, Siyuan Hua <siyuan@datatorrent.com>
> > > wrote:
> > >
> > > > @Pramod
> > > >
> > > > My understanding of local mode is the application is run in the same
> > > > process as dtcli, so the default classloader can only load the
> classes
> > in
> > > > dtcli classpath.
> > > > When it tries to launch the application that has other dependencies,
> in
> > > > StramAppLauncher.loadDependencies() method, it tries to create a new
> > > > ClassLoader to include those libs and set it as context classloader.
> > > > So if the operator needs to load classes in it's own dependencies, it
> > > needs
> > > > to get that context classloader.
> > > >
> > > > On Tue, Jan 26, 2016 at 8:46 AM, Pramod Immaneni <
> > pramod@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > > > The JIRA does not explain why it happens. Can you add the
> > explanation.
> > > It
> > > > > might be useful if similar situations arise in future in other
> > > scenarios.
> > > > >
> > > > > On Tue, Jan 26, 2016 at 7:17 AM, Sandesh Hegde <
> > > sandesh@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Here is the jira for the issue
> > > > > > https://issues.apache.org/jira/browse/APEXMALHAR-1984
> > > > > >
> > > > > > It affects only local mode not on the cluster. I was trying
it on
> > > > > > DataTorrent Sandbox, in that machine local mode failed and passed
> > on
> > > > > yarn.
> > > > > >
> > > > > > On Tue, Jan 26, 2016 at 7:13 AM Shubham Pathak <
> > > > shubham@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Could we have  some more details as to what exactly is
the
> issue
> > > and
> > > > > why
> > > > > > is
> > > > > > > it failing in local mode ?
> > > > > > > Also, i couldn't get why only local mode is mentioned
> > specifically
> > > ?
> > > > > Does
> > > > > > > it not affect when operator runs on the cluster ?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Shubham
> > > > > > >
> > > > > > > On Tue, Jan 26, 2016 at 7:40 PM, Pramod Immaneni <
> > > > > pramod@datatorrent.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Siyuan can you explain the statement some more "It
would fail
> > in
> > > > > local
> > > > > > > mode
> > > > > > > > if it tries to load class that only in app package
> classpath."
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Mon, Jan 25, 2016 at 4:52 PM, Siyuan Hua <
> > > > siyuan@datatorrent.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Sandesh and I found an issue in almost all the
operators
> that
> > > use
> > > > > > Kryo
> > > > > > > to
> > > > > > > > > clone operator itself  in definePartition method.
 They all
> > use
> > > > > > default
> > > > > > > > > Kryo instance without setting context classloader
from the
> > > thread
> > > > > > that
> > > > > > > > > execute the code. It would fail in local mode
if it tries
> to
> > > load
> > > > > > class
> > > > > > > > > that only in app package classpath.
> > > > > > > > >
> > > > > > > > > When I'm trying to fix it in all the places,
I found
> repeated
> > > > > pattern
> > > > > > > in
> > > > > > > > > definePartitions. And it's very error-prone since
no one
> > > actually
> > > > > did
> > > > > > > it
> > > > > > > > in
> > > > > > > > > right way until we found the issue.  I'm thinking
to make a
> > > Util
> > > > > > > functin
> > > > > > > > to
> > > > > > > > > create instance from operator itself and also
hide the Kryo
> > > > > > dependency
> > > > > > > > from
> > > > > > > > > the operator.  Do you have any better suggestions
to do
> this?
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > Siyuan
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message