Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAOi1F6Xi47LKO1k_QSXdGemUH6d4Wa8MN6nVA3w3GbQE8EybJQ@mail.gmail.com>
References: 
 <CAOi1F6XAC4nwbA6VotkaYGZ-W6OdWRWk+SpMLvs_324YFXYXuw@mail.gmail.com>
 <CAA7LveG1FgEt4jH8UZNaKm-PiuVeKBoVyfsh2T2tAya-DuWaEQ@mail.gmail.com>
 <CANMVRknz4Sf7j_Wp_VisOoh0Ef-UaYMExz75G_VK4t-U9SxaBQ@mail.gmail.com>
 <CAJhKGzrbOJnC3Akf=aP6ZE2R-EX4uCrPXbCewkN3m66yVmMnDA@mail.gmail.com>
 <CAA7LveFmypeesqEdVmL1zf5=hGwx3pST6D41bnhXDrrdLTqgDw@mail.gmail.com>
 <CAOi1F6Xi47LKO1k_QSXdGemUH6d4Wa8MN6nVA3w3GbQE8EybJQ@mail.gmail.com>
From: Pramod Immaneni <pramod@datatorrent.com>
Date: Tue, 26 Jan 2016 09:41:29 -0800
Message-ID: 
 <CAA7LveGo1q5H6yaYKNS-pO9-6-evn0fiLg20spNv+OkoSECAxg@mail.gmail.com>
Subject: Re: Kryo misuse in definePartitions method
To: dev@apex.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a1139126ccdc4e8052a40332d

--001a1139126ccdc4e8052a40332d
Content-Type: text/plain; charset=UTF-8

That is a good explanation, can you add it to the JIRA as well.

Thanks

On Tue, Jan 26, 2016 at 9:39 AM, Siyuan Hua <siyuan@datatorrent.com> wrote:

> @Pramod
>
> My understanding of local mode is the application is run in the same
> process as dtcli, so the default classloader can only load the classes in
> dtcli classpath.
> When it tries to launch the application that has other dependencies, in
> StramAppLauncher.loadDependencies() method, it tries to create a new
> ClassLoader to include those libs and set it as context classloader.
> So if the operator needs to load classes in it's own dependencies, it needs
> to get that context classloader.
>
> On Tue, Jan 26, 2016 at 8:46 AM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>
> > The JIRA does not explain why it happens. Can you add the explanation. It
> > might be useful if similar situations arise in future in other scenarios.
> >
> > On Tue, Jan 26, 2016 at 7:17 AM, Sandesh Hegde <sandesh@datatorrent.com>
> > wrote:
> >
> > > Here is the jira for the issue
> > > https://issues.apache.org/jira/browse/APEXMALHAR-1984
> > >
> > > It affects only local mode not on the cluster. I was trying it on
> > > DataTorrent Sandbox, in that machine local mode failed and passed on
> > yarn.
> > >
> > > On Tue, Jan 26, 2016 at 7:13 AM Shubham Pathak <
> shubham@datatorrent.com>
> > > wrote:
> > >
> > > > Could we have  some more details as to what exactly is the issue and
> > why
> > > is
> > > > it failing in local mode ?
> > > > Also, i couldn't get why only local mode is mentioned specifically ?
> > Does
> > > > it not affect when operator runs on the cluster ?
> > > >
> > > > Thanks,
> > > > Shubham
> > > >
> > > > On Tue, Jan 26, 2016 at 7:40 PM, Pramod Immaneni <
> > pramod@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > > > Siyuan can you explain the statement some more "It would fail in
> > local
> > > > mode
> > > > > if it tries to load class that only in app package classpath."
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Mon, Jan 25, 2016 at 4:52 PM, Siyuan Hua <
> siyuan@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Sandesh and I found an issue in almost all the operators that use
> > > Kryo
> > > > to
> > > > > > clone operator itself  in definePartition method.  They all use
> > > default
> > > > > > Kryo instance without setting context classloader from the thread
> > > that
> > > > > > execute the code. It would fail in local mode if it tries to load
> > > class
> > > > > > that only in app package classpath.
> > > > > >
> > > > > > When I'm trying to fix it in all the places, I found repeated
> > pattern
> > > > in
> > > > > > definePartitions. And it's very error-prone since no one actually
> > did
> > > > it
> > > > > in
> > > > > > right way until we found the issue.  I'm thinking to make a Util
> > > > functin
> > > > > to
> > > > > > create instance from operator itself and also hide the Kryo
> > > dependency
> > > > > from
> > > > > > the operator.  Do you have any better suggestions to do this?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Siyuan
> > > > > >
> > > > >
> > > >
> > >
> >
>

--001a1139126ccdc4e8052a40332d--