pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheolsoo Park <piaozhe...@gmail.com>
Subject Re: [Discussion] Keep or revert PIG-3419 in trunk?
Date Thu, 17 Oct 2013 17:34:43 GMT
To close the loop, here is the current status of issues that PIG-3419
introduced-

1) Backward incompatibility: Daniel provided a clever solution (PIG-3520).
In summary, JobStats and PigStatsUtil will keep MR-specific methods as
abstract methods. All the sub-classes that extend them (e.g.
SimpleJobStats, TezJobStats, etc) must implement those methods. Non-MR
classes will simply throw a "not implemented" exception for those methods.

2) HangingJobKiller: PIG-3524 brings it back to the launcher. Pending
review.

3) Explain shows unoptimized plan: PIG-3508 keeps track of the issue. To be
fixed.

4) PigStats.get() and ScriptState.get() return MR-specific objects:
PIG-3525 keeps track of the issue. To be fixed.

Given the current status, I think we can keep PIG-3419 in trunk. Please let
me know if anyone thinks otherwise.

Thanks,
Cheolsoo

On Tue, Oct 15, 2013 at 1:54 PM, Daniel Dai <daijy@hortonworks.com> wrote:

> I still want to try a little bit. Give me one more day.
>
> Thanks,
> Daniel
>
>
> On Tue, Oct 15, 2013 at 12:40 PM, Jeremy Karn <jkarn@mortardata.com>
> wrote:
>
> > I'm +1 on reverting PIG-3419 on trunk and then trying to redo it in a
> > better way.  Either way it would be good to get this resolved soon.
> >
> >
> > On Mon, Oct 7, 2013 at 5:49 PM, Daniel Dai <daijy@hortonworks.com>
> wrote:
> >
> > > Let me first revert PIG-3457, that complicates things. I will try to
> > > explore for a solution in next the few days. But if we cannot settle
> > > quickly, I would suggest to revert PIG-3419 on trunk first, and redo it
> > > later in a better way. Otherwise, it will be very hard to revert.
> > >
> > > Thanks,
> > > Daniel
> > >
> > >
> > > On Mon, Oct 7, 2013 at 12:05 PM, Cheolsoo Park <piaozhexiu@gmail.com>
> > > wrote:
> > >
> > > > Hi devs,
> > > >
> > > > This is a follow-up discussion about how to resolve the backward
> > > > incompatibility of PIG-3419 (Pluggable execution engine). Per the
> > > previous
> > > > discussion <http://search-hadoop.com/m/wYz6hz9CoE>, I reverted it
in
> > > 0.12
> > > > but kept it in trunk. As we keep committing more changes into trunk,
> it
> > > > gets harder to back out PIG-3419 cleanly. So I suggest we should
> make a
> > > > decision sooner rather than later.
> > > >
> > > > The crux of the problem is as follows:
> > > >
> > > > PIG-3419 removes all the MR-specific things from JobStats. However,
> > > > PigRunner and PigServer returns PigStats that in turn exposes
> JobStats
> > to
> > > > end users. So changing JobStats breaks backward compatibility for
> > > > downstream projects such as Oozie. While changing the semantics of
> > > JobStats
> > > > is acceptable, we must provide a deprecation path so that end users
> can
> > > > upgrade their applications smoothly.
> > > >
> > > > Proposed solutions:
> > > >
> > > > 1) Provide backward compatibility in source code: Maybe possible, but
> > no
> > > > one has come up with a clean solution. For eg, I failed. :(
> > > >
> > > > 2) Publish two jars: We keep PIG-3419 only in tez-branch and publish
> > two
> > > > jars (one for old API and one for new API) for a couple of future
> > > releases.
> > > > If we do this, we're going to revert PIG-3419 in trunk.
> > > >
> > > > What do you think?
> > > >
> > > > Thanks,
> > > > Cheolsoo
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
> >
> >
> > --
> >
> > Jeremy Karn / Lead Developer
> > MORTAR DATA / 519 277 4391 / www.mortardata.com
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message