spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hamstra <m...@clearstorydata.com>
Subject Re: Spark Improvement Proposals
Date Mon, 10 Oct 2016 22:36:32 GMT
If I'm correctly understanding the kind of voting that you are talking
about, then to be accurate, it is only the PMC members that have a vote,
not all committers:
https://www.apache.org/foundation/how-it-works.html#pmc-members

On Mon, Oct 10, 2016 at 12:02 PM, Cody Koeninger <cody@koeninger.org> wrote:

> I think the main value is in being honest about what's going on.  No
> one other than committers can cast a meaningful vote, that's the
> reality.  Beyond that, if people think it's more open to allow formal
> proposals from anyone, I'm not necessarily against it, but my main
> question would be this:
>
> If anyone can submit a proposal, are committers actually going to
> clearly reject and close proposals that don't meet the requirements?
>
> Right now we have a serious problem with lack of clarity regarding
> contributions, and that cannot spill over into goal-setting.
>
> On Mon, Oct 10, 2016 at 1:54 PM, Ryan Blue <rblue@netflix.com> wrote:
> > +1 to votes to approve proposals. I agree that proposals should have an
> > official mechanism to be accepted, and a vote is an established means of
> > doing that well. I like that it includes a period to review the proposal
> and
> > I think proposals should have been discussed enough ahead of a vote to
> > survive the possibility of a veto.
> >
> > I also like the names that are short and (mostly) unique, like SEP.
> >
> > Where I disagree is with the requirement that a committer must formally
> > propose an enhancement. I don't see the value of restricting this: if
> > someone has the will to write up a proposal then they should be
> encouraged
> > to do so and start a discussion about it. Even if there is a political
> > reality as Cody says, what is the value of codifying that in our
> process? I
> > think restricting who can submit proposals would only undermine them by
> > pushing contributors out. Maybe I'm missing something here?
> >
> > rb
> >
> >
> >
> > On Mon, Oct 10, 2016 at 7:41 AM, Cody Koeninger <cody@koeninger.org>
> wrote:
> >>
> >> Yes, users suggesting SIPs is a good thing and is explicitly called
> >> out in the linked document under the Who? section.  Formally proposing
> >> them, not so much, because of the political realities.
> >>
> >> Yes, implementation strategy definitely affects goals.  There are all
> >> kinds of examples of this, I'll pick one that's my fault so as to
> >> avoid sounding like I'm blaming:
> >>
> >> When I implemented the Kafka DStream, one of my (not explicitly agreed
> >> upon by the community) goals was to make sure people could use the
> >> Dstream with however they were already using Kafka at work.  The lack
> >> of explicit agreement on that goal led to all kinds of fighting with
> >> committers, that could have been avoided.  The lack of explicit
> >> up-front strategy discussion led to the DStream not really working
> >> with compacted topics.  I knew about compacted topics, but don't have
> >> a use for them, so had a blind spot there.  If there was explicit
> >> up-front discussion that my strategy was "assume that batches can be
> >> defined on the driver solely by beginning and ending offsets", there's
> >> a greater chance that a user would have seen that and said, "hey, what
> >> about non-contiguous offsets in a compacted topic".
> >>
> >> This kind of thing is only going to happen smoothly if we have a
> >> lightweight user-visible process with clear outcomes.
> >>
> >> On Mon, Oct 10, 2016 at 1:34 AM, assaf.mendelson
> >> <assaf.mendelson@rsa.com> wrote:
> >> > I agree with most of what Cody said.
> >> >
> >> > Two things:
> >> >
> >> > First we can always have other people suggest SIPs but mark them as
> >> > “unreviewed” and have committers basically move them forward. The
> >> > problem is
> >> > that writing a good document takes time. This way we can leverage non
> >> > committers to do some of this work (it is just another way to
> >> > contribute).
> >> >
> >> >
> >> >
> >> > As for strategy, in many cases implementation strategy can affect the
> >> > goals.
> >> > I will give  a small example: In the current structured streaming
> >> > strategy,
> >> > we group by the time to achieve a sliding window. This is definitely
> an
> >> > implementation decision and not a goal. However, I can think of
> several
> >> > aggregation functions which have the time inside their calculation
> >> > buffer.
> >> > For example, let’s say we want to return a set of all distinct values.
> >> > One
> >> > way to implement this would be to make the set into a map and have the
> >> > value
> >> > contain the last time seen. Multiplying it across the groupby would
> cost
> >> > a
> >> > lot in performance. So adding such a strategy would have a great
> effect
> >> > on
> >> > the type of aggregations and their performance which does affect the
> >> > goal.
> >> > Without adding the strategy, it is easy for whoever goes to the design
> >> > document to not think about these cases. Furthermore, it might be
> >> > decided
> >> > that these cases are rare enough so that the strategy is still good
> >> > enough
> >> > but how would we know it without user feedback?
> >> >
> >> > I believe this example is exactly what Cody was talking about. Since
> >> > many
> >> > times implementation strategies have a large effect on the goal, we
> >> > should
> >> > have it discussed when discussing the goals. In addition, while it is
> >> > often
> >> > easy to throw out completely infeasible goals, it is often much harder
> >> > to
> >> > figure out that the goals are unfeasible without fine tuning.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Assaf.
> >> >
> >> >
> >> >
> >> > From: Cody Koeninger-2 [via Apache Spark Developers List]
> >> > [mailto:ml-node+[hidden email]]
> >> > Sent: Monday, October 10, 2016 2:25 AM
> >> > To: Mendelson, Assaf
> >> > Subject: Re: Spark Improvement Proposals
> >> >
> >> >
> >> >
> >> > Only committers should formally submit SIPs because in an apache
> >> > project only commiters have explicit political power.  If a user can't
> >> > find a commiter willing to sponsor an SIP idea, they have no way to
> >> > get the idea passed in any case.  If I can't find a committer to
> >> > sponsor this meta-SIP idea, I'm out of luck.
> >> >
> >> > I do not believe unrealistic goals can be found solely by inspection.
> >> > We've managed to ignore unrealistic goals even after implementation!
> >> > Focusing on APIs can allow people to think they've solved something,
> >> > when there's really no way of implementing that API while meeting the
> >> > goals.  Rapid iteration is clearly the best way to address this, but
> >> > we've already talked about why that hasn't really worked.  If adding a
> >> > non-binding API section to the template is important to you, I'm not
> >> > against it, but I don't think it's sufficient.
> >> >
> >> > On your PRD vs design doc spectrum, I'm saying this is closer to a
> >> > PRD.  Clear agreement on goals is the most important thing and that's
> >> > why it's the thing I want binding agreement on.  But I cannot agree to
> >> > goals unless I have enough minimal technical info to judge whether the
> >> > goals are likely to actually be accomplished.
> >> >
> >> >
> >> >
> >> > On Sun, Oct 9, 2016 at 5:35 PM, Matei Zaharia <[hidden email]> wrote:
> >> >
> >> >
> >> >> Well, I think there are a few things here that don't make sense.
> First,
> >> >> why
> >> >> should only committers submit SIPs? Development in the project should
> >> >> be
> >> >> open to all contributors, whether they're committers or not. Second,
> I
> >> >> think
> >> >> unrealistic goals can be found just by inspecting the goals, and I'm
> >> >> not
> >> >> super worried that we'll accept a lot of SIPs that are then
> infeasible
> >> >> --
> >> >> we
> >> >> can then submit new ones. But this depends on whether you want this
> >> >> process
> >> >> to be a "design doc lite", where people also agree on implementation
> >> >> strategy, or just a way to agree on goals. This is what I asked
> earlier
> >> >> about PRDs vs design docs (and I'm open to either one but I'd just
> like
> >> >> clarity). Finally, both as a user and designer of software, I always
> >> >> want
> >> >> to
> >> >> give feedback on APIs, so I'd really like a culture of having those
> >> >> early.
> >> >> People don't argue about prettiness when they discuss APIs, they
> argue
> >> >> about
> >> >> the core concepts to expose in order to meet various goals, and then
> >> >> they're
> >> >> stuck maintaining those for a long time.
> >> >>
> >> >> Matei
> >> >>
> >> >> On Oct 9, 2016, at 3:10 PM, Cody Koeninger <[hidden email]> wrote:
> >> >>
> >> >> Users instead of people, sure.  Commiters and contributors are (or at
> >> >> least
> >> >> should be) a subset of users.
> >> >>
> >> >> Non goals, sure. I don't care what the name is, but we need to
> clearly
> >> >> say
> >> >> e.g. 'no we are not maintaining compatibility with XYZ right now'.
> >> >>
> >> >> API, what I care most about is whether it allows me to accomplish the
> >> >> goals.
> >> >> Arguing about how ugly or pretty it is can be saved for design/
> >> >> implementation imho.
> >> >>
> >> >> Strategy, this is necessary because otherwise goals can be out of
> line
> >> >> with
> >> >> reality.  Don't propose goals you don't have at least some idea of
> how
> >> >> to
> >> >> implement.
> >> >>
> >> >> Rejected strategies, given that commiters are the only ones I'm
> saying
> >> >> should formally submit SPARKLIs or SIPs, if they put junk in a
> required
> >> >> section then slap them down for it and tell them to fix it.
> >> >>
> >> >>
> >> >> On Oct 9, 2016 4:36 PM, "Matei Zaharia" <[hidden email]> wrote:
> >> >>>
> >> >>> Yup, this is the stuff that I found unclear. Thanks for clarifying
> >> >>> here,
> >> >>> but we should also clarify it in the writeup. In particular:
> >> >>>
> >> >>> - Goals needs to be about user-facing behavior ("people" is broad)
> >> >>>
> >> >>> - I'd rename Rejected Goals to Non-Goals. Otherwise someone will dig
> >> >>> up
> >> >>> one of these and say "Spark's developers have officially rejected X,
> >> >>> which
> >> >>> our awesome system has".
> >> >>>
> >> >>> - For user-facing stuff, I think you need a section on API.
> Virtually
> >> >>> all
> >> >>> other *IPs I've seen have that.
> >> >>>
> >> >>> - I'm still not sure why the strategy section is needed if the
> purpose
> >> >>> is
> >> >>> to define user-facing behavior -- unless this is the strategy for
> >> >>> setting
> >> >>> the goals or for defining the API. That sounds squarely like a
> design
> >> >>> doc
> >> >>> issue. In some sense, who cares whether the proposal is technically
> >> >>> feasible
> >> >>> right now? If it's infeasible, that will be discovered later during
> >> >>> design
> >> >>> and implementation. Same thing with rejected strategies -- listing
> >> >>> some
> >> >>> of
> >> >>> those is definitely useful sometimes, but if you make this a
> >> >>> *required*
> >> >>> section, people are just going to fill it in with bogus stuff (I've
> >> >>> seen
> >> >>> this happen before).
> >> >>>
> >> >>> Matei
> >> >>>
> >> >
> >> >>> > On Oct 9, 2016, at 2:14 PM, Cody Koeninger <[hidden email]> wrote:
> >> >>> >
> >> >>> > So to focus the discussion on the specific strategy I'm
> suggesting,
> >> >>> > documented at
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark-
> improvement-proposals.md
> >> >>> >
> >> >>> > "Goals: What must this allow people to do, that they can't
> >> >>> > currently?"
> >> >>> >
> >> >>> > Is it unclear that this is focusing specifically on people-visible
> >> >>> > behavior?
> >> >>> >
> >> >>> > Rejected goals -  are important because otherwise people keep
> trying
> >> >>> > to argue about scope.  Of course you can change things later with
> a
> >> >>> > different SIP and different vote, the point is to focus.
> >> >>> >
> >> >>> > Use cases - are something that people are going to bring up in
> >> >>> > discussion.  If they aren't clearly documented as a goal ("This
> must
> >> >>> > allow me to connect using SSL"), they should be added.
> >> >>> >
> >> >>> > Internal architecture - if the people who need specific behavior
> are
> >> >>> > implementers of other parts of the system, that's fine.
> >> >>> >
> >> >>> > Rejected strategies - If you have none of these, you have no
> >> >>> > evidence
> >> >>> > that the proponent didn't just go with the first thing they had in
> >> >>> > mind (or have already implemented), which is a big problem
> >> >>> > currently.
> >> >>> > Approval isn't binding as to specifics of implementation, so these
> >> >>> > aren't handcuffs.  The goals are the contract, the strategy is
> >> >>> > evidence that contract can actually be met.
> >> >>> >
> >> >>> > Design docs - I'm not touching design docs.  The markdown file I
> >> >>> > linked specifically says of the strategy section "This is not a
> full
> >> >>> > design document."  Is this unclear?  Design docs can be worked on
> >> >>> > obviously, but that's not what I'm concerned with here.
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > On Sun, Oct 9, 2016 at 2:34 PM, Matei Zaharia <[hidden email]>
> >> >>> > wrote:
> >> >>> >> Hi Cody,
> >> >>> >>
> >> >>> >> I think this would be a lot more concrete if we had a more
> detailed
> >> >>> >> template
> >> >>> >> for SIPs. Right now, it's not super clear what's in scope -- e.g.
> >> >>> >> are
> >> >>> >> they
> >> >>> >> a way to solicit feedback on the user-facing behavior or on the
> >> >>> >> internals?
> >> >>> >> "Goals" can cover both things. I've been thinking of SIPs more as
> >> >>> >> Product
> >> >>> >> Requirements Docs (PRDs), which focus on *what* a code change
> >> >>> >> should
> >> >>> >> do
> >> >>> >> as
> >> >>> >> opposed to how.
> >> >>> >>
> >> >>> >> In particular, here are some things that you may or may not
> >> >>> >> consider
> >> >>> >> in
> >> >>> >> scope for SIPs:
> >> >>> >>
> >> >>> >> - Goals and non-goals: This is definitely in scope, and IMO
> should
> >> >>> >> focus on
> >> >>> >> user-visible behavior (e.g. "system supports SQL window
> functions"
> >> >>> >> or
> >> >>> >> "system continues working if one node fails"). BTW I wouldn't say
> >> >>> >> "rejected
> >> >>> >> goals" because some of them might become goals later, so we're
> not
> >> >>> >> definitively rejecting them.
> >> >>> >>
> >> >>> >> - Public API: Probably should be included in most SIPs unless
> it's
> >> >>> >> too
> >> >>> >> large
> >> >>> >> to fully specify then (e.g. "let's add an ML library").
> >> >>> >>
> >> >>> >> - Use cases: I usually find this very useful in PRDs to better
> >> >>> >> communicate
> >> >>> >> the goals.
> >> >>> >>
> >> >>> >> - Internal architecture: This is usually *not* a thing users can
> >> >>> >> easily
> >> >>> >> comment on and it sounds more like a design doc item. Of course
> >> >>> >> it's
> >> >>> >> important to show that the SIP is feasible to implement. One
> >> >>> >> exception,
> >> >>> >> however, is that I think we'll have some SIPs primarily on
> >> >>> >> internals
> >> >>> >> (e.g.
> >> >>> >> if somebody wants to refactor Spark's query optimizer or
> >> >>> >> something).
> >> >>> >>
> >> >>> >> - Rejected strategies: I personally wouldn't put this, because
> >> >>> >> what's
> >> >>> >> the
> >> >>> >> point of voting to reject a strategy before you've really begun
> >> >>> >> designing
> >> >>> >> and implementing something? What if you discover that the
> strategy
> >> >>> >> is
> >> >>> >> actually better when you start doing stuff?
> >> >>> >>
> >> >>> >> At a super high level, it depends on whether you want the SIPs to
> >> >>> >> be
> >> >>> >> PRDs
> >> >>> >> for getting some quick feedback on the goals of a feature before
> it
> >> >>> >> is
> >> >>> >> designed, or something more like full-fledged design docs (just a
> >> >>> >> more
> >> >>> >> visible design doc for bigger changes). I looked at Kafka's KIPs,
> >> >>> >> and
> >> >>> >> they
> >> >>> >> actually seem to be more like design docs. This can work too but
> it
> >> >>> >> does
> >> >>> >> require more work from the proposer and it can lead to the same
> >> >>> >> problems you
> >> >>> >> mentioned with people already having a design and implementation
> in
> >> >>> >> mind.
> >> >>> >>
> >> >>> >> Basically, the question is, are you trying to iterate faster on
> >> >>> >> design
> >> >>> >> by
> >> >>> >> adding a step for user feedback earlier? Or are you just trying
> to
> >> >>> >> make
> >> >>> >> design docs for key features more visible (and their approval
> more
> >> >>> >> formal)?
> >> >>> >>
> >> >>> >> BTW note that in either case, I'd like to have a template for
> >> >>> >> design
> >> >>> >> docs
> >> >>> >> too, which should also include goals. I think that would've
> avoided
> >> >>> >> some of
> >> >>> >> the issues you brought up.
> >> >>> >>
> >> >>> >> Matei
> >> >>> >>
> >> >>> >> On Oct 9, 2016, at 10:40 AM, Cody Koeninger <[hidden email]>
> wrote:
> >> >>> >>
> >> >>> >> Here's my specific proposal (meta-proposal?)
> >> >>> >>
> >> >>> >> Spark Improvement Proposals (SIP)
> >> >>> >>
> >> >>> >>
> >> >>> >> Background:
> >> >>> >>
> >> >>> >> The current problem is that design and implementation of large
> >> >>> >> features
> >> >>> >> are
> >> >>> >> often done in private, before soliciting user feedback.
> >> >>> >>
> >> >>> >> When feedback is solicited, it is often as to detailed design
> >> >>> >> specifics, not
> >> >>> >> focused on goals.
> >> >>> >>
> >> >>> >> When implementation does take place after design, there is often
> >> >>> >> disagreement as to what goals are or are not in scope.
> >> >>> >>
> >> >>> >> This results in commits that don't fully meet user needs.
> >> >>> >>
> >> >>> >>
> >> >>> >> Goals:
> >> >>> >>
> >> >>> >> - Ensure user, contributor, and committer goals are clearly
> >> >>> >> identified
> >> >>> >> and
> >> >>> >> agreed upon, before implementation takes place.
> >> >>> >>
> >> >>> >> - Ensure that a technically feasible strategy is chosen that is
> >> >>> >> likely
> >> >>> >> to
> >> >>> >> meet the goals.
> >> >>> >>
> >> >>> >>
> >> >>> >> Rejected Goals:
> >> >>> >>
> >> >>> >> - SIPs are not for detailed design.  Design by committee doesn't
> >> >>> >> work.
> >> >>> >>
> >> >>> >> - SIPs are not for every change.  We dont need that much process.
> >> >>> >>
> >> >>> >>
> >> >>> >> Strategy:
> >> >>> >>
> >> >>> >> My suggestion is outlined as a Spark Improvement Proposal process
> >> >>> >> documented
> >> >>> >> at
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark-
> improvement-proposals.md
> >> >>> >>
> >> >>> >> Specifics of Jira manipulation are an implementation detail we
> can
> >> >>> >> figure
> >> >>> >> out.
> >> >>> >>
> >> >>> >> I'm suggesting voting; the need here is for a _clear_ outcome.
> >> >>> >>
> >> >>> >>
> >> >>> >> Rejected Strategies:
> >> >>> >>
> >> >>> >> Having someone who understands the problem implement it first
> >> >>> >> works,
> >> >>> >> but
> >> >>> >> only if significant iteration after user feedback is allowed.
> >> >>> >>
> >> >>> >> Historically this has been problematic due to pressure to limit
> >> >>> >> public
> >> >>> >> api
> >> >>> >> changes.
> >> >>> >>
> >> >>> >>
> >> >>> >> On Fri, Oct 7, 2016 at 5:16 PM, Reynold Xin <[hidden email]>
> >> >>> >> wrote:
> >> >>> >>>
> >> >>> >>> Alright looks like there are quite a bit of support. We should
> >> >>> >>> wait
> >> >>> >>> to
> >> >>> >>> hear from more people too.
> >> >>> >>>
> >> >>> >>> To push this forward, Cody and I will be working together in the
> >> >>> >>> next
> >> >>> >>> couple of weeks to come up with a concrete, detailed proposal on
> >> >>> >>> what
> >> >>> >>> this
> >> >>> >>> entails, and then we can discuss this the specific proposal as
> >> >>> >>> well.
> >> >>> >>>
> >> >>> >>>
> >> >>> >>> On Fri, Oct 7, 2016 at 2:29 PM, Cody Koeninger <[hidden email]>
> >> >>> >>> wrote:
> >> >>> >>>>
> >> >>> >>>> Yeah, in case it wasn't clear, I was talking about SIPs for
> major
> >> >>> >>>> user-facing or cross-cutting changes, not minor feature adds.
> >> >>> >>>>
> >> >>> >>>> On Fri, Oct 7, 2016 at 3:58 PM, Stavros Kontopoulos
> >> >>> >>>> <[hidden email]> wrote:
> >> >>> >>>>>
> >> >>> >>>>> +1 to the SIP label as long as it does not slow down things
> and
> >> >>> >>>>> it
> >> >>> >>>>> targets optimizing efforts, coordination etc. For example
> really
> >> >>> >>>>> small
> >> >>> >>>>> features should not need to go through this process (assuming
> >> >>> >>>>> they
> >> >>> >>>>> dont
> >> >>> >>>>> touch public interfaces)  or re-factorings and hope it will be
> >> >>> >>>>> kept
> >> >>> >>>>> this
> >> >>> >>>>> way. So as a guideline doc should be provided, like in the KIP
> >> >>> >>>>> case.
> >> >>> >>>>>
> >> >>> >>>>> IMHO so far aside from tagging things and linking them
> elsewhere
> >> >>> >>>>> simply
> >> >>> >>>>> having design docs and prototypes implementations in PRs is
> not
> >> >>> >>>>> something
> >> >>> >>>>> that has not worked so far. What is really a pain in many
> >> >>> >>>>> projects
> >> >>> >>>>> out there
> >> >>> >>>>> is discontinuity in progress of PRs, missing features, slow
> >> >>> >>>>> reviews
> >> >>> >>>>> which is
> >> >>> >>>>> understandable to some extent... it is not only about Spark
> but
> >> >>> >>>>> things can
> >> >>> >>>>> be improved for sure for this project in particular as already
> >> >>> >>>>> stated.
> >> >>> >>>>>
> >> >>> >>>>> On Fri, Oct 7, 2016 at 11:14 PM, Cody Koeninger <[hidden
> email]>
> >> >>> >>>>> wrote:
> >> >>> >>>>>>
> >> >>> >>>>>> +1 to adding an SIP label and linking it from the website.  I
> >> >>> >>>>>> think
> >> >>> >>>>>> it
> >> >>> >>>>>> needs
> >> >>> >>>>>>
> >> >>> >>>>>> - template that focuses it towards soliciting user goals /
> non
> >> >>> >>>>>> goals
> >> >>> >>>>>> - clear resolution as to which strategy was chosen to pursue.
> >> >>> >>>>>> I'd
> >> >>> >>>>>> recommend a vote.
> >> >>> >>>>>>
> >> >>> >>>>>> Matei asked me to clarify what I meant by changing
> interfaces,
> >> >>> >>>>>> I
> >> >>> >>>>>> think
> >> >>> >>>>>> it's directly relevant to the SIP idea so I'll clarify here,
> >> >>> >>>>>> and
> >> >>> >>>>>> split
> >> >>> >>>>>> a thread for the other discussion per Nicholas' request.
> >> >>> >>>>>>
> >> >>> >>>>>> I meant changing public user interfaces.  I think the first
> >> >>> >>>>>> design
> >> >>> >>>>>> is
> >> >>> >>>>>> unlikely to be right, because it's done at a time when you
> have
> >> >>> >>>>>> the
> >> >>> >>>>>> least information.  As a user, I find it considerably more
> >> >>> >>>>>> frustrating
> >> >>> >>>>>> to be unable to use a tool to get my job done, than I do
> having
> >> >>> >>>>>> to
> >> >>> >>>>>> make minor changes to my code in order to take advantage of
> >> >>> >>>>>> features.
> >> >>> >>>>>> I've seen committers be seriously reluctant to allow changes
> to
> >> >>> >>>>>> @experimental code that are needed in order for it to really
> >> >>> >>>>>> work
> >> >>> >>>>>> right.  You need to be able to iterate, and if people on both
> >> >>> >>>>>> sides
> >> >>> >>>>>> of
> >> >>> >>>>>> the fence aren't going to respect that some newer apis are
> >> >>> >>>>>> subject
> >> >>> >>>>>> to
> >> >>> >>>>>> change, then why even mark them as such?
> >> >>> >>>>>>
> >> >>> >>>>>> Ideally a finished SIP should give me a checklist of things
> >> >>> >>>>>> that
> >> >>> >>>>>> an
> >> >>> >>>>>> implementation must do, and things that it doesn't need to
> do.
> >> >>> >>>>>> Contributors/committers should be seriously discouraged from
> >> >>> >>>>>> putting
> >> >>> >>>>>> out a version 0.1 that doesn't have at least a prototype
> >> >>> >>>>>> implementation of all those things, especially if they're
> then
> >> >>> >>>>>> going
> >> >>> >>>>>> to argue against interface changes necessary to get the the
> >> >>> >>>>>> rest
> >> >>> >>>>>> of
> >> >>> >>>>>> the things done in the 0.2 version.
> >> >>> >>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>> On Fri, Oct 7, 2016 at 2:18 PM, Reynold Xin <[hidden email]>
> >> >>> >>>>>> wrote:
> >> >>> >>>>>>> I like the lightweight proposal to add a SIP label.
> >> >>> >>>>>>>
> >> >>> >>>>>>> During Spark 2.0 development, Tom (Graves) and I suggested
> >> >>> >>>>>>> using
> >> >>> >>>>>>> wiki
> >> >>> >>>>>>> to
> >> >>> >>>>>>> track the list of major changes, but that never really
> >> >>> >>>>>>> materialized
> >> >>> >>>>>>> due to
> >> >>> >>>>>>> the overhead. Adding a SIP label on major JIRAs and then
> link
> >> >>> >>>>>>> to
> >> >>> >>>>>>> them
> >> >>> >>>>>>> prominently on the Spark website makes a lot of sense.
> >> >>> >>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>> On Fri, Oct 7, 2016 at 10:50 AM, Matei Zaharia
> >> >>> >>>>>>> <[hidden email]>
> >> >>> >>>>>>> wrote:
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> For the improvement proposals, I think one major point was
> to
> >> >>> >>>>>>>> make
> >> >>> >>>>>>>> them
> >> >>> >>>>>>>> really visible to users who are not contributors, so we
> >> >>> >>>>>>>> should
> >> >>> >>>>>>>> do
> >> >>> >>>>>>>> more than
> >> >>> >>>>>>>> sending stuff to dev@. One very lightweight idea is to
> have a
> >> >>> >>>>>>>> new
> >> >>> >>>>>>>> type of
> >> >>> >>>>>>>> JIRA called a SIP and have a link to a filter that shows
> all
> >> >>> >>>>>>>> such
> >> >>> >>>>>>>> JIRAs from
> >> >>> >>>>>>>> http://spark.apache.org. I also like the idea of SIP and
> >> >>> >>>>>>>> design
> >> >>> >>>>>>>> doc
> >> >>> >>>>>>>> templates (in fact many projects have them).
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> Matei
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> On Oct 7, 2016, at 10:38 AM, Reynold Xin <[hidden email]>
> >> >>> >>>>>>>> wrote:
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> I called Cody last night and talked about some of the
> topics
> >> >>> >>>>>>>> in
> >> >>> >>>>>>>> his
> >> >>> >>>>>>>> email.
> >> >>> >>>>>>>> It became clear to me Cody genuinely cares about the
> project.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> Some of the frustrations come from the success of the
> project
> >> >>> >>>>>>>> itself
> >> >>> >>>>>>>> becoming very "hot", and it is difficult to get clarity
> from
> >> >>> >>>>>>>> people
> >> >>> >>>>>>>> who
> >> >>> >>>>>>>> don't dedicate all their time to Spark. In fact, it is in
> >> >>> >>>>>>>> some
> >> >>> >>>>>>>> ways
> >> >>> >>>>>>>> similar
> >> >>> >>>>>>>> to scaling an engineering team in a successful startup: old
> >> >>> >>>>>>>> processes that
> >> >>> >>>>>>>> worked well might not work so well when it gets to a
> certain
> >> >>> >>>>>>>> size,
> >> >>> >>>>>>>> cultures
> >> >>> >>>>>>>> can get diluted, building culture vs building process, etc.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> I also really like to have a more visible process for
> larger
> >> >>> >>>>>>>> changes,
> >> >>> >>>>>>>> especially major user facing API changes. Historically we
> >> >>> >>>>>>>> upload
> >> >>> >>>>>>>> design docs
> >> >>> >>>>>>>> for major changes, but it is not always consistent and
> >> >>> >>>>>>>> difficult
> >> >>> >>>>>>>> to
> >> >>> >>>>>>>> quality
> >> >>> >>>>>>>> of the docs, due to the volunteering nature of the
> >> >>> >>>>>>>> organization.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> Some of the more concrete ideas we discussed focus on
> >> >>> >>>>>>>> building a
> >> >>> >>>>>>>> culture
> >> >>> >>>>>>>> to improve clarity:
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> - Process: Large changes should have design docs posted on
> >> >>> >>>>>>>> JIRA.
> >> >>> >>>>>>>> One
> >> >>> >>>>>>>> thing
> >> >>> >>>>>>>> Cody and I didn't discuss but an idea that just came to me
> is
> >> >>> >>>>>>>> we
> >> >>> >>>>>>>> should
> >> >>> >>>>>>>> create a design doc template for the project and ask
> >> >>> >>>>>>>> everybody
> >> >>> >>>>>>>> to
> >> >>> >>>>>>>> follow.
> >> >>> >>>>>>>> The design doc template should also explicitly list goals
> and
> >> >>> >>>>>>>> non-goals, to
> >> >>> >>>>>>>> make design doc more consistent.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> - Process: Email dev@ to solicit feedback. We have some
> this
> >> >>> >>>>>>>> with
> >> >>> >>>>>>>> some
> >> >>> >>>>>>>> changes, but again very inconsistent. Just posting
> something
> >> >>> >>>>>>>> on
> >> >>> >>>>>>>> JIRA
> >> >>> >>>>>>>> isn't
> >> >>> >>>>>>>> sufficient, because there are simply too many JIRAs and the
> >> >>> >>>>>>>> signal
> >> >>> >>>>>>>> get lost
> >> >>> >>>>>>>> in the noise. While this is generally impossible to enforce
> >> >>> >>>>>>>> because
> >> >>> >>>>>>>> we can't
> >> >>> >>>>>>>> force all volunteers to conform to a process (or they might
> >> >>> >>>>>>>> not
> >> >>> >>>>>>>> even
> >> >>> >>>>>>>> be
> >> >>> >>>>>>>> aware of this),  those who are more familiar with the
> project
> >> >>> >>>>>>>> can
> >> >>> >>>>>>>> help by
> >> >>> >>>>>>>> emailing the dev@ when they see something that hasn't
> been.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> - Culture: The design doc author(s) should be open to
> >> >>> >>>>>>>> feedback.
> >> >>> >>>>>>>> A
> >> >>> >>>>>>>> design
> >> >>> >>>>>>>> doc should serve as the base for discussion and is by no
> >> >>> >>>>>>>> means
> >> >>> >>>>>>>> the
> >> >>> >>>>>>>> final
> >> >>> >>>>>>>> design. Of course, this does not mean the author has to
> >> >>> >>>>>>>> accept
> >> >>> >>>>>>>> every
> >> >>> >>>>>>>> feedback. They should also be comfortable accepting /
> >> >>> >>>>>>>> rejecting
> >> >>> >>>>>>>> ideas on
> >> >>> >>>>>>>> technical grounds.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> - Process / Culture: For major ongoing projects, it can be
> >> >>> >>>>>>>> useful
> >> >>> >>>>>>>> to
> >> >>> >>>>>>>> have
> >> >>> >>>>>>>> some monthly Google hangouts that are open to the world. I
> am
> >> >>> >>>>>>>> actually not
> >> >>> >>>>>>>> sure how well this will work, because of the volunteering
> >> >>> >>>>>>>> nature
> >> >>> >>>>>>>> and
> >> >>> >>>>>>>> we need
> >> >>> >>>>>>>> to adjust for timezones for people across the globe, but it
> >> >>> >>>>>>>> seems
> >> >>> >>>>>>>> worth
> >> >>> >>>>>>>> trying.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> - Culture: Contributors (including committers) should be
> more
> >> >>> >>>>>>>> direct
> >> >>> >>>>>>>> in
> >> >>> >>>>>>>> setting expectations, including whether they are working
> on a
> >> >>> >>>>>>>> specific
> >> >>> >>>>>>>> issue, whether they will be working on a specific issue,
> and
> >> >>> >>>>>>>> whether
> >> >>> >>>>>>>> an
> >> >>> >>>>>>>> issue or pr or jira should be rejected. Most people I know
> in
> >> >>> >>>>>>>> this
> >> >>> >>>>>>>> community
> >> >>> >>>>>>>> are nice and don't enjoy telling other people no, but it is
> >> >>> >>>>>>>> often
> >> >>> >>>>>>>> more
> >> >>> >>>>>>>> annoying to a contributor to not know anything than
> getting a
> >> >>> >>>>>>>> no.
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>> On Fri, Oct 7, 2016 at 10:03 AM, Matei Zaharia
> >> >>> >>>>>>>> <[hidden email]>
> >> >>> >>>>>>>> wrote:
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>> Love the idea of a more visible "Spark Improvement
> Proposal"
> >> >>> >>>>>>>>> process that
> >> >>> >>>>>>>>> solicits user input on new APIs. For what it's worth, I
> >> >>> >>>>>>>>> don't
> >> >>> >>>>>>>>> think
> >> >>> >>>>>>>>> committers are trying to minimize their own work -- every
> >> >>> >>>>>>>>> committer
> >> >>> >>>>>>>>> cares
> >> >>> >>>>>>>>> about making the software useful for users. However, it is
> >> >>> >>>>>>>>> always
> >> >>> >>>>>>>>> hard to
> >> >>> >>>>>>>>> get user input and so it helps to have this kind of
> process.
> >> >>> >>>>>>>>> I've
> >> >>> >>>>>>>>> certainly
> >> >>> >>>>>>>>> looked at the *IPs a lot in other software I use just to
> see
> >> >>> >>>>>>>>> the
> >> >>> >>>>>>>>> biggest
> >> >>> >>>>>>>>> things on the roadmap.
> >> >>> >>>>>>>>>
> >> >>> >>>>>>>>> When you're talking about "changing interfaces", are you
> >> >>> >>>>>>>>> talking
> >> >>> >>>>>>>>> about
> >> >>> >>>>>>>>> public or internal APIs? I do think many people hate
> >> >>> >>>>>>>>> changing
> >> >>> >>>>>>>>> public APIs
> >> >>> >>>>>>>>> and I actually think that's for the best of the project.
> >> >>> >>>>>>>>> That's
> >> >>> >>>>>>>>> a
> >> >>> >>>>>>>>> technical
> >> >>> >>>>>>>>> debate, but basically, the worst thing when you're using a
> >> >>> >>>>>>>>> piece
> >> >>> >>>>>>>>> of
> >> >>> >>>>>>>>> software
> >> >>> >>>>>>>>> is that the developers constantly ask you to rewrite your
> >> >>> >>>>>>>>> app
> >> >>> >>>>>>>>> to
> >> >>> >>>>>>>>> update to a
> >> >>> >>>>>>>>> new version (and thus benefit from bug fixes, etc). Cue
> >> >>> >>>>>>>>> anyone
> >> >>> >>>>>>>>> who's used
> >> >>> >>>>>>>>> Protobuf, or Guava. The "let's get everyone to change
> their
> >> >>> >>>>>>>>> code
> >> >>> >>>>>>>>> this
> >> >>> >>>>>>>>> release" model works well within a single large company,
> but
> >> >>> >>>>>>>>> doesn't work
> >> >>> >>>>>>>>> well for a community, which is why nearly all *very*
> widely
> >> >>> >>>>>>>>> used
> >> >>> >>>>>>>>> programming
> >> >>> >>>>>>>>> interfaces (I'm talking things like Java standard library,
> >> >>> >>>>>>>>> Windows
> >> >>> >>>>>>>>> API, etc)
> >> >>> >>>>>>>>> almost *never* break backwards compatibility. All this is
> >> >>> >>>>>>>>> done
> >> >>> >>>>>>>>> within reason
> >> >>> >>>>>>>>> though, e.g. we do change things in major releases (2.x,
> >> >>> >>>>>>>>> 3.x,
> >> >>> >>>>>>>>> etc).
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>>
> >> >>> >>>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>>
> >> >>> >>>>>> ------------------------------------------------------------
> ---------
> >> >>> >>>>>> To unsubscribe e-mail: [hidden email]
> >> >>> >>>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>> --
> >> >>> >>>>> Stavros Kontopoulos
> >> >>> >>>>> Senior Software Engineer
> >> >>> >>>>> Lightbend, Inc.
> >> >>> >>>>> p:  +30 6977967274
> >> >>> >>>>> e: [hidden email]
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>
> >> >>> >>>
> >> >>> >>
> >> >>> >>
> >> >>>
> >> >>
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe e-mail: [hidden email]
> >> >
> >> >
> >> > ________________________________
> >> >
> >> > If you reply to this email, your message will be added to the
> discussion
> >> > below:
> >> >
> >> >
> >> > http://apache-spark-developers-list.1001551.n3.
> nabble.com/Spark-Improvement-Proposals-tp19268p19359.html
> >> >
> >> > To start a new topic under Apache Spark Developers List, email [hidden
> >> > email]
> >> > To unsubscribe from Apache Spark Developers List, click here.
> >> > NAML
> >> >
> >> >
> >> > ________________________________
> >> > View this message in context: RE: Spark Improvement Proposals
> >> > Sent from the Apache Spark Developers List mailing list archive at
> >> > Nabble.com.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>
> >
> >
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Netflix
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Mime
View raw message