spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: Spark Improvement Proposals
Date Thu, 13 Oct 2016 03:30:03 GMT
Some of you guys may have already seen this but in case if you haven't you
may want to check it out.

http://www.slideshare.net/sbaltagi/flink-vs-spark



On Tue, Oct 11, 2016 at 1:57 PM, Ryan Blue <rblue@netflix.com.invalid>
wrote:

> I don't think we will have trouble with whatever rule that is adopted for
> accepting proposals. Considering committers' votes binding (if that is what
> we choose) is an established practice as long as it isn't for specific
> votes, like a release vote. From the Apache docs: "Who is permitted to vote
> is, to some extent, a community-specific thing." [1] And, I also don't see
> why it would be a problem to choose consensus, as long as we have an open
> discussion and vote about these rules.
>
> rb
>
> On Mon, Oct 10, 2016 at 4:15 PM, Cody Koeninger <cody@koeninger.org>
> wrote:
>
>> If someone wants to tell me that it's OK and "The Apache Way" for
>> Kafka and Flink to have a proposal process that ends in a lazy
>> majority, but it's not OK for Spark to have a proposal process that
>> ends in a non-lazy consensus...
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+
>> Improvement+Proposals#KafkaImprovementProposals-Process
>>
>> In practice any PMC member can stop a proposal they don't like, so I'm
>> not sure how much it matters.
>>
>>
>>
>> On Mon, Oct 10, 2016 at 5:59 PM, Mark Hamstra <mark@clearstorydata.com>
>> wrote:
>> > There is a larger issue to keep in mind, and that is that what you are
>> > proposing is a procedure that, as far as I am aware, hasn't previously
>> been
>> > adopted in an Apache project, and thus is not an easy or exact fit with
>> > established practices that have been blessed as "The Apache Way".  As
>> such,
>> > we need to be careful, because we have run into some trouble in the past
>> > with some inside the ASF but essentially outside the Spark community who
>> > didn't like the way we were doing things.
>> >
>> > On Mon, Oct 10, 2016 at 3:53 PM, Cody Koeninger <cody@koeninger.org>
>> wrote:
>> >>
>> >> Apache documents say lots of confusing stuff, including that commiters
>> are
>> >> in practice given a vote.
>> >>
>> >> https://www.apache.org/foundation/voting.html
>> >>
>> >> I don't care either way, if someone wants me to sub commiter for PMC in
>> >> the voting section, fine, we just need a clear outcome.
>> >>
>> >>
>> >> On Oct 10, 2016 17:36, "Mark Hamstra" <mark@clearstorydata.com> wrote:
>> >>>
>> >>> If I'm correctly understanding the kind of voting that you are talking
>> >>> about, then to be accurate, it is only the PMC members that have a
>> vote, not
>> >>> all committers:
>> >>> https://www.apache.org/foundation/how-it-works.html#pmc-members
>> >>>
>> >>> On Mon, Oct 10, 2016 at 12:02 PM, Cody Koeninger <cody@koeninger.org>
>> >>> wrote:
>> >>>>
>> >>>> I think the main value is in being honest about what's going on.  No
>> >>>> one other than committers can cast a meaningful vote, that's the
>> >>>> reality.  Beyond that, if people think it's more open to allow formal
>> >>>> proposals from anyone, I'm not necessarily against it, but my main
>> >>>> question would be this:
>> >>>>
>> >>>> If anyone can submit a proposal, are committers actually going to
>> >>>> clearly reject and close proposals that don't meet the requirements?
>> >>>>
>> >>>> Right now we have a serious problem with lack of clarity regarding
>> >>>> contributions, and that cannot spill over into goal-setting.
>> >>>>
>> >>>> On Mon, Oct 10, 2016 at 1:54 PM, Ryan Blue <rblue@netflix.com>
>> wrote:
>> >>>> > +1 to votes to approve proposals. I agree that proposals should
>> have
>> >>>> > an
>> >>>> > official mechanism to be accepted, and a vote is an established
>> means
>> >>>> > of
>> >>>> > doing that well. I like that it includes a period to review the
>> >>>> > proposal and
>> >>>> > I think proposals should have been discussed enough ahead of a
>> vote to
>> >>>> > survive the possibility of a veto.
>> >>>> >
>> >>>> > I also like the names that are short and (mostly) unique, like SEP.
>> >>>> >
>> >>>> > Where I disagree is with the requirement that a committer must
>> >>>> > formally
>> >>>> > propose an enhancement. I don't see the value of restricting this:
>> if
>> >>>> > someone has the will to write up a proposal then they should be
>> >>>> > encouraged
>> >>>> > to do so and start a discussion about it. Even if there is a
>> political
>> >>>> > reality as Cody says, what is the value of codifying that in our
>> >>>> > process? I
>> >>>> > think restricting who can submit proposals would only undermine
>> them
>> >>>> > by
>> >>>> > pushing contributors out. Maybe I'm missing something here?
>> >>>> >
>> >>>> > rb
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > On Mon, Oct 10, 2016 at 7:41 AM, Cody Koeninger <
>> cody@koeninger.org>
>> >>>> > wrote:
>> >>>> >>
>> >>>> >> Yes, users suggesting SIPs is a good thing and is explicitly
>> called
>> >>>> >> out in the linked document under the Who? section.  Formally
>> >>>> >> proposing
>> >>>> >> them, not so much, because of the political realities.
>> >>>> >>
>> >>>> >> Yes, implementation strategy definitely affects goals.  There are
>> all
>> >>>> >> kinds of examples of this, I'll pick one that's my fault so as to
>> >>>> >> avoid sounding like I'm blaming:
>> >>>> >>
>> >>>> >> When I implemented the Kafka DStream, one of my (not explicitly
>> >>>> >> agreed
>> >>>> >> upon by the community) goals was to make sure people could use the
>> >>>> >> Dstream with however they were already using Kafka at work.  The
>> lack
>> >>>> >> of explicit agreement on that goal led to all kinds of fighting
>> with
>> >>>> >> committers, that could have been avoided.  The lack of explicit
>> >>>> >> up-front strategy discussion led to the DStream not really working
>> >>>> >> with compacted topics.  I knew about compacted topics, but don't
>> have
>> >>>> >> a use for them, so had a blind spot there.  If there was explicit
>> >>>> >> up-front discussion that my strategy was "assume that batches can
>> be
>> >>>> >> defined on the driver solely by beginning and ending offsets",
>> >>>> >> there's
>> >>>> >> a greater chance that a user would have seen that and said, "hey,
>> >>>> >> what
>> >>>> >> about non-contiguous offsets in a compacted topic".
>> >>>> >>
>> >>>> >> This kind of thing is only going to happen smoothly if we have a
>> >>>> >> lightweight user-visible process with clear outcomes.
>> >>>> >>
>> >>>> >> On Mon, Oct 10, 2016 at 1:34 AM, assaf.mendelson
>> >>>> >> <assaf.mendelson@rsa.com> wrote:
>> >>>> >> > I agree with most of what Cody said.
>> >>>> >> >
>> >>>> >> > Two things:
>> >>>> >> >
>> >>>> >> > First we can always have other people suggest SIPs but mark
>> them as
>> >>>> >> > “unreviewed” and have committers basically move them forward.
>> The
>> >>>> >> > problem is
>> >>>> >> > that writing a good document takes time. This way we can
>> leverage
>> >>>> >> > non
>> >>>> >> > committers to do some of this work (it is just another way to
>> >>>> >> > contribute).
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > As for strategy, in many cases implementation strategy can
>> affect
>> >>>> >> > the
>> >>>> >> > goals.
>> >>>> >> > I will give  a small example: In the current structured
>> streaming
>> >>>> >> > strategy,
>> >>>> >> > we group by the time to achieve a sliding window. This is
>> >>>> >> > definitely an
>> >>>> >> > implementation decision and not a goal. However, I can think of
>> >>>> >> > several
>> >>>> >> > aggregation functions which have the time inside their
>> calculation
>> >>>> >> > buffer.
>> >>>> >> > For example, let’s say we want to return a set of all distinct
>> >>>> >> > values.
>> >>>> >> > One
>> >>>> >> > way to implement this would be to make the set into a map and
>> have
>> >>>> >> > the
>> >>>> >> > value
>> >>>> >> > contain the last time seen. Multiplying it across the groupby
>> would
>> >>>> >> > cost
>> >>>> >> > a
>> >>>> >> > lot in performance. So adding such a strategy would have a great
>> >>>> >> > effect
>> >>>> >> > on
>> >>>> >> > the type of aggregations and their performance which does affect
>> >>>> >> > the
>> >>>> >> > goal.
>> >>>> >> > Without adding the strategy, it is easy for whoever goes to the
>> >>>> >> > design
>> >>>> >> > document to not think about these cases. Furthermore, it might
>> be
>> >>>> >> > decided
>> >>>> >> > that these cases are rare enough so that the strategy is still
>> good
>> >>>> >> > enough
>> >>>> >> > but how would we know it without user feedback?
>> >>>> >> >
>> >>>> >> > I believe this example is exactly what Cody was talking about.
>> >>>> >> > Since
>> >>>> >> > many
>> >>>> >> > times implementation strategies have a large effect on the
>> goal, we
>> >>>> >> > should
>> >>>> >> > have it discussed when discussing the goals. In addition, while
>> it
>> >>>> >> > is
>> >>>> >> > often
>> >>>> >> > easy to throw out completely infeasible goals, it is often much
>> >>>> >> > harder
>> >>>> >> > to
>> >>>> >> > figure out that the goals are unfeasible without fine tuning.
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > Assaf.
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > From: Cody Koeninger-2 [via Apache Spark Developers List]
>> >>>> >> > [mailto:ml-node+[hidden email]]
>> >>>> >> > Sent: Monday, October 10, 2016 2:25 AM
>> >>>> >> > To: Mendelson, Assaf
>> >>>> >> > Subject: Re: Spark Improvement Proposals
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > Only committers should formally submit SIPs because in an apache
>> >>>> >> > project only commiters have explicit political power.  If a user
>> >>>> >> > can't
>> >>>> >> > find a commiter willing to sponsor an SIP idea, they have no
>> way to
>> >>>> >> > get the idea passed in any case.  If I can't find a committer to
>> >>>> >> > sponsor this meta-SIP idea, I'm out of luck.
>> >>>> >> >
>> >>>> >> > I do not believe unrealistic goals can be found solely by
>> >>>> >> > inspection.
>> >>>> >> > We've managed to ignore unrealistic goals even after
>> >>>> >> > implementation!
>> >>>> >> > Focusing on APIs can allow people to think they've solved
>> >>>> >> > something,
>> >>>> >> > when there's really no way of implementing that API while
>> meeting
>> >>>> >> > the
>> >>>> >> > goals.  Rapid iteration is clearly the best way to address this,
>> >>>> >> > but
>> >>>> >> > we've already talked about why that hasn't really worked.  If
>> >>>> >> > adding a
>> >>>> >> > non-binding API section to the template is important to you, I'm
>> >>>> >> > not
>> >>>> >> > against it, but I don't think it's sufficient.
>> >>>> >> >
>> >>>> >> > On your PRD vs design doc spectrum, I'm saying this is closer
>> to a
>> >>>> >> > PRD.  Clear agreement on goals is the most important thing and
>> >>>> >> > that's
>> >>>> >> > why it's the thing I want binding agreement on.  But I cannot
>> agree
>> >>>> >> > to
>> >>>> >> > goals unless I have enough minimal technical info to judge
>> whether
>> >>>> >> > the
>> >>>> >> > goals are likely to actually be accomplished.
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > On Sun, Oct 9, 2016 at 5:35 PM, Matei Zaharia <[hidden email]>
>> >>>> >> > wrote:
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >> Well, I think there are a few things here that don't make
>> sense.
>> >>>> >> >> First,
>> >>>> >> >> why
>> >>>> >> >> should only committers submit SIPs? Development in the project
>> >>>> >> >> should
>> >>>> >> >> be
>> >>>> >> >> open to all contributors, whether they're committers or not.
>> >>>> >> >> Second, I
>> >>>> >> >> think
>> >>>> >> >> unrealistic goals can be found just by inspecting the goals,
>> and
>> >>>> >> >> I'm
>> >>>> >> >> not
>> >>>> >> >> super worried that we'll accept a lot of SIPs that are then
>> >>>> >> >> infeasible
>> >>>> >> >> --
>> >>>> >> >> we
>> >>>> >> >> can then submit new ones. But this depends on whether you want
>> >>>> >> >> this
>> >>>> >> >> process
>> >>>> >> >> to be a "design doc lite", where people also agree on
>> >>>> >> >> implementation
>> >>>> >> >> strategy, or just a way to agree on goals. This is what I asked
>> >>>> >> >> earlier
>> >>>> >> >> about PRDs vs design docs (and I'm open to either one but I'd
>> just
>> >>>> >> >> like
>> >>>> >> >> clarity). Finally, both as a user and designer of software, I
>> >>>> >> >> always
>> >>>> >> >> want
>> >>>> >> >> to
>> >>>> >> >> give feedback on APIs, so I'd really like a culture of having
>> >>>> >> >> those
>> >>>> >> >> early.
>> >>>> >> >> People don't argue about prettiness when they discuss APIs,
>> they
>> >>>> >> >> argue
>> >>>> >> >> about
>> >>>> >> >> the core concepts to expose in order to meet various goals, and
>> >>>> >> >> then
>> >>>> >> >> they're
>> >>>> >> >> stuck maintaining those for a long time.
>> >>>> >> >>
>> >>>> >> >> Matei
>> >>>> >> >>
>> >>>> >> >> On Oct 9, 2016, at 3:10 PM, Cody Koeninger <[hidden email]>
>> wrote:
>> >>>> >> >>
>> >>>> >> >> Users instead of people, sure.  Commiters and contributors are
>> (or
>> >>>> >> >> at
>> >>>> >> >> least
>> >>>> >> >> should be) a subset of users.
>> >>>> >> >>
>> >>>> >> >> Non goals, sure. I don't care what the name is, but we need to
>> >>>> >> >> clearly
>> >>>> >> >> say
>> >>>> >> >> e.g. 'no we are not maintaining compatibility with XYZ right
>> now'.
>> >>>> >> >>
>> >>>> >> >> API, what I care most about is whether it allows me to
>> accomplish
>> >>>> >> >> the
>> >>>> >> >> goals.
>> >>>> >> >> Arguing about how ugly or pretty it is can be saved for design/
>> >>>> >> >> implementation imho.
>> >>>> >> >>
>> >>>> >> >> Strategy, this is necessary because otherwise goals can be out
>> of
>> >>>> >> >> line
>> >>>> >> >> with
>> >>>> >> >> reality.  Don't propose goals you don't have at least some
>> idea of
>> >>>> >> >> how
>> >>>> >> >> to
>> >>>> >> >> implement.
>> >>>> >> >>
>> >>>> >> >> Rejected strategies, given that commiters are the only ones I'm
>> >>>> >> >> saying
>> >>>> >> >> should formally submit SPARKLIs or SIPs, if they put junk in a
>> >>>> >> >> required
>> >>>> >> >> section then slap them down for it and tell them to fix it.
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> On Oct 9, 2016 4:36 PM, "Matei Zaharia" <[hidden email]> wrote:
>> >>>> >> >>>
>> >>>> >> >>> Yup, this is the stuff that I found unclear. Thanks for
>> >>>> >> >>> clarifying
>> >>>> >> >>> here,
>> >>>> >> >>> but we should also clarify it in the writeup. In particular:
>> >>>> >> >>>
>> >>>> >> >>> - Goals needs to be about user-facing behavior ("people" is
>> >>>> >> >>> broad)
>> >>>> >> >>>
>> >>>> >> >>> - I'd rename Rejected Goals to Non-Goals. Otherwise someone
>> will
>> >>>> >> >>> dig
>> >>>> >> >>> up
>> >>>> >> >>> one of these and say "Spark's developers have officially
>> rejected
>> >>>> >> >>> X,
>> >>>> >> >>> which
>> >>>> >> >>> our awesome system has".
>> >>>> >> >>>
>> >>>> >> >>> - For user-facing stuff, I think you need a section on API.
>> >>>> >> >>> Virtually
>> >>>> >> >>> all
>> >>>> >> >>> other *IPs I've seen have that.
>> >>>> >> >>>
>> >>>> >> >>> - I'm still not sure why the strategy section is needed if the
>> >>>> >> >>> purpose
>> >>>> >> >>> is
>> >>>> >> >>> to define user-facing behavior -- unless this is the strategy
>> for
>> >>>> >> >>> setting
>> >>>> >> >>> the goals or for defining the API. That sounds squarely like a
>> >>>> >> >>> design
>> >>>> >> >>> doc
>> >>>> >> >>> issue. In some sense, who cares whether the proposal is
>> >>>> >> >>> technically
>> >>>> >> >>> feasible
>> >>>> >> >>> right now? If it's infeasible, that will be discovered later
>> >>>> >> >>> during
>> >>>> >> >>> design
>> >>>> >> >>> and implementation. Same thing with rejected strategies --
>> >>>> >> >>> listing
>> >>>> >> >>> some
>> >>>> >> >>> of
>> >>>> >> >>> those is definitely useful sometimes, but if you make this a
>> >>>> >> >>> *required*
>> >>>> >> >>> section, people are just going to fill it in with bogus stuff
>> >>>> >> >>> (I've
>> >>>> >> >>> seen
>> >>>> >> >>> this happen before).
>> >>>> >> >>>
>> >>>> >> >>> Matei
>> >>>> >> >>>
>> >>>> >> >
>> >>>> >> >>> > On Oct 9, 2016, at 2:14 PM, Cody Koeninger <[hidden email]>
>> >>>> >> >>> > wrote:
>> >>>> >> >>> >
>> >>>> >> >>> > So to focus the discussion on the specific strategy I'm
>> >>>> >> >>> > suggesting,
>> >>>> >> >>> > documented at
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> > https://github.com/koeninger/s
>> park-1/blob/SIP-0/docs/spark-improvement-proposals.md
>> >>>> >> >>> >
>> >>>> >> >>> > "Goals: What must this allow people to do, that they can't
>> >>>> >> >>> > currently?"
>> >>>> >> >>> >
>> >>>> >> >>> > Is it unclear that this is focusing specifically on
>> >>>> >> >>> > people-visible
>> >>>> >> >>> > behavior?
>> >>>> >> >>> >
>> >>>> >> >>> > Rejected goals -  are important because otherwise people
>> keep
>> >>>> >> >>> > trying
>> >>>> >> >>> > to argue about scope.  Of course you can change things later
>> >>>> >> >>> > with a
>> >>>> >> >>> > different SIP and different vote, the point is to focus.
>> >>>> >> >>> >
>> >>>> >> >>> > Use cases - are something that people are going to bring up
>> in
>> >>>> >> >>> > discussion.  If they aren't clearly documented as a goal
>> ("This
>> >>>> >> >>> > must
>> >>>> >> >>> > allow me to connect using SSL"), they should be added.
>> >>>> >> >>> >
>> >>>> >> >>> > Internal architecture - if the people who need specific
>> >>>> >> >>> > behavior are
>> >>>> >> >>> > implementers of other parts of the system, that's fine.
>> >>>> >> >>> >
>> >>>> >> >>> > Rejected strategies - If you have none of these, you have no
>> >>>> >> >>> > evidence
>> >>>> >> >>> > that the proponent didn't just go with the first thing they
>> had
>> >>>> >> >>> > in
>> >>>> >> >>> > mind (or have already implemented), which is a big problem
>> >>>> >> >>> > currently.
>> >>>> >> >>> > Approval isn't binding as to specifics of implementation, so
>> >>>> >> >>> > these
>> >>>> >> >>> > aren't handcuffs.  The goals are the contract, the strategy
>> is
>> >>>> >> >>> > evidence that contract can actually be met.
>> >>>> >> >>> >
>> >>>> >> >>> > Design docs - I'm not touching design docs.  The markdown
>> file
>> >>>> >> >>> > I
>> >>>> >> >>> > linked specifically says of the strategy section "This is
>> not a
>> >>>> >> >>> > full
>> >>>> >> >>> > design document."  Is this unclear?  Design docs can be
>> worked
>> >>>> >> >>> > on
>> >>>> >> >>> > obviously, but that's not what I'm concerned with here.
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> >
>> >>>> >> >>> > On Sun, Oct 9, 2016 at 2:34 PM, Matei Zaharia <[hidden
>> email]>
>> >>>> >> >>> > wrote:
>> >>>> >> >>> >> Hi Cody,
>> >>>> >> >>> >>
>> >>>> >> >>> >> I think this would be a lot more concrete if we had a more
>> >>>> >> >>> >> detailed
>> >>>> >> >>> >> template
>> >>>> >> >>> >> for SIPs. Right now, it's not super clear what's in scope
>> --
>> >>>> >> >>> >> e.g.
>> >>>> >> >>> >> are
>> >>>> >> >>> >> they
>> >>>> >> >>> >> a way to solicit feedback on the user-facing behavior or on
>> >>>> >> >>> >> the
>> >>>> >> >>> >> internals?
>> >>>> >> >>> >> "Goals" can cover both things. I've been thinking of SIPs
>> more
>> >>>> >> >>> >> as
>> >>>> >> >>> >> Product
>> >>>> >> >>> >> Requirements Docs (PRDs), which focus on *what* a code
>> change
>> >>>> >> >>> >> should
>> >>>> >> >>> >> do
>> >>>> >> >>> >> as
>> >>>> >> >>> >> opposed to how.
>> >>>> >> >>> >>
>> >>>> >> >>> >> In particular, here are some things that you may or may not
>> >>>> >> >>> >> consider
>> >>>> >> >>> >> in
>> >>>> >> >>> >> scope for SIPs:
>> >>>> >> >>> >>
>> >>>> >> >>> >> - Goals and non-goals: This is definitely in scope, and IMO
>> >>>> >> >>> >> should
>> >>>> >> >>> >> focus on
>> >>>> >> >>> >> user-visible behavior (e.g. "system supports SQL window
>> >>>> >> >>> >> functions"
>> >>>> >> >>> >> or
>> >>>> >> >>> >> "system continues working if one node fails"). BTW I
>> wouldn't
>> >>>> >> >>> >> say
>> >>>> >> >>> >> "rejected
>> >>>> >> >>> >> goals" because some of them might become goals later, so
>> we're
>> >>>> >> >>> >> not
>> >>>> >> >>> >> definitively rejecting them.
>> >>>> >> >>> >>
>> >>>> >> >>> >> - Public API: Probably should be included in most SIPs
>> unless
>> >>>> >> >>> >> it's
>> >>>> >> >>> >> too
>> >>>> >> >>> >> large
>> >>>> >> >>> >> to fully specify then (e.g. "let's add an ML library").
>> >>>> >> >>> >>
>> >>>> >> >>> >> - Use cases: I usually find this very useful in PRDs to
>> better
>> >>>> >> >>> >> communicate
>> >>>> >> >>> >> the goals.
>> >>>> >> >>> >>
>> >>>> >> >>> >> - Internal architecture: This is usually *not* a thing
>> users
>> >>>> >> >>> >> can
>> >>>> >> >>> >> easily
>> >>>> >> >>> >> comment on and it sounds more like a design doc item. Of
>> >>>> >> >>> >> course
>> >>>> >> >>> >> it's
>> >>>> >> >>> >> important to show that the SIP is feasible to implement.
>> One
>> >>>> >> >>> >> exception,
>> >>>> >> >>> >> however, is that I think we'll have some SIPs primarily on
>> >>>> >> >>> >> internals
>> >>>> >> >>> >> (e.g.
>> >>>> >> >>> >> if somebody wants to refactor Spark's query optimizer or
>> >>>> >> >>> >> something).
>> >>>> >> >>> >>
>> >>>> >> >>> >> - Rejected strategies: I personally wouldn't put this,
>> because
>> >>>> >> >>> >> what's
>> >>>> >> >>> >> the
>> >>>> >> >>> >> point of voting to reject a strategy before you've really
>> >>>> >> >>> >> begun
>> >>>> >> >>> >> designing
>> >>>> >> >>> >> and implementing something? What if you discover that the
>> >>>> >> >>> >> strategy
>> >>>> >> >>> >> is
>> >>>> >> >>> >> actually better when you start doing stuff?
>> >>>> >> >>> >>
>> >>>> >> >>> >> At a super high level, it depends on whether you want the
>> SIPs
>> >>>> >> >>> >> to
>> >>>> >> >>> >> be
>> >>>> >> >>> >> PRDs
>> >>>> >> >>> >> for getting some quick feedback on the goals of a feature
>> >>>> >> >>> >> before it
>> >>>> >> >>> >> is
>> >>>> >> >>> >> designed, or something more like full-fledged design docs
>> >>>> >> >>> >> (just a
>> >>>> >> >>> >> more
>> >>>> >> >>> >> visible design doc for bigger changes). I looked at Kafka's
>> >>>> >> >>> >> KIPs,
>> >>>> >> >>> >> and
>> >>>> >> >>> >> they
>> >>>> >> >>> >> actually seem to be more like design docs. This can work
>> too
>> >>>> >> >>> >> but it
>> >>>> >> >>> >> does
>> >>>> >> >>> >> require more work from the proposer and it can lead to the
>> >>>> >> >>> >> same
>> >>>> >> >>> >> problems you
>> >>>> >> >>> >> mentioned with people already having a design and
>> >>>> >> >>> >> implementation in
>> >>>> >> >>> >> mind.
>> >>>> >> >>> >>
>> >>>> >> >>> >> Basically, the question is, are you trying to iterate
>> faster
>> >>>> >> >>> >> on
>> >>>> >> >>> >> design
>> >>>> >> >>> >> by
>> >>>> >> >>> >> adding a step for user feedback earlier? Or are you just
>> >>>> >> >>> >> trying to
>> >>>> >> >>> >> make
>> >>>> >> >>> >> design docs for key features more visible (and their
>> approval
>> >>>> >> >>> >> more
>> >>>> >> >>> >> formal)?
>> >>>> >> >>> >>
>> >>>> >> >>> >> BTW note that in either case, I'd like to have a template
>> for
>> >>>> >> >>> >> design
>> >>>> >> >>> >> docs
>> >>>> >> >>> >> too, which should also include goals. I think that would've
>> >>>> >> >>> >> avoided
>> >>>> >> >>> >> some of
>> >>>> >> >>> >> the issues you brought up.
>> >>>> >> >>> >>
>> >>>> >> >>> >> Matei
>> >>>> >> >>> >>
>> >>>> >> >>> >> On Oct 9, 2016, at 10:40 AM, Cody Koeninger <[hidden
>> email]>
>> >>>> >> >>> >> wrote:
>> >>>> >> >>> >>
>> >>>> >> >>> >> Here's my specific proposal (meta-proposal?)
>> >>>> >> >>> >>
>> >>>> >> >>> >> Spark Improvement Proposals (SIP)
>> >>>> >> >>> >>
>> >>>> >> >>> >>
>> >>>> >> >>> >> Background:
>> >>>> >> >>> >>
>> >>>> >> >>> >> The current problem is that design and implementation of
>> large
>> >>>> >> >>> >> features
>> >>>> >> >>> >> are
>> >>>> >> >>> >> often done in private, before soliciting user feedback.
>> >>>> >> >>> >>
>> >>>> >> >>> >> When feedback is solicited, it is often as to detailed
>> design
>> >>>> >> >>> >> specifics, not
>> >>>> >> >>> >> focused on goals.
>> >>>> >> >>> >>
>> >>>> >> >>> >> When implementation does take place after design, there is
>> >>>> >> >>> >> often
>> >>>> >> >>> >> disagreement as to what goals are or are not in scope.
>> >>>> >> >>> >>
>> >>>> >> >>> >> This results in commits that don't fully meet user needs.
>> >>>> >> >>> >>
>> >>>> >> >>> >>
>> >>>> >> >>> >> Goals:
>> >>>> >> >>> >>
>> >>>> >> >>> >> - Ensure user, contributor, and committer goals are clearly
>> >>>> >> >>> >> identified
>> >>>> >> >>> >> and
>> >>>> >> >>> >> agreed upon, before implementation takes place.
>> >>>> >> >>> >>
>> >>>> >> >>> >> - Ensure that a technically feasible strategy is chosen
>> that
>> >>>> >> >>> >> is
>> >>>> >> >>> >> likely
>> >>>> >> >>> >> to
>> >>>> >> >>> >> meet the goals.
>> >>>> >> >>> >>
>> >>>> >> >>> >>
>> >>>> >> >>> >> Rejected Goals:
>> >>>> >> >>> >>
>> >>>> >> >>> >> - SIPs are not for detailed design.  Design by committee
>> >>>> >> >>> >> doesn't
>> >>>> >> >>> >> work.
>> >>>> >> >>> >>
>> >>>> >> >>> >> - SIPs are not for every change.  We dont need that much
>> >>>> >> >>> >> process.
>> >>>> >> >>> >>
>> >>>> >> >>> >>
>> >>>> >> >>> >> Strategy:
>> >>>> >> >>> >>
>> >>>> >> >>> >> My suggestion is outlined as a Spark Improvement Proposal
>> >>>> >> >>> >> process
>> >>>> >> >>> >> documented
>> >>>> >> >>> >> at
>> >>>> >> >>> >>
>> >>>> >> >>> >>
>> >>>> >> >>> >>
>> >>>> >> >>> >>
>> >>>> >> >>> >>
>> >>>> >> >>> >> https://github.com/koeninger/s
>> park-1/blob/SIP-0/docs/spark-improvement-proposals.md
>> >>>> >> >>> >>
>> >>>> >> >>> >> Specifics of Jira manipulation are an implementation
>> detail we
>> >>>> >> >>> >> can
>> >>>> >> >>> >> figure
>> >>>> >> >>> >> out.
>> >>>> >> >>> >>
>> >>>> >> >>> >> I'm suggesting voting; the need here is for a _clear_
>> outcome.
>> >>>> >> >>> >>
>> >>>> >> >>> >>
>> >>>> >> >>> >> Rejected Strategies:
>> >>>> >> >>> >>
>> >>>> >> >>> >> Having someone who understands the problem implement it
>> first
>> >>>> >> >>> >> works,
>> >>>> >> >>> >> but
>> >>>> >> >>> >> only if significant iteration after user feedback is
>> allowed.
>> >>>> >> >>> >>
>> >>>> >> >>> >> Historically this has been problematic due to pressure to
>> >>>> >> >>> >> limit
>> >>>> >> >>> >> public
>> >>>> >> >>> >> api
>> >>>> >> >>> >> changes.
>> >>>> >> >>> >>
>> >>>> >> >>> >>
>> >>>> >> >>> >> On Fri, Oct 7, 2016 at 5:16 PM, Reynold Xin <[hidden
>> email]>
>> >>>> >> >>> >> wrote:
>> >>>> >> >>> >>>
>> >>>> >> >>> >>> Alright looks like there are quite a bit of support. We
>> >>>> >> >>> >>> should
>> >>>> >> >>> >>> wait
>> >>>> >> >>> >>> to
>> >>>> >> >>> >>> hear from more people too.
>> >>>> >> >>> >>>
>> >>>> >> >>> >>> To push this forward, Cody and I will be working together
>> in
>> >>>> >> >>> >>> the
>> >>>> >> >>> >>> next
>> >>>> >> >>> >>> couple of weeks to come up with a concrete, detailed
>> proposal
>> >>>> >> >>> >>> on
>> >>>> >> >>> >>> what
>> >>>> >> >>> >>> this
>> >>>> >> >>> >>> entails, and then we can discuss this the specific
>> proposal
>> >>>> >> >>> >>> as
>> >>>> >> >>> >>> well.
>> >>>> >> >>> >>>
>> >>>> >> >>> >>>
>> >>>> >> >>> >>> On Fri, Oct 7, 2016 at 2:29 PM, Cody Koeninger <[hidden
>> >>>> >> >>> >>> email]>
>> >>>> >> >>> >>> wrote:
>> >>>> >> >>> >>>>
>> >>>> >> >>> >>>> Yeah, in case it wasn't clear, I was talking about SIPs
>> for
>> >>>> >> >>> >>>> major
>> >>>> >> >>> >>>> user-facing or cross-cutting changes, not minor feature
>> >>>> >> >>> >>>> adds.
>> >>>> >> >>> >>>>
>> >>>> >> >>> >>>> On Fri, Oct 7, 2016 at 3:58 PM, Stavros Kontopoulos
>> >>>> >> >>> >>>> <[hidden email]> wrote:
>> >>>> >> >>> >>>>>
>> >>>> >> >>> >>>>> +1 to the SIP label as long as it does not slow down
>> things
>> >>>> >> >>> >>>>> and
>> >>>> >> >>> >>>>> it
>> >>>> >> >>> >>>>> targets optimizing efforts, coordination etc. For
>> example
>> >>>> >> >>> >>>>> really
>> >>>> >> >>> >>>>> small
>> >>>> >> >>> >>>>> features should not need to go through this process
>> >>>> >> >>> >>>>> (assuming
>> >>>> >> >>> >>>>> they
>> >>>> >> >>> >>>>> dont
>> >>>> >> >>> >>>>> touch public interfaces)  or re-factorings and hope it
>> will
>> >>>> >> >>> >>>>> be
>> >>>> >> >>> >>>>> kept
>> >>>> >> >>> >>>>> this
>> >>>> >> >>> >>>>> way. So as a guideline doc should be provided, like in
>> the
>> >>>> >> >>> >>>>> KIP
>> >>>> >> >>> >>>>> case.
>> >>>> >> >>> >>>>>
>> >>>> >> >>> >>>>> IMHO so far aside from tagging things and linking them
>> >>>> >> >>> >>>>> elsewhere
>> >>>> >> >>> >>>>> simply
>> >>>> >> >>> >>>>> having design docs and prototypes implementations in
>> PRs is
>> >>>> >> >>> >>>>> not
>> >>>> >> >>> >>>>> something
>> >>>> >> >>> >>>>> that has not worked so far. What is really a pain in
>> many
>> >>>> >> >>> >>>>> projects
>> >>>> >> >>> >>>>> out there
>> >>>> >> >>> >>>>> is discontinuity in progress of PRs, missing features,
>> slow
>> >>>> >> >>> >>>>> reviews
>> >>>> >> >>> >>>>> which is
>> >>>> >> >>> >>>>> understandable to some extent... it is not only about
>> Spark
>> >>>> >> >>> >>>>> but
>> >>>> >> >>> >>>>> things can
>> >>>> >> >>> >>>>> be improved for sure for this project in particular as
>> >>>> >> >>> >>>>> already
>> >>>> >> >>> >>>>> stated.
>> >>>> >> >>> >>>>>
>> >>>> >> >>> >>>>> On Fri, Oct 7, 2016 at 11:14 PM, Cody Koeninger <[hidden
>> >>>> >> >>> >>>>> email]>
>> >>>> >> >>> >>>>> wrote:
>> >>>> >> >>> >>>>>>
>> >>>> >> >>> >>>>>> +1 to adding an SIP label and linking it from the
>> website.
>> >>>> >> >>> >>>>>> I
>> >>>> >> >>> >>>>>> think
>> >>>> >> >>> >>>>>> it
>> >>>> >> >>> >>>>>> needs
>> >>>> >> >>> >>>>>>
>> >>>> >> >>> >>>>>> - template that focuses it towards soliciting user
>> goals /
>> >>>> >> >>> >>>>>> non
>> >>>> >> >>> >>>>>> goals
>> >>>> >> >>> >>>>>> - clear resolution as to which strategy was chosen to
>> >>>> >> >>> >>>>>> pursue.
>> >>>> >> >>> >>>>>> I'd
>> >>>> >> >>> >>>>>> recommend a vote.
>> >>>> >> >>> >>>>>>
>> >>>> >> >>> >>>>>> Matei asked me to clarify what I meant by changing
>> >>>> >> >>> >>>>>> interfaces,
>> >>>> >> >>> >>>>>> I
>> >>>> >> >>> >>>>>> think
>> >>>> >> >>> >>>>>> it's directly relevant to the SIP idea so I'll clarify
>> >>>> >> >>> >>>>>> here,
>> >>>> >> >>> >>>>>> and
>> >>>> >> >>> >>>>>> split
>> >>>> >> >>> >>>>>> a thread for the other discussion per Nicholas'
>> request.
>> >>>> >> >>> >>>>>>
>> >>>> >> >>> >>>>>> I meant changing public user interfaces.  I think the
>> >>>> >> >>> >>>>>> first
>> >>>> >> >>> >>>>>> design
>> >>>> >> >>> >>>>>> is
>> >>>> >> >>> >>>>>> unlikely to be right, because it's done at a time when
>> you
>> >>>> >> >>> >>>>>> have
>> >>>> >> >>> >>>>>> the
>> >>>> >> >>> >>>>>> least information.  As a user, I find it considerably
>> more
>> >>>> >> >>> >>>>>> frustrating
>> >>>> >> >>> >>>>>> to be unable to use a tool to get my job done, than I
>> do
>> >>>> >> >>> >>>>>> having
>> >>>> >> >>> >>>>>> to
>> >>>> >> >>> >>>>>> make minor changes to my code in order to take
>> advantage
>> >>>> >> >>> >>>>>> of
>> >>>> >> >>> >>>>>> features.
>> >>>> >> >>> >>>>>> I've seen committers be seriously reluctant to allow
>> >>>> >> >>> >>>>>> changes to
>> >>>> >> >>> >>>>>> @experimental code that are needed in order for it to
>> >>>> >> >>> >>>>>> really
>> >>>> >> >>> >>>>>> work
>> >>>> >> >>> >>>>>> right.  You need to be able to iterate, and if people
>> on
>> >>>> >> >>> >>>>>> both
>> >>>> >> >>> >>>>>> sides
>> >>>> >> >>> >>>>>> of
>> >>>> >> >>> >>>>>> the fence aren't going to respect that some newer apis
>> are
>> >>>> >> >>> >>>>>> subject
>> >>>> >> >>> >>>>>> to
>> >>>> >> >>> >>>>>> change, then why even mark them as such?
>> >>>> >> >>> >>>>>>
>> >>>> >> >>> >>>>>> Ideally a finished SIP should give me a checklist of
>> >>>> >> >>> >>>>>> things
>> >>>> >> >>> >>>>>> that
>> >>>> >> >>> >>>>>> an
>> >>>> >> >>> >>>>>> implementation must do, and things that it doesn't
>> need to
>> >>>> >> >>> >>>>>> do.
>> >>>> >> >>> >>>>>> Contributors/committers should be seriously discouraged
>> >>>> >> >>> >>>>>> from
>> >>>> >> >>> >>>>>> putting
>> >>>> >> >>> >>>>>> out a version 0.1 that doesn't have at least a
>> prototype
>> >>>> >> >>> >>>>>> implementation of all those things, especially if
>> they're
>> >>>> >> >>> >>>>>> then
>> >>>> >> >>> >>>>>> going
>> >>>> >> >>> >>>>>> to argue against interface changes necessary to get the
>> >>>> >> >>> >>>>>> the
>> >>>> >> >>> >>>>>> rest
>> >>>> >> >>> >>>>>> of
>> >>>> >> >>> >>>>>> the things done in the 0.2 version.
>> >>>> >> >>> >>>>>>
>> >>>> >> >>> >>>>>>
>> >>>> >> >>> >>>>>> On Fri, Oct 7, 2016 at 2:18 PM, Reynold Xin <[hidden
>> >>>> >> >>> >>>>>> email]>
>> >>>> >> >>> >>>>>> wrote:
>> >>>> >> >>> >>>>>>> I like the lightweight proposal to add a SIP label.
>> >>>> >> >>> >>>>>>>
>> >>>> >> >>> >>>>>>> During Spark 2.0 development, Tom (Graves) and I
>> >>>> >> >>> >>>>>>> suggested
>> >>>> >> >>> >>>>>>> using
>> >>>> >> >>> >>>>>>> wiki
>> >>>> >> >>> >>>>>>> to
>> >>>> >> >>> >>>>>>> track the list of major changes, but that never really
>> >>>> >> >>> >>>>>>> materialized
>> >>>> >> >>> >>>>>>> due to
>> >>>> >> >>> >>>>>>> the overhead. Adding a SIP label on major JIRAs and
>> then
>> >>>> >> >>> >>>>>>> link
>> >>>> >> >>> >>>>>>> to
>> >>>> >> >>> >>>>>>> them
>> >>>> >> >>> >>>>>>> prominently on the Spark website makes a lot of sense.
>> >>>> >> >>> >>>>>>>
>> >>>> >> >>> >>>>>>>
>> >>>> >> >>> >>>>>>> On Fri, Oct 7, 2016 at 10:50 AM, Matei Zaharia
>> >>>> >> >>> >>>>>>> <[hidden email]>
>> >>>> >> >>> >>>>>>> wrote:
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>> For the improvement proposals, I think one major
>> point
>> >>>> >> >>> >>>>>>>> was to
>> >>>> >> >>> >>>>>>>> make
>> >>>> >> >>> >>>>>>>> them
>> >>>> >> >>> >>>>>>>> really visible to users who are not contributors, so
>> we
>> >>>> >> >>> >>>>>>>> should
>> >>>> >> >>> >>>>>>>> do
>> >>>> >> >>> >>>>>>>> more than
>> >>>> >> >>> >>>>>>>> sending stuff to dev@. One very lightweight idea is
>> to
>> >>>> >> >>> >>>>>>>> have a
>> >>>> >> >>> >>>>>>>> new
>> >>>> >> >>> >>>>>>>> type of
>> >>>> >> >>> >>>>>>>> JIRA called a SIP and have a link to a filter that
>> shows
>> >>>> >> >>> >>>>>>>> all
>> >>>> >> >>> >>>>>>>> such
>> >>>> >> >>> >>>>>>>> JIRAs from
>> >>>> >> >>> >>>>>>>> http://spark.apache.org. I also like the idea of
>> SIP and
>> >>>> >> >>> >>>>>>>> design
>> >>>> >> >>> >>>>>>>> doc
>> >>>> >> >>> >>>>>>>> templates (in fact many projects have them).
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>> Matei
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>> On Oct 7, 2016, at 10:38 AM, Reynold Xin <[hidden
>> >>>> >> >>> >>>>>>>> email]>
>> >>>> >> >>> >>>>>>>> wrote:
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>> I called Cody last night and talked about some of the
>> >>>> >> >>> >>>>>>>> topics
>> >>>> >> >>> >>>>>>>> in
>> >>>> >> >>> >>>>>>>> his
>> >>>> >> >>> >>>>>>>> email.
>> >>>> >> >>> >>>>>>>> It became clear to me Cody genuinely cares about the
>> >>>> >> >>> >>>>>>>> project.
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>> Some of the frustrations come from the success of the
>> >>>> >> >>> >>>>>>>> project
>> >>>> >> >>> >>>>>>>> itself
>> >>>> >> >>> >>>>>>>> becoming very "hot", and it is difficult to get
>> clarity
>> >>>> >> >>> >>>>>>>> from
>> >>>> >> >>> >>>>>>>> people
>> >>>> >> >>> >>>>>>>> who
>> >>>> >> >>> >>>>>>>> don't dedicate all their time to Spark. In fact, it
>> is
>> >>>> >> >>> >>>>>>>> in
>> >>>> >> >>> >>>>>>>> some
>> >>>> >> >>> >>>>>>>> ways
>> >>>> >> >>> >>>>>>>> similar
>> >>>> >> >>> >>>>>>>> to scaling an engineering team in a successful
>> startup:
>> >>>> >> >>> >>>>>>>> old
>> >>>> >> >>> >>>>>>>> processes that
>> >>>> >> >>> >>>>>>>> worked well might not work so well when it gets to a
>> >>>> >> >>> >>>>>>>> certain
>> >>>> >> >>> >>>>>>>> size,
>> >>>> >> >>> >>>>>>>> cultures
>> >>>> >> >>> >>>>>>>> can get diluted, building culture vs building
>> process,
>> >>>> >> >>> >>>>>>>> etc.
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>> I also really like to have a more visible process for
>> >>>> >> >>> >>>>>>>> larger
>> >>>> >> >>> >>>>>>>> changes,
>> >>>> >> >>> >>>>>>>> especially major user facing API changes.
>> Historically
>> >>>> >> >>> >>>>>>>> we
>> >>>> >> >>> >>>>>>>> upload
>> >>>> >> >>> >>>>>>>> design docs
>> >>>> >> >>> >>>>>>>> for major changes, but it is not always consistent
>> and
>> >>>> >> >>> >>>>>>>> difficult
>> >>>> >> >>> >>>>>>>> to
>> >>>> >> >>> >>>>>>>> quality
>> >>>> >> >>> >>>>>>>> of the docs, due to the volunteering nature of the
>> >>>> >> >>> >>>>>>>> organization.
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>> Some of the more concrete ideas we discussed focus on
>> >>>> >> >>> >>>>>>>> building a
>> >>>> >> >>> >>>>>>>> culture
>> >>>> >> >>> >>>>>>>> to improve clarity:
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>> - Process: Large changes should have design docs
>> posted
>> >>>> >> >>> >>>>>>>> on
>> >>>> >> >>> >>>>>>>> JIRA.
>> >>>> >> >>> >>>>>>>> One
>> >>>> >> >>> >>>>>>>> thing
>> >>>> >> >>> >>>>>>>> Cody and I didn't discuss but an idea that just came
>> to
>> >>>> >> >>> >>>>>>>> me is
>> >>>> >> >>> >>>>>>>> we
>> >>>> >> >>> >>>>>>>> should
>> >>>> >> >>> >>>>>>>> create a design doc template for the project and ask
>> >>>> >> >>> >>>>>>>> everybody
>> >>>> >> >>> >>>>>>>> to
>> >>>> >> >>> >>>>>>>> follow.
>> >>>> >> >>> >>>>>>>> The design doc template should also explicitly list
>> >>>> >> >>> >>>>>>>> goals and
>> >>>> >> >>> >>>>>>>> non-goals, to
>> >>>> >> >>> >>>>>>>> make design doc more consistent.
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>> - Process: Email dev@ to solicit feedback. We have
>> some
>> >>>> >> >>> >>>>>>>> this
>> >>>> >> >>> >>>>>>>> with
>> >>>> >> >>> >>>>>>>> some
>> >>>> >> >>> >>>>>>>> changes, but again very inconsistent. Just posting
>> >>>> >> >>> >>>>>>>> something
>> >>>> >> >>> >>>>>>>> on
>> >>>> >> >>> >>>>>>>> JIRA
>> >>>> >> >>> >>>>>>>> isn't
>> >>>> >> >>> >>>>>>>> sufficient, because there are simply too many JIRAs
>> and
>> >>>> >> >>> >>>>>>>> the
>> >>>> >> >>> >>>>>>>> signal
>> >>>> >> >>> >>>>>>>> get lost
>> >>>> >> >>> >>>>>>>> in the noise. While this is generally impossible to
>> >>>> >> >>> >>>>>>>> enforce
>> >>>> >> >>> >>>>>>>> because
>> >>>> >> >>> >>>>>>>> we can't
>> >>>> >> >>> >>>>>>>> force all volunteers to conform to a process (or they
>> >>>> >> >>> >>>>>>>> might
>> >>>> >> >>> >>>>>>>> not
>> >>>> >> >>> >>>>>>>> even
>> >>>> >> >>> >>>>>>>> be
>> >>>> >> >>> >>>>>>>> aware of this),  those who are more familiar with the
>> >>>> >> >>> >>>>>>>> project
>> >>>> >> >>> >>>>>>>> can
>> >>>> >> >>> >>>>>>>> help by
>> >>>> >> >>> >>>>>>>> emailing the dev@ when they see something that
>> hasn't
>> >>>> >> >>> >>>>>>>> been.
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>> - Culture: The design doc author(s) should be open to
>> >>>> >> >>> >>>>>>>> feedback.
>> >>>> >> >>> >>>>>>>> A
>> >>>> >> >>> >>>>>>>> design
>> >>>> >> >>> >>>>>>>> doc should serve as the base for discussion and is
>> by no
>> >>>> >> >>> >>>>>>>> means
>> >>>> >> >>> >>>>>>>> the
>> >>>> >> >>> >>>>>>>> final
>> >>>> >> >>> >>>>>>>> design. Of course, this does not mean the author has
>> to
>> >>>> >> >>> >>>>>>>> accept
>> >>>> >> >>> >>>>>>>> every
>> >>>> >> >>> >>>>>>>> feedback. They should also be comfortable accepting /
>> >>>> >> >>> >>>>>>>> rejecting
>> >>>> >> >>> >>>>>>>> ideas on
>> >>>> >> >>> >>>>>>>> technical grounds.
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>> - Process / Culture: For major ongoing projects, it
>> can
>> >>>> >> >>> >>>>>>>> be
>> >>>> >> >>> >>>>>>>> useful
>> >>>> >> >>> >>>>>>>> to
>> >>>> >> >>> >>>>>>>> have
>> >>>> >> >>> >>>>>>>> some monthly Google hangouts that are open to the
>> world.
>> >>>> >> >>> >>>>>>>> I am
>> >>>> >> >>> >>>>>>>> actually not
>> >>>> >> >>> >>>>>>>> sure how well this will work, because of the
>> >>>> >> >>> >>>>>>>> volunteering
>> >>>> >> >>> >>>>>>>> nature
>> >>>> >> >>> >>>>>>>> and
>> >>>> >> >>> >>>>>>>> we need
>> >>>> >> >>> >>>>>>>> to adjust for timezones for people across the globe,
>> but
>> >>>> >> >>> >>>>>>>> it
>> >>>> >> >>> >>>>>>>> seems
>> >>>> >> >>> >>>>>>>> worth
>> >>>> >> >>> >>>>>>>> trying.
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>> - Culture: Contributors (including committers)
>> should be
>> >>>> >> >>> >>>>>>>> more
>> >>>> >> >>> >>>>>>>> direct
>> >>>> >> >>> >>>>>>>> in
>> >>>> >> >>> >>>>>>>> setting expectations, including whether they are
>> working
>> >>>> >> >>> >>>>>>>> on a
>> >>>> >> >>> >>>>>>>> specific
>> >>>> >> >>> >>>>>>>> issue, whether they will be working on a specific
>> issue,
>> >>>> >> >>> >>>>>>>> and
>> >>>> >> >>> >>>>>>>> whether
>> >>>> >> >>> >>>>>>>> an
>> >>>> >> >>> >>>>>>>> issue or pr or jira should be rejected. Most people I
>> >>>> >> >>> >>>>>>>> know in
>> >>>> >> >>> >>>>>>>> this
>> >>>> >> >>> >>>>>>>> community
>> >>>> >> >>> >>>>>>>> are nice and don't enjoy telling other people no,
>> but it
>> >>>> >> >>> >>>>>>>> is
>> >>>> >> >>> >>>>>>>> often
>> >>>> >> >>> >>>>>>>> more
>> >>>> >> >>> >>>>>>>> annoying to a contributor to not know anything than
>> >>>> >> >>> >>>>>>>> getting a
>> >>>> >> >>> >>>>>>>> no.
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>> On Fri, Oct 7, 2016 at 10:03 AM, Matei Zaharia
>> >>>> >> >>> >>>>>>>> <[hidden email]>
>> >>>> >> >>> >>>>>>>> wrote:
>> >>>> >> >>> >>>>>>>>>
>> >>>> >> >>> >>>>>>>>>
>> >>>> >> >>> >>>>>>>>> Love the idea of a more visible "Spark Improvement
>> >>>> >> >>> >>>>>>>>> Proposal"
>> >>>> >> >>> >>>>>>>>> process that
>> >>>> >> >>> >>>>>>>>> solicits user input on new APIs. For what it's
>> worth, I
>> >>>> >> >>> >>>>>>>>> don't
>> >>>> >> >>> >>>>>>>>> think
>> >>>> >> >>> >>>>>>>>> committers are trying to minimize their own work --
>> >>>> >> >>> >>>>>>>>> every
>> >>>> >> >>> >>>>>>>>> committer
>> >>>> >> >>> >>>>>>>>> cares
>> >>>> >> >>> >>>>>>>>> about making the software useful for users.
>> However, it
>> >>>> >> >>> >>>>>>>>> is
>> >>>> >> >>> >>>>>>>>> always
>> >>>> >> >>> >>>>>>>>> hard to
>> >>>> >> >>> >>>>>>>>> get user input and so it helps to have this kind of
>> >>>> >> >>> >>>>>>>>> process.
>> >>>> >> >>> >>>>>>>>> I've
>> >>>> >> >>> >>>>>>>>> certainly
>> >>>> >> >>> >>>>>>>>> looked at the *IPs a lot in other software I use
>> just
>> >>>> >> >>> >>>>>>>>> to see
>> >>>> >> >>> >>>>>>>>> the
>> >>>> >> >>> >>>>>>>>> biggest
>> >>>> >> >>> >>>>>>>>> things on the roadmap.
>> >>>> >> >>> >>>>>>>>>
>> >>>> >> >>> >>>>>>>>> When you're talking about "changing interfaces", are
>> >>>> >> >>> >>>>>>>>> you
>> >>>> >> >>> >>>>>>>>> talking
>> >>>> >> >>> >>>>>>>>> about
>> >>>> >> >>> >>>>>>>>> public or internal APIs? I do think many people hate
>> >>>> >> >>> >>>>>>>>> changing
>> >>>> >> >>> >>>>>>>>> public APIs
>> >>>> >> >>> >>>>>>>>> and I actually think that's for the best of the
>> >>>> >> >>> >>>>>>>>> project.
>> >>>> >> >>> >>>>>>>>> That's
>> >>>> >> >>> >>>>>>>>> a
>> >>>> >> >>> >>>>>>>>> technical
>> >>>> >> >>> >>>>>>>>> debate, but basically, the worst thing when you're
>> >>>> >> >>> >>>>>>>>> using a
>> >>>> >> >>> >>>>>>>>> piece
>> >>>> >> >>> >>>>>>>>> of
>> >>>> >> >>> >>>>>>>>> software
>> >>>> >> >>> >>>>>>>>> is that the developers constantly ask you to rewrite
>> >>>> >> >>> >>>>>>>>> your
>> >>>> >> >>> >>>>>>>>> app
>> >>>> >> >>> >>>>>>>>> to
>> >>>> >> >>> >>>>>>>>> update to a
>> >>>> >> >>> >>>>>>>>> new version (and thus benefit from bug fixes, etc).
>> Cue
>> >>>> >> >>> >>>>>>>>> anyone
>> >>>> >> >>> >>>>>>>>> who's used
>> >>>> >> >>> >>>>>>>>> Protobuf, or Guava. The "let's get everyone to
>> change
>> >>>> >> >>> >>>>>>>>> their
>> >>>> >> >>> >>>>>>>>> code
>> >>>> >> >>> >>>>>>>>> this
>> >>>> >> >>> >>>>>>>>> release" model works well within a single large
>> >>>> >> >>> >>>>>>>>> company, but
>> >>>> >> >>> >>>>>>>>> doesn't work
>> >>>> >> >>> >>>>>>>>> well for a community, which is why nearly all *very*
>> >>>> >> >>> >>>>>>>>> widely
>> >>>> >> >>> >>>>>>>>> used
>> >>>> >> >>> >>>>>>>>> programming
>> >>>> >> >>> >>>>>>>>> interfaces (I'm talking things like Java standard
>> >>>> >> >>> >>>>>>>>> library,
>> >>>> >> >>> >>>>>>>>> Windows
>> >>>> >> >>> >>>>>>>>> API, etc)
>> >>>> >> >>> >>>>>>>>> almost *never* break backwards compatibility. All
>> this
>> >>>> >> >>> >>>>>>>>> is
>> >>>> >> >>> >>>>>>>>> done
>> >>>> >> >>> >>>>>>>>> within reason
>> >>>> >> >>> >>>>>>>>> though, e.g. we do change things in major releases
>> >>>> >> >>> >>>>>>>>> (2.x,
>> >>>> >> >>> >>>>>>>>> 3.x,
>> >>>> >> >>> >>>>>>>>> etc).
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>>
>> >>>> >> >>> >>>>>>>
>> >>>> >> >>> >>>>>>
>> >>>> >> >>> >>>>>>
>> >>>> >> >>> >>>>>>
>> >>>> >> >>> >>>>>>
>> >>>> >> >>> >>>>>>
>> >>>> >> >>> >>>>>> ------------------------------
>> ---------------------------------------
>> >>>> >> >>> >>>>>> To unsubscribe e-mail: [hidden email]
>> >>>> >> >>> >>>>>>
>> >>>> >> >>> >>>>>
>> >>>> >> >>> >>>>>
>> >>>> >> >>> >>>>>
>> >>>> >> >>> >>>>> --
>> >>>> >> >>> >>>>> Stavros Kontopoulos
>> >>>> >> >>> >>>>> Senior Software Engineer
>> >>>> >> >>> >>>>> Lightbend, Inc.
>> >>>> >> >>> >>>>> p:  +30 6977967274
>> >>>> >> >>> >>>>> e: [hidden email]
>> >>>> >> >>> >>>>>
>> >>>> >> >>> >>>>>
>> >>>> >> >>> >>>>
>> >>>> >> >>> >>>
>> >>>> >> >>> >>
>> >>>> >> >>> >>
>> >>>> >> >>>
>> >>>> >> >>
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > ------------------------------------------------------------
>> ---------
>> >>>> >> > To unsubscribe e-mail: [hidden email]
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > ________________________________
>> >>>> >> >
>> >>>> >> > If you reply to this email, your message will be added to the
>> >>>> >> > discussion
>> >>>> >> > below:
>> >>>> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > http://apache-spark-developers-list.1001551.n3.nabble.com/
>> Spark-Improvement-Proposals-tp19268p19359.html
>> >>>> >> >
>> >>>> >> > To start a new topic under Apache Spark Developers List, email
>> >>>> >> > [hidden
>> >>>> >> > email]
>> >>>> >> > To unsubscribe from Apache Spark Developers List, click here.
>> >>>> >> > NAML
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > ________________________________
>> >>>> >> > View this message in context: RE: Spark Improvement Proposals
>> >>>> >> > Sent from the Apache Spark Developers List mailing list archive
>> at
>> >>>> >> > Nabble.com.
>> >>>> >>
>> >>>> >> ------------------------------------------------------------
>> ---------
>> >>>> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >>>> >>
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > --
>> >>>> > Ryan Blue
>> >>>> > Software Engineer
>> >>>> > Netflix
>> >>>>
>> >>>> ------------------------------------------------------------
>> ---------
>> >>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >>>>
>> >>>
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Mime
View raw message