spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Re: Spark Improvement Proposals
Date Mon, 02 Jan 2017 15:45:55 GMT
I'm bumping this one more time for the new year, and then I'm giving up.

Please, fix your process, even if it isn't exactly the way I suggested.

On Tue, Nov 8, 2016 at 11:14 AM, Ryan Blue <rblue@netflix.com> wrote:
> On lazy consensus as opposed to voting:
>
> First, why lazy consensus? The proposal was for consensus, which is at least
> three +1 votes and no vetos. Consensus has no losing side, it requires
> getting to a point where there is agreement. Isn't that agreement what we
> want to achieve with these proposals?
>
> Second, lazy consensus only removes the requirement for three +1 votes. Why
> would we not want at least three committers to think something is a good
> idea before adopting the proposal?
>
> rb
>
> On Tue, Nov 8, 2016 at 8:13 AM, Cody Koeninger <cody@koeninger.org> wrote:
>>
>> So there are some minor things (the Where section heading appears to
>> be dropped; wherever this document is posted it needs to actually link
>> to a jira filter showing current / past SIPs) but it doesn't look like
>> I can comment on the google doc.
>>
>> The major substantive issue that I have is that this version is
>> significantly less clear as to the outcome of an SIP.
>>
>> The apache example of lazy consensus at
>> http://apache.org/foundation/voting.html#LazyConsensus involves an
>> explicit announcement of an explicit deadline, which I think are
>> necessary for clarity.
>>
>>
>>
>> On Mon, Nov 7, 2016 at 1:55 PM, Reynold Xin <rxin@databricks.com> wrote:
>> > It turned out suggested edits (trackable) don't show up for non-owners,
>> > so
>> > I've just merged all the edits in place. It should be visible now.
>> >
>> > On Mon, Nov 7, 2016 at 10:10 AM, Reynold Xin <rxin@databricks.com>
>> > wrote:
>> >>
>> >> Oops. Let me try figure that out.
>> >>
>> >>
>> >> On Monday, November 7, 2016, Cody Koeninger <cody@koeninger.org> wrote:
>> >>>
>> >>> Thanks for picking up on this.
>> >>>
>> >>> Maybe I fail at google docs, but I can't see any edits on the document
>> >>> you linked.
>> >>>
>> >>> Regarding lazy consensus, if the board in general has less of an issue
>> >>> with that, sure.  As long as it is clearly announced, lasts at least
>> >>> 72 hours, and has a clear outcome.
>> >>>
>> >>> The other points are hard to comment on without being able to see the
>> >>> text in question.
>> >>>
>> >>>
>> >>> On Mon, Nov 7, 2016 at 3:11 AM, Reynold Xin <rxin@databricks.com>
>> >>> wrote:
>> >>> > I just looked through the entire thread again tonight - there are
a
>> >>> > lot
>> >>> > of
>> >>> > great ideas being discussed. Thanks Cody for taking the first crack
>> >>> > at
>> >>> > the
>> >>> > proposal.
>> >>> >
>> >>> > I want to first comment on the context. Spark is one of the most
>> >>> > innovative
>> >>> > and important projects in (big) data -- overall technical decisions
>> >>> > made in
>> >>> > Apache Spark are sound. But of course, a project as large and active
>> >>> > as
>> >>> > Spark always have room for improvement, and we as a community should
>> >>> > strive
>> >>> > to take it to the next level.
>> >>> >
>> >>> > To that end, the two biggest areas for improvements in my opinion
>> >>> > are:
>> >>> >
>> >>> > 1. Visibility: There are so much happening that it is difficult
to
>> >>> > know
>> >>> > what
>> >>> > really is going on. For people that don't follow closely, it is
>> >>> > difficult to
>> >>> > know what the important initiatives are. Even for people that do
>> >>> > follow, it
>> >>> > is difficult to know what specific things require their attention,
>> >>> > since the
>> >>> > number of pull requests and JIRA tickets are high and it's difficult
>> >>> > to
>> >>> > extract signal from noise.
>> >>> >
>> >>> > 2. Solicit user (broadly defined, including developers themselves)
>> >>> > input
>> >>> > more proactively: At the end of the day the project provides value
>> >>> > because
>> >>> > users use it. Users can't tell us exactly what to build, but it
is
>> >>> > important
>> >>> > to get their inputs.
>> >>> >
>> >>> >
>> >>> > I've taken Cody's doc and edited it:
>> >>> >
>> >>> >
>> >>> > https://docs.google.com/document/d/1-Zdi_W-wtuxS9hTK0P9qb2x-nRanvXmnZ7SUi4qMljg/edit#heading=h.36ut37zh7w2b
>> >>> > (I've made all my modifications trackable)
>> >>> >
>> >>> > There are couple high level changes I made:
>> >>> >
>> >>> > 1. I've consulted a board member and he recommended lazy consensus
>> >>> > as
>> >>> > opposed to voting. The reason being in voting there can easily
be a
>> >>> > "loser'
>> >>> > that gets outvoted.
>> >>> >
>> >>> > 2. I made it lighter weight, and renamed "strategy" to "optional
>> >>> > design
>> >>> > sketch". Echoing one of the earlier email: "IMHO so far aside from
>> >>> > tagging
>> >>> > things and linking them elsewhere simply having design docs and
>> >>> > prototypes
>> >>> > implementations in PRs is not something that has not worked so
far".
>> >>> >
>> >>> > 3. I made some the language tweaks to focus more on visibility.
For
>> >>> > example,
>> >>> > "The purpose of an SIP is to inform and involve", rather than just
>> >>> > "involve". SIPs should also have at least two emails that go to
>> >>> > dev@.
>> >>> >
>> >>> >
>> >>> > While I was editing this, I thought we really needed a suggested
>> >>> > template
>> >>> > for design doc too. I will get to that too ...
>> >>> >
>> >>> >
>> >>> > On Tue, Nov 1, 2016 at 12:09 AM, Reynold Xin <rxin@databricks.com>
>> >>> > wrote:
>> >>> >>
>> >>> >> Most things looked OK to me too, although I do plan to take
a
>> >>> >> closer
>> >>> >> look
>> >>> >> after Nov 1st when we cut the release branch for 2.1.
>> >>> >>
>> >>> >>
>> >>> >> On Mon, Oct 31, 2016 at 3:12 PM, Marcelo Vanzin
>> >>> >> <vanzin@cloudera.com>
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> The proposal looks OK to me. I assume, even though it's
not
>> >>> >>> explicitly
>> >>> >>> called, that voting would happen by e-mail? A template
for the
>> >>> >>> proposal document (instead of just a bullet nice) would
also be
>> >>> >>> nice,
>> >>> >>> but that can be done at any time.
>> >>> >>>
>> >>> >>> BTW, shameless plug: I filed SPARK-18085 which I consider
a
>> >>> >>> candidate
>> >>> >>> for a SIP, given the scope of the work. The document attached
even
>> >>> >>> somewhat matches the proposed format. So if anyone wants
to try
>> >>> >>> out
>> >>> >>> the process...
>> >>> >>>
>> >>> >>> On Mon, Oct 31, 2016 at 10:34 AM, Cody Koeninger
>> >>> >>> <cody@koeninger.org>
>> >>> >>> wrote:
>> >>> >>> > Now that spark summit europe is over, are any committers
>> >>> >>> > interested
>> >>> >>> > in
>> >>> >>> > moving forward with this?
>> >>> >>> >
>> >>> >>> >
>> >>> >>> >
>> >>> >>> >
>> >>> >>> > https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark-improvement-proposals.md
>> >>> >>> >
>> >>> >>> > Or are we going to let this discussion die on the
vine?
>> >>> >>> >
>> >>> >>> > On Mon, Oct 17, 2016 at 10:05 AM, Tomasz Gawęda
>> >>> >>> > <tomasz.gaweda@outlook.com> wrote:
>> >>> >>> >> Maybe my mail was not clear enough.
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>> >> I didn't want to write "lets focus on Flink" or
any other
>> >>> >>> >> framework.
>> >>> >>> >> The
>> >>> >>> >> idea with benchmarks was to show two things:
>> >>> >>> >>
>> >>> >>> >> - why some people are doing bad PR for Spark
>> >>> >>> >>
>> >>> >>> >> - how - in easy way - we can change it and show
that Spark is
>> >>> >>> >> still on
>> >>> >>> >> the
>> >>> >>> >> top
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>> >> No more, no less. Benchmarks will be helpful,
but I don't think
>> >>> >>> >> they're the
>> >>> >>> >> most important thing in Spark :) On the Spark
main page there
>> >>> >>> >> is
>> >>> >>> >> still
>> >>> >>> >> chart
>> >>> >>> >> "Spark vs Hadoop". It is important to show that
framework is
>> >>> >>> >> not
>> >>> >>> >> the
>> >>> >>> >> same
>> >>> >>> >> Spark with other API, but much faster and optimized,
comparable
>> >>> >>> >> or
>> >>> >>> >> even
>> >>> >>> >> faster than other frameworks.
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>> >> About real-time streaming, I think it would be
just good to see
>> >>> >>> >> it
>> >>> >>> >> in
>> >>> >>> >> Spark.
>> >>> >>> >> I very like current Spark model, but many voices
that says "we
>> >>> >>> >> need
>> >>> >>> >> more" -
>> >>> >>> >> community should listen also them and try to help
them. With
>> >>> >>> >> SIPs
>> >>> >>> >> it
>> >>> >>> >> would
>> >>> >>> >> be easier, I've just posted this example as "thing
that may be
>> >>> >>> >> changed
>> >>> >>> >> with
>> >>> >>> >> SIP".
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>> >> I very like unification via Datasets, but there
is a lot of
>> >>> >>> >> algorithms
>> >>> >>> >> inside - let's make easy API, but with strong
background
>> >>> >>> >> (articles,
>> >>> >>> >> benchmarks, descriptions, etc) that shows that
Spark is still
>> >>> >>> >> modern
>> >>> >>> >> framework.
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>> >> Maybe now my intention will be clearer :) As I
said
>> >>> >>> >> organizational
>> >>> >>> >> ideas
>> >>> >>> >> were already mentioned and I agree with them,
my mail was just
>> >>> >>> >> to
>> >>> >>> >> show
>> >>> >>> >> some
>> >>> >>> >> aspects from my side, so from theside of developer
and person
>> >>> >>> >> who
>> >>> >>> >> is
>> >>> >>> >> trying
>> >>> >>> >> to help others with Spark (via StackOverflow or
other ways)
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>> >> Pozdrawiam / Best regards,
>> >>> >>> >>
>> >>> >>> >> Tomasz
>> >>> >>> >>
>> >>> >>> >>
>> >>> >>> >> ________________________________
>> >>> >>> >> Od: Cody Koeninger <cody@koeninger.org>
>> >>> >>> >> Wysłane: 17 października 2016 16:46
>> >>> >>> >> Do: Debasish Das
>> >>> >>> >> DW: Tomasz Gawęda; dev@spark.apache.org
>> >>> >>> >> Temat: Re: Spark Improvement Proposals
>> >>> >>> >>
>> >>> >>> >> I think narrowly focusing on Flink or benchmarks
is missing my
>> >>> >>> >> point.
>> >>> >>> >>
>> >>> >>> >> My point is evolve or die.  Spark's governance
and organization
>> >>> >>> >> is
>> >>> >>> >> hampering its ability to evolve technologically,
and it needs
>> >>> >>> >> to
>> >>> >>> >> change.
>> >>> >>> >>
>> >>> >>> >> On Sun, Oct 16, 2016 at 9:21 PM, Debasish Das
>> >>> >>> >> <debasish.das83@gmail.com>
>> >>> >>> >> wrote:
>> >>> >>> >>> Thanks Cody for bringing up a valid point...I
picked up Spark
>> >>> >>> >>> in
>> >>> >>> >>> 2014
>> >>> >>> >>> as
>> >>> >>> >>> soon as I looked into it since compared to
writing Java
>> >>> >>> >>> map-reduce
>> >>> >>> >>> and
>> >>> >>> >>> Cascading code, Spark made writing distributed
code fun...But
>> >>> >>> >>> now
>> >>> >>> >>> as
>> >>> >>> >>> we
>> >>> >>> >>> went
>> >>> >>> >>> deeper with Spark and real-time streaming
use-case gets more
>> >>> >>> >>> prominent, I
>> >>> >>> >>> think it is time to bring a messaging model
in conjunction
>> >>> >>> >>> with
>> >>> >>> >>> the
>> >>> >>> >>> batch/micro-batch API that Spark is good at....akka-streams
>> >>> >>> >>> close
>> >>> >>> >>> integration with spark micro-batching APIs
looks like a great
>> >>> >>> >>> direction to
>> >>> >>> >>> stay in the game with Apache Flink...Spark
2.0 integrated
>> >>> >>> >>> streaming
>> >>> >>> >>> with
>> >>> >>> >>> batch with the assumption is that micro-batching
is sufficient
>> >>> >>> >>> to
>> >>> >>> >>> run
>> >>> >>> >>> SQL
>> >>> >>> >>> commands on stream but do we really have time
to do SQL
>> >>> >>> >>> processing at
>> >>> >>> >>> streaming data within 1-2 seconds ?
>> >>> >>> >>>
>> >>> >>> >>> After reading the email chain, I started to
look into Flink
>> >>> >>> >>> documentation
>> >>> >>> >>> and if you compare it with Spark documentation,
I think we
>> >>> >>> >>> have
>> >>> >>> >>> major
>> >>> >>> >>> work
>> >>> >>> >>> to do detailing out Spark internals so that
more people from
>> >>> >>> >>> community
>> >>> >>> >>> start
>> >>> >>> >>> to take active role in improving the issues
so that Spark
>> >>> >>> >>> stays
>> >>> >>> >>> strong
>> >>> >>> >>> compared to Flink.
>> >>> >>> >>>
>> >>> >>> >>>
>> >>> >>> >>> https://cwiki.apache.org/confluence/display/SPARK/Spark+Internals
>> >>> >>> >>>
>> >>> >>> >>>
>> >>> >>> >>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
>> >>> >>> >>>
>> >>> >>> >>> Spark is no longer an engine that works for
micro-batch and
>> >>> >>> >>> batch...We
>> >>> >>> >>> (and
>> >>> >>> >>> I am sure many others) are pushing spark as
an engine for
>> >>> >>> >>> stream
>> >>> >>> >>> and
>> >>> >>> >>> query
>> >>> >>> >>> processing.....we need to make it a state-of-the-art
engine
>> >>> >>> >>> for
>> >>> >>> >>> high
>> >>> >>> >>> speed
>> >>> >>> >>> streaming data and user queries as well !
>> >>> >>> >>>
>> >>> >>> >>> On Sun, Oct 16, 2016 at 1:30 PM, Tomasz Gawęda
>> >>> >>> >>> <tomasz.gaweda@outlook.com>
>> >>> >>> >>> wrote:
>> >>> >>> >>>>
>> >>> >>> >>>> Hi everyone,
>> >>> >>> >>>>
>> >>> >>> >>>> I'm quite late with my answer, but I think
my suggestions may
>> >>> >>> >>>> help a
>> >>> >>> >>>> little bit. :) Many technical and organizational
topics were
>> >>> >>> >>>> mentioned,
>> >>> >>> >>>> but I want to focus on these negative
posts about Spark and
>> >>> >>> >>>> about
>> >>> >>> >>>> "haters"
>> >>> >>> >>>>
>> >>> >>> >>>> I really like Spark. Easy of use, speed,
very good community
>> >>> >>> >>>> -
>> >>> >>> >>>> it's
>> >>> >>> >>>> everything here. But Every project has
to "flight" on
>> >>> >>> >>>> "framework
>> >>> >>> >>>> market"
>> >>> >>> >>>> to be still no 1. I'm following many Spark
and Big Data
>> >>> >>> >>>> communities,
>> >>> >>> >>>> maybe my mail will inspire someone :)
>> >>> >>> >>>>
>> >>> >>> >>>> You (every Spark developer; so far I didn't
have enough time
>> >>> >>> >>>> to
>> >>> >>> >>>> join
>> >>> >>> >>>> contributing to Spark) has done excellent
job. So why are
>> >>> >>> >>>> some
>> >>> >>> >>>> people
>> >>> >>> >>>> saying that Flink (or other framework)
is better, like it was
>> >>> >>> >>>> posted
>> >>> >>> >>>> in
>> >>> >>> >>>> this mailing list? No, not because that
framework is better
>> >>> >>> >>>> in
>> >>> >>> >>>> all
>> >>> >>> >>>> cases.. In my opinion, many of these discussions
where
>> >>> >>> >>>> started
>> >>> >>> >>>> after
>> >>> >>> >>>> Flink marketing-like posts. Please look
at StackOverflow
>> >>> >>> >>>> "Flink
>> >>> >>> >>>> vs
>> >>> >>> >>>> ...."
>> >>> >>> >>>> posts, almost every post in "winned" by
Flink. Answers are
>> >>> >>> >>>> sometimes
>> >>> >>> >>>> saying nothing about other frameworks,
Flink's users (often
>> >>> >>> >>>> PMC's)
>> >>> >>> >>>> are
>> >>> >>> >>>> just posting same information about real-time
streaming,
>> >>> >>> >>>> about
>> >>> >>> >>>> delta
>> >>> >>> >>>> iterations, etc. It look smart and very
often it is marked as
>> >>> >>> >>>> an
>> >>> >>> >>>> aswer,
>> >>> >>> >>>> even if - in my opinion - there wasn't
told all the truth.
>> >>> >>> >>>>
>> >>> >>> >>>>
>> >>> >>> >>>> My suggestion: I don't have enough money
and knowledgle to
>> >>> >>> >>>> perform
>> >>> >>> >>>> huge
>> >>> >>> >>>> performance test. Maybe some company,
that supports Spark
>> >>> >>> >>>> (Databricks,
>> >>> >>> >>>> Cloudera? - just saying you're most visible
in community :) )
>> >>> >>> >>>> could
>> >>> >>> >>>> perform performance test of:
>> >>> >>> >>>>
>> >>> >>> >>>> - streaming engine - probably Spark will
loose because of
>> >>> >>> >>>> mini-batch
>> >>> >>> >>>> model, however currently the difference
should be much lower
>> >>> >>> >>>> that in
>> >>> >>> >>>> previous versions
>> >>> >>> >>>>
>> >>> >>> >>>> - Machine Learning models
>> >>> >>> >>>>
>> >>> >>> >>>> - batch jobs
>> >>> >>> >>>>
>> >>> >>> >>>> - Graph jobs
>> >>> >>> >>>>
>> >>> >>> >>>> - SQL queries
>> >>> >>> >>>>
>> >>> >>> >>>> People will see that Spark is envolving
and is also a modern
>> >>> >>> >>>> framework,
>> >>> >>> >>>> because after reading posts mentioned
above people may think
>> >>> >>> >>>> "it
>> >>> >>> >>>> is
>> >>> >>> >>>> outdated, future is in framework X".
>> >>> >>> >>>>
>> >>> >>> >>>> Matei Zaharia posted excellent blog post
about how Spark
>> >>> >>> >>>> Structured
>> >>> >>> >>>> Streaming beats every other framework
in terms of easy-of-use
>> >>> >>> >>>> and
>> >>> >>> >>>> reliability. Performance tests, done in
various environments
>> >>> >>> >>>> (in
>> >>> >>> >>>> example: laptop, small 2 node cluster,
10-node cluster,
>> >>> >>> >>>> 20-node
>> >>> >>> >>>> cluster), could be also very good marketing
stuff to say
>> >>> >>> >>>> "hey,
>> >>> >>> >>>> you're
>> >>> >>> >>>> telling that you're better, but Spark
is still faster and is
>> >>> >>> >>>> still
>> >>> >>> >>>> getting even more fast!". This would be
based on facts (just
>> >>> >>> >>>> numbers),
>> >>> >>> >>>> not opinions. It would be good for companies,
for marketing
>> >>> >>> >>>> puproses
>> >>> >>> >>>> and
>> >>> >>> >>>> for every Spark developer
>> >>> >>> >>>>
>> >>> >>> >>>>
>> >>> >>> >>>> Second: real-time streaming. I've written
some time ago about
>> >>> >>> >>>> real-time
>> >>> >>> >>>> streaming support in Spark Structured
Streaming. Some work
>> >>> >>> >>>> should be
>> >>> >>> >>>> done to make SSS more low-latency, but
I think it's possible.
>> >>> >>> >>>> Maybe
>> >>> >>> >>>> Spark may look at Gearpump, which is also
built on top of
>> >>> >>> >>>> Akka?
>> >>> >>> >>>> I
>> >>> >>> >>>> don't
>> >>> >>> >>>> know yet, it is good topic for SIP. However
I think that
>> >>> >>> >>>> Spark
>> >>> >>> >>>> should
>> >>> >>> >>>> have real-time streaming support. Currently
I see many
>> >>> >>> >>>> posts/comments
>> >>> >>> >>>> that "Spark has too big latency". Spark
Streaming is doing
>> >>> >>> >>>> very
>> >>> >>> >>>> good
>> >>> >>> >>>> jobs with micro-batches, however I think
it is possible to
>> >>> >>> >>>> add
>> >>> >>> >>>> also
>> >>> >>> >>>> more
>> >>> >>> >>>> real-time processing.
>> >>> >>> >>>>
>> >>> >>> >>>> Other people said much more and I agree
with proposal of SIP.
>> >>> >>> >>>> I'm
>> >>> >>> >>>> also
>> >>> >>> >>>> happy that PMC's are not saying that they
will not listen to
>> >>> >>> >>>> users,
>> >>> >>> >>>> but
>> >>> >>> >>>> they really want to make Spark better
for every user.
>> >>> >>> >>>>
>> >>> >>> >>>>
>> >>> >>> >>>> What do you think about these two topics?
Especially I'm
>> >>> >>> >>>> looking
>> >>> >>> >>>> at
>> >>> >>> >>>> Cody
>> >>> >>> >>>> (who has started this topic) and PMCs
:)
>> >>> >>> >>>>
>> >>> >>> >>>> Pozdrawiam / Best regards,
>> >>> >>> >>>>
>> >>> >>> >>>> Tomasz
>> >>> >>> >>>>
>> >>> >>> >>>>
>> >>> >>>
>> >>> >>
>> >>> >
>> >>> >
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message