spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <>
Subject Re: Should Flume integration be behind a profile?
Date Sun, 01 Oct 2017 22:50:49 GMT
Probably should do 1, and then it is an easier transition in 3.0.

On Sun, Oct 1, 2017 at 1:28 AM Sean Owen <> wrote:

> I tried and failed to do this in
> because it became clear
> that the Flume examples would have to be removed to make this work, too.
> (Well, you can imagine other solutions with extra source dirs or modules
> for flume examples enabled by a profile, but that doesn't help the docs and
> is nontrivial complexity for little gain.)
> It kind of suggests Flume support should be deprecated if it's put behind
> a profile. Like with Kafka 0.8. (This is why I'm raising it again to the
> whole list.)
> Any preferences among:
> 1. Put Flume behind a profile, remove examples, deprecate
> 2. Put Flume behind a profile, remove examples, but don't deprecate
> 3. Punt until Spark 3.0, when this integration would probably be removed
> entirely (?)
> On Tue, Sep 26, 2017 at 10:36 AM Sean Owen <> wrote:
>> Not a big deal, but I'm wondering whether Flume integration should at
>> least be opt-in and behind a profile? it still sees some use (at least on
>> our end) but not applicable to the majority of users. Most other
>> third-party framework integrations are behind a profile, like YARN, Mesos,
>> Kinesis, Kafka 0.8, Docker. Just soliciting comments, not arguing for it.
>> (Well, actually it annoys me that the Flume integration always fails to
>> compile in IntelliJ unless you generate the sources manually)

View raw message