Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
From: Bolke de Bruin <bdbruin@gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\))
Subject: Re: SparkOperator - tips and feedback?
Date: Sat, 18 Mar 2017 19:24:31 -0700
References: <CANSvDjo+u-irKGxAo-sAj-rA+u1EnDtMx_spU4J_HCwEi3WmDQ@mail.gmail.com>
 <CACdThQ2cgaBRq3g7rt8LtCQ4zV+3xDzKx8B2uwST-zww4_ZpAA@mail.gmail.com>
 <CANSvDjo1T05igAUHHMREwYd_E_-VrQaTdPBh8NgkOVCEzm4GMA@mail.gmail.com>
To: dev@airflow.incubator.apache.org
In-Reply-To: <CANSvDjo1T05igAUHHMREwYd_E_-VrQaTdPBh8NgkOVCEzm4GMA@mail.gmail.com>
Message-Id: <B0BC78BB-D05A-4802-B1B8-AD3011A450F8@gmail.com>
archived-at: Sun, 19 Mar 2017 02:48:50 -0000

A spark operator exists as of 1.8.0 (which will be released tomorrow), =
you might want to take a look at that. I know that an update is coming =
to that operator that improves communication with Yarn.

Bolke

> On 18 Mar 2017, at 18:43, Russell Jurney <russell.jurney@gmail.com> =
wrote:
>=20
> Ruslan, thanks for your feedback.
>=20
> You mean the spark-submit context? Or like the SparkContext and
> SparkSession? I don't think we could keep that alive, because it =
wouldn't
> work out with multiple calls to spark-submit. I do feel your pain, =
though.
> Maybe someone else can see how this might be done?
>=20
> If SparkContext was able to open the spark/pyspark console, then =
multiple
> job submissions would be possible. I didn't have this in mind but an
> InteractiveSparkContext or SparkConsoleContext might be able to do =
this?
>=20
> Russell Jurney @rjurney <http://twitter.com/rjurney>
> russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney> FB
> <http://facebook.com/jurney> datasyndrome.com
>=20
> On Sat, Mar 18, 2017 at 3:02 PM, Ruslan Dautkhanov =
<dautkhanov@gmail.com>
> wrote:
>=20
>> +1 Great idea.
>>=20
>> my two cents - it would be nice (as an option) if SparkOperator would =
be
>> able to keep context open between different calls,
>> as it takes 30+ seconds to create a new context (on our cluster). Not =
sure
>> how well it fits Airflow architecture.
>>=20
>>=20
>>=20
>> --
>> Ruslan Dautkhanov
>>=20
>> On Sat, Mar 18, 2017 at 3:45 PM, Russell Jurney =
<russell.jurney@gmail.com>
>> wrote:
>>=20
>>> What do people think about creating a SparkOperator that uses
>> spark-submit
>>> to submit jobs? Would work for Scala/Java Spark and PySpark. The =
patterns
>>> outlined in my presentation on Airflow and PySpark
>>> <http://bit.ly/airflow_pyspark> would fit well inside an Operator, I
>>> think.
>>> BashOperator works, but why not tailor something to spark-submit?
>>>=20
>>> I'm open to doing the work, but I wanted to see what people though =
about
>> it
>>> and get feedback about things they would like to see in =
SparkOperator and
>>> get any pointers people had to doing the implementation.
>>>=20
>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>> russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney> =
FB
>>> <http://facebook.com/jurney> datasyndrome.com
>>>=20
>>=20