airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Jurney <russell.jur...@gmail.com>
Subject SparkOperator - tips and feedback?
Date Sat, 18 Mar 2017 21:45:52 GMT
What do people think about creating a SparkOperator that uses spark-submit
to submit jobs? Would work for Scala/Java Spark and PySpark. The patterns
outlined in my presentation on Airflow and PySpark
<http://bit.ly/airflow_pyspark> would fit well inside an Operator, I think.
BashOperator works, but why not tailor something to spark-submit?

I'm open to doing the work, but I wanted to see what people though about it
and get feedback about things they would like to see in SparkOperator and
get any pointers people had to doing the implementation.

Russell Jurney @rjurney <http://twitter.com/rjurney>
russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney> FB
<http://facebook.com/jurney> datasyndrome.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message