airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lightning Analytics (Jira)" <>
Subject [jira] [Created] (AIRFLOW-6384) Cusomt Spark Livy Operator
Date Sat, 28 Dec 2019 18:56:00 GMT
Lightning Analytics created AIRFLOW-6384:

             Summary: Cusomt Spark Livy Operator
                 Key: AIRFLOW-6384
             Project: Apache Airflow
          Issue Type: New Feature
          Components: operators
    Affects Versions: 1.10.6, 1.10.5
            Reporter: Lightning Analytics

Here at lightning analytics, we always thrive for innovation. Keeping in mind the challenges
posed by open source technologies in terms of integration, the aim is to build custom solutions
to bridge the gap, thereby enabling ease of deployment. The orchestration tool, Airflow, presents
certain challenges for integration with components in the cloud environment. Airflow has in-built
support for operators which is an integral component for integration. It was observed that
the operator for Livy Rest API is not available.

An operator, Spark Livy Operator, has been developed and tested that submits spark jobs to
the cluster using the Livy Rest API. Livy provides two sessions namely, Batch and Interactive.
The custom Spark Livy Operator provides Interactive session submission and additionally sends
heart beats to check the status of the batch. Extensive testing has been performed using Celery
executor, and all available hooks and libraries in Airflow has been integrated.

We would like to publish the same to the Apache open source community for enhancement, use
and distribution. Please inform about the steps for creating a Git pull request for Airflow.

This message was sent by Atlassian Jira

View raw message