airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 蒋晓峰 <thanosxnicho...@gmail.com>
Subject [AIP-35] Add Signal Based Scheduling To Airflow
Date Tue, 16 Jun 2020 12:59:47 GMT
Hello everyone,

Sending a message to everyone and collecting feedback on the AIP-35 on
adding signal-based scheduling. This was previously briefly mentioned in
the discussion of development slack channel. The key motivation of this
proposal is to support a mixture of batch and stream jobs in the same
workflow.

In practice, there are many business logics that need collaboration between
streaming and batch jobs. For example, in machine learning, an online
learning job is a streaming job running forever. It emits a model
periodically and a batch evaluate job should run to validate each model. So
this is a streaming-triggered-batch case. If we continue from the above
example, once the model passes the validation, the model needs to be
deployed to a serving job. That serving job could be a streaming job that
keeps polling records from Kafka and uses a model to do prediction. And the
availability of a new model requires that streaming prediction job either
to restart, or to reload the new model on the fly. In either case, it is a
batch-to-streaming job triggering.

At this point above, people basically have to invent their own way to deal
with such workflows consisting of both batch and streaming jobs. I think
having a system that can help smoothly work with a mixture of streaming and
batch jobs is valuable here.

This AIP-35 focuses on signal-based scheduling to let the operators send
signals to the scheduler to trigger a scheduling action, such as starting
jobs, stopping jobs and restarting jobs. With the change of
scheduling mechanism, a streaming job can send signals to the scheduler to
indicate the state change of the dependencies.

Signal-based scheduling allows the scheduler to know the change of the
dependency state immediately without periodically querying the metadata
database. This also allows potential support for richer scheduling
semantics such as periodic execution and manual trigger at per operator
granularity.

Changes proposed:

- Signals are used to define conditions that must be met to run an
operator. State change of the upstream tasks is one type of the signals.
There may be other types of signals. The scheduler may take different
actions when receiving different signals. To let the operators take signals
as their starting condition, we propose to introduce SignalOperator which
is mentioned in the public interface section.
- A notification service is necessary to help receive and propagate the
signals from the operators and other sources to the scheduler. Upon
receiving a signal, the scheduler can take action according to the
predefined signal-based conditions on the operators. Therefore we propose
to introduce a Signal Notification Service component to Airflow.

Please see related documents and PRs for details:

AIP:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-35+Add+Signal+Based+Scheduling+To+Airflow
Issue: https://github.com/apache/airflow/issues/9321

Please let me know if there are any aspects that you agree/disagree with or
need more clarification (especially the SignalOperator and SignalService).
Any comments are welcome and I am looking forward to it!

Thanks,
Nicholas

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message