airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kaxil Naik <kaxiln...@gmail.com>
Subject Re: [VOTE] Release Airflow 1.10.4 from RC3
Date Fri, 19 Jul 2019 18:57:20 GMT
If you are using num_runs or run_duration, remove them. We have deprecated
run_duration in master (we should probably cherry-pick that in next
release) -
https://github.com/apache/airflow/blob/30defe130df1399d8ff41a350f714af820113161/UPDATING.md#remove-run_duration
(Note in updating.md)

We should remove `num_runs` too and we have a jira open for that -
https://jira.apache.org/jira/browse/AIRFLOW-3533



On Thu, Jul 18, 2019 at 11:15 PM James Meickle
<jmeickle@quantopian.com.invalid> wrote:

> Yes, it's zombie processes owned by systemd. Here's a writeup with some
> various logs:
> https://gist.github.com/Eronarn/c5bc2df607d168d6fda3a70700c941d9
>
> It's reproducible (lmk if you want to co-pilot on debugging), and we're up
> to 3 zombies in the hour or two since our most recent reboot. So it won't
> immediately take a cluster down, but it is a steady resource leak that
> isn't fixed by restarting the scheduler process.
>
> On Thu, Jul 18, 2019 at 12:31 PM Ash Berlin-Taylor <ash@apache.org> wrote:
>
> > Errk, that's not good.
> >
> > When you say orphaned do you mean zombie processes owned by pid 1, or
> just
> > "process that are hanging around?"
> >
> > https://github.com/apache/airflow/pull/5605 might help if it's the later
> > (but I suspect it isn't)
> >
> > We've also had reports of the scheduler having a memory leak since
> ~1.10.2
> > https://issues.apache.org/jira/browse/AIRFLOW-4593 so that may be part
> of
> > the OOMing too. (If this is the cause of your problem then .4 isn't worse
> > than .3 I don't think?)
> >
> > -ash
> >
> > > On 18 Jul 2019, at 15:54, James Meickle <jmeickle@quantopian.com
> .INVALID>
> > wrote:
> > >
> > > Hi folks,
> > >
> > > Sorry to throw a wrench into this, but we're found that this release is
> > > leaving orphaned processes that eventually OOMing the scheduler. 4
> > clusters
> > > have this problem post-upgrade, while 1 non-upgraded cluster doesn't
> have
> > > this problem.
> > >
> > > In our case we're using Supervisor to manage scheduler restarts. I am
> > still
> > > investigating whether this is an interaction with our unique Supervisor
> > > config, or if it's actually purely an Airflow problem. This switch to
> > > multiprocessing.Manager looks like a promising avenue of investigation
> > > though:
> > >
> >
> https://github.com/apache/airflow/compare/1.10.3...1.10.4rc3#diff-ced4fd65ce02db58eed692eef6e01d05L495
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Jul 17, 2019 at 3:35 PM Bolke de Bruin <bdbruin@gmail.com>
> > wrote:
> > >
> > >> +1, binding.
> > >>
> > >> Thanks ash!
> > >>
> > >> Verstuurd vanaf mijn iPad
> > >>
> > >>> Op 17 jul. 2019 om 20:45 heeft Kaxil Naik <kaxilnaik@gmail.com> het
> > >> volgende geschreven:
> > >>>
> > >>> +1 (binding) Ran DAGs in both UIs. LGTM
> > >>>
> > >>> Great Job Ash. Appreciate all the effort you put into this.
> > >>>
> > >>> Regards,
> > >>> Kaxil
> > >>>
> > >>> On Wed, Jul 17, 2019 at 10:30 PM Felix Uellendall
> > <feluelle@pm.me.invalid
> > >>>
> > >>> wrote:
> > >>>
> > >>>> +1 (non-binding) flawlessly passed all my tests via rbac and classic
> > ui.
> > >>>> Also tested it on production-level dags. Great job Ash, thanks :)
> > >>>>
> > >>>> Kind regards,
> > >>>> Felix
> > >>>>
> > >>>> Sent from ProtonMail mobile
> > >>>>
> > >>>> -------- Original Message --------
> > >>>>> On Jul 17, 2019, 16:41, Ash Berlin-Taylor wrote:
> > >>>>>
> > >>>>> Thanks Andrii - I missed it to when reviewing the PR so it's my
> fault
> > >>>> too. Make the PR against the v[1-10](tel:110)-stable branch please.
> > >>>>>
> > >>>>> -ash
> > >>>>>
> > >>>>>> On 17 Jul [2019](tel:2019), at 14:25, Andrii Soldatenko <
> > >>>> andrii.soldatenko@gmail.com> wrote:
> > >>>>>>
> > >>>>>> @Ash, i'll fix wrong section bug.
> > >>>>>>
> > >>>>>> Sorry about that.
> > >>>>>>
> > >>>>>> On Wed, Jul 17, [2019](tel:2019) at 4:11 PM Robin Edwards <
> > >>>> rob@bidnamic.com> wrote:
> > >>>>>>
> > >>>>>>> +1 (none binding) - been running in production since RC2.
> > >>>>>>>
> > >>>>>>> Thanks for all your hard work
> > >>>>>>>
> > >>>>>>> R
> > >>>>>>>
> > >>>>>>> On Tue, 16 Jul [2019](tel:2019) at 21:15, Ash Berlin-Taylor <
> > >>>> ash@apache.org> wrote:
> > >>>>>>>>
> > >>>>>>>> Thanks for testing.
> > >>>>>>>>
> > >>>>>>>> On 1) everyone should run upgradedb on every upgrade. The
> > behaviour
> > >> of
> > >>>>>>> not running it wasn't great.
> > >>>>>>>>
> > >>>>>>>> 2) I thought we set deprecation on the ES logging config vars,
> > >> except
> > >>>> we
> > >>>>>>> put the depreciation under the wrong section:
> > >>>>>>>
> > >>>>
> > >>
> >
> https://github.com/apache/airflow/blob/1.10.4rc3/airflow/configuration.py#L147-L153
> > >>>>>>> :(
> > >>>>>>>>
> > >>>>>>>> On 16 July [2019 21](tel:201921):06:40 BST, James Meickle <
> > >>>> jmeickle@quantopian.com.INVALID>
> > >>>>>>> wrote:
> > >>>>>>>>> +1 (nonbinding) to the release, it fixes a lot of UI issues
> we've
> > >>>> been
> > >>>>>>>>> seeing lately. Though two notes:
> > >>>>>>>>>
> > >>>>>>>>> 1) Tasks were unscheduleable until I ran an upgradedb due the
> > >> default
> > >>>>>>>>> pool
> > >>>>>>>>> change.
> > >>>>>>>>>
> > >>>>>>>>> 2) I got crash loops because I based our custom logging file
> off
> > of
> > >>>> the
> > >>>>>>>>> previous version's template. The change to the Elasticsearch
> conf
> > >>>>>>>>> variables
> > >>>>>>>>> broke this, even though we don't directly use Elasticsearch,
> > since
> > >> it
> > >>>>>>>>> was
> > >>>>>>>>> now looking for undefined conf variables. It was confusing
> until
> > I
> > >>>>>>>>> hunted
> > >>>>>>>>> around the diff and backported the changes in the default
> > template
> > >> to
> > >>>>>>>>> the
> > >>>>>>>>> template that lives in our repo.
> > >>>>>>>>>
> > >>>>>>>>> On Mon, Jul 15, [2019](tel:2019) at 10:17 AM Ash Berlin-Taylor
> <
> > >>>> ash@apache.org>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hello Airflow community,
> > >>>>>>>>>>
> > >>>>>>>>>> This email is calling a vote on the release, which will last
> for
> > >> 72
> > >>>>>>>>> hours
> > >>>>>>>>>> (2019-07-08 15:15 Z), and until three binding votes have been
> > >> cast.
> > >>>>>>>>>> Consider this my (binding) +1.
> > >>>>>>>>>>
> > >>>>>>>>>> Airflow [1.10.4](tel:1104) RC3 is available at:
> > >>>>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/1.10.4rc3/
> > >>>>>>>>>>
> > >>>>>>>>>> *apache-airflow-[1.10.4](tel:1104)rc3-source.tar.gz* is a
> source
> > >>>> release that
> > >>>>>>>>> comes
> > >>>>>>>>>> with INSTALL instructions.
> > >>>>>>>>>> *apache-airflow-[1.10.4](tel:1104)rc3-bin.tar.gz* is the
> binary
> > >>>> Python "sdist"
> > >>>>>>>>> release.
> > >>>>>>>>>> *apache_airflow-[1.10.4](tel:1104)rc3-py2.py3-none-any.whl* is
> > the
> > >>>> binary Python
> > >>>>>>>>>> "wheel" release.
> > >>>>>>>>>>
> > >>>>>>>>>> For connivence of testers the RC is on PYPI too. It can be
> > >> installed
> > >>>>>>>>> with:
> > >>>>>>>>>>
> > >>>>>>>>>> pip install 'apache-airflow==[1.10.4](tel:1104)rc3'
> > >>>>>>>>>>
> > >>>>>>>>>> Public keys are available at:
> > >>>>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
> > >>>>>>>>>>
> > >>>>>>>>>> Only votes from PMC members are binding (sorry committers),
> but
> > >>>>>>>>> members of
> > >>>>>>>>>> the community are encouraged to test the release and vote with
> > >>>>>>>>>> "(non-binding)".
> > >>>>>>>>>>
> > >>>>>>>>>> Please note that the version number excludes the `rcX` string,
> > so
> > >>>>>>>>> it's now
> > >>>>>>>>>> simply [1.10.4](tel:1104). This will allow us to rename the
> > >>>> artefact without
> > >>>>>>>>> modifying
> > >>>>>>>>>> the checksums when we actually release.
> > >>>>>>>>>>
> > >>>>>>>>>> [ ] +1 Release this package as Apache Airflow
> [1.10.4](tel:1104)
> > >>>>>>>>>> [ ] 0 No opinion
> > >>>>>>>>>> [ ] -1 Do not release this package because...
> > >>>>>>>>>>
> > >>>>>>>>>> Changes since RC2:
> > >>>>>>>>>>
> > >>>>>>>>>> [AIRFLOW-XXX] Update changelog for [1.10.4](tel:1104)rc3 [Ash
> > >>>> Berlin-Taylor]
> > >>>>>>>>>> [AIRFLOW-XXX] Disable intersphinx loading of `requests`
> modules
> > >>>>>>>>> (#[5590](tel:5590))
> > >>>>>>>>>> [Ash Berlin-Taylor]
> > >>>>>>>>>> [AIRFLOW-[4797](tel:4797)] Improve performance and behaviour
> of
> > >>>> zombie detection
> > >>>>>>>>>> (#[5511](tel:5511)) [Stefan Seelmann]
> > >>>>>>>>>> [AIRFLOW-[4911](tel:4911)] Silence the FORBIDDEN errors from
> the
> > >>>>>>>>> KubernetesExecutor
> > >>>>>>>>>> (#[5547](tel:5547)) [Andrii Soldatenko]
> > >>>>>>>>>> [AIRFLOW-[4884](tel:4884)] Roll up import_errors in RBAC UI
> > >>>> (#[5516](tel:5516))
> > >>>>>>>>> [shuwen-zhang]
> > >>>>>>>>>> [AIRFLOW-[4937](tel:4937)] Fix lodash security issue with
> > version
> > >>>> below [4.17.13](tel:41713)
> > >>>>>>>>>> (#[5572](tel:5572)) [Jarek Potiuk]
> > >>>>>>>>>> [AIRFLOW-[4934](tel:4934)] Bump Flask to resolve Werkzeug
> > ProxyFix
> > >>>> (#[5571](tel:5571))
> > >>>>>>>>> [Nathaniel
> > >>>>>>>>>> Ritholtz]
> > >>>>>>>>>> [AIRFLOW-[4934](tel:4934)] Fix ProxyFix due to Werkzeug
> upgrade
> > >>>> (#[5563](tel:5563))
> > >>>>>>>>> [Nathaniel
> > >>>>>>>>>> Ritholtz]
> > >>>>>>>>>> [AIRFLOW-[4136](tel:4136)] fix key_file of hook is overwritten
> > by
> > >>>> SSHHook
> > >>>>>>>>> connection
> > >>>>>>>>>> (#[5558](tel:5558)) [eladkal]
> > >>>>>>>>>> [AIRFLOW-XXX] Correct BaseSensorOperator docs
> > (#[5562](tel:5562))
> > >>>> [Matthew
> > >>>>>>>>> Bowden]
> > >>>>>>>>>> [AIRFLOW-[3495](tel:3495)] Validate one of query and query_uri
> > >>>> passed to
> > >>>>>>>>>> DataProcSparkSqlOperator (#[5510](tel:5510)) [eladkal]
> > >>>>>>>>>> [AIRFLOW-[4868](tel:4868)] Fix typo in
> > kubernetes/docker/build.sh
> > >>>> (#[5505](tel:5505)) [Facu]
> > >>>>>>>>>> [AIRFLOW-[4925](tel:4925)] Improve css style for Variables
> > Import
> > >>>> file field
> > >>>>>>>>> (#[5552](tel:5552))
> > >>>>>>>>>> [Felix Uellendall]
> > >>>>>>>>>> [AIRFLOW-[4906](tel:4906)] Improve debugging for the
> > >>>> SparkSubmitHook (#[5542](tel:5542))
> > >>>>>>>>> [Fokko
> > >>>>>>>>>> Driesprong]
> > >>>>>>>>>> [AIRFLOW-[4904](tel:4904)] Retrieve test config file from
> > >>>> $AIRFLOW_TEST_CONFIG
> > >>>>>>>>> (#[5540](tel:5540))
> > >>>>>>>>>> [Matthew Rastovac]
> > >>>>>>>>>> [AIRFLOW-[4926](tel:4926)] Fix example dags where its
> start_date
> > >> is
> > >>>>>>>>> datetime.utcnow()
> > >>>>>>>>>> (#[5553](tel:5553)) [Felix Uellendall]
> > >>>>>>>>>> [AIRFLOW-[4587](tel:4587)] Replace self.conn with
> > self.get_conn()
> > >> in
> > >>>>>>>>> AWSAthenaHook
> > >>>>>>>>>> (#[5545](tel:5545)) [Bhavika Tekwani]
> > >>>>>>>>>> [AIRFLOW-[1740](tel:1740)] Fix xcom creation and update via UI
> > >>>> (#[5530](tel:5530)) [Felix
> > >>>>>>>>>> Uellendall]
> > >>>>>>>>>> [AIRFLOW-[1740](tel:1740)] Fix xcom creation and update via
> RBAC
> > >> UI
> > >>>> (#[5561](tel:5561))
> > >>>>>>>>> [Felix
> > >>>>>>>>>> Uellendall]
> > >>>>>>>>>> [AIRFLOW-[4920](tel:4920)] Use html.escape instead of
> cgi.escape
> > >> to
> > >>>> fix
> > >>>>>>>>>> DeprecationWarning (#[5551](tel:5551)) [Xtreak]
> > >>>>>>>>>> [AIRFLOW-[4919](tel:4919)] DataProcJobBaseOperator
> > >>>> dataproc_*_properties
> > >>>>>>>>> templated
> > >>>>>>>>>> (#[5555](tel:5555)) [eladkal]
> > >>>>>>>>>> [AIRFLOW-[4900](tel:4900)] Resolve incompatible version of
> > >> Werkzeug
> > >>>> (#[5535](tel:5535))
> > >>>>>>>>> [Luqman
> > >>>>>>>>>> Abdul Mushawwir]
> > >>>>>>>>>> [AIRFLOW-[4478](tel:4478)] Lazily instantiate default
> resources
> > >>>> objects. (#[5259](tel:5259))
> > >>>>>>>>>> [Joshua Carp]
> > >>>>>>>>>> [AIRFLOW-[4510](tel:4510)] Don't mutate default_args during
> DAG
> > >>>> initialization
> > >>>>>>>>> (#[5277](tel:5277))
> > >>>>>>>>>> [Abhishek Ray]
> > >>>>>>>>>> [AIRFLOW-[3360](tel:3360)] Make the DAGs search respect other
> > >>>> querystring
> > >>>>>>>>> parameters
> > >>>>>>>>>> with url-search-params-polyfill for IE support
> > (#[5503](tel:5503))
> > >>>> [Alex]
> > >>>>>>>>>> [AIRFLOW-[4896](tel:4896)] Make KubernetesExecutorConfig's
> > default
> > >>>> args immutable
> > >>>>>>>>>> (#[5534](tel:5534)) [Philippe Gagnon]
> > >>>>>>>>>> [AIRFLOW-[4494](tel:4494)] Remove `shell=True` in DaskExecutor
> > >>>> (#[5273](tel:5273)) [MisLink]
> > >>>>>>>>>> [AIRFLOW-[4890](tel:4890)] Fix Log link in TaskInstance's View
> > for
> > >>>> Non-RBAC
> > >>>>>>>>> (#[5525](tel:5525))
> > >>>>>>>>>> [Felix Uellendall]
> > >>>>>>>>>> [AIRFLOW-[4892](tel:4892)] Fix connection creation via UIs
> > >>>> (#[5527](tel:5527)) [Ash
> > >>>>>>>>> Berlin-Taylor]
> > >>>>>>>>>> [AIRFLOW-[4862](tel:4862)] Fix bug for earlier change to allow
> > >>>> using IP as
> > >>>>>>>>> hostname
> > >>>>>>>>>> (#[5513](tel:5513)) [Xiaodong]
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Full Changelog since [1.10.3](tel:1103):
> > >>>>>>>>>>
> > >>>>>>>>>> Airflow [1.10.4](tel:1104), - [2019-07-18](tel:20190718)
> > >>>>>>>>>> ----------------------------
> > >>>>>>>>>>
> > >>>>>>>>>> New Features
> > >>>>>>>>>> """"""""""""
> > >>>>>>>>>> - [AIRFLOW-[4884](tel:4884)] Roll up import_errors in RBAC UI
> > >>>> (#[5516](tel:5516))
> > >>>>>>>>>> - [AIRFLOW-[4871](tel:4871)] Allow creating DagRuns via RBAC
> UI
> > >>>> (#[5507](tel:5507))
> > >>>>>>>>>> - [AIRFLOW-[4591](tel:4591)] Make default_pool a real pool
> > >>>> (#[5349](tel:5349))
> > >>>>>>>>>> - [AIRFLOW-[4844](tel:4844)] Add optional
> > is_paused_upon_creation
> > >>>> argument to DAG
> > >>>>>>>>>> (#[5473](tel:5473))
> > >>>>>>>>>> - [AIRFLOW-[4456](tel:4456)] Add sub-classable
> > BaseBranchOperator
> > >>>> (#[5231](tel:5231))
> > >>>>>>>>>> - [AIRFLOW-[4343](tel:4343)] Show warning in UI if scheduler
> is
> > >> not
> > >>>> running
> > >>>>>>>>> (#[5127](tel:5127))
> > >>>>>>>>>> - [AIRFLOW-[4739](tel:4739)] Add ability to arbitrarily define
> > >>>> kubernetes worker
> > >>>>>>>>> pod
> > >>>>>>>>>> labels (#[5376](tel:5376))
> > >>>>>>>>>> - [AIRFLOW-[4348](tel:4348)] Add GCP console link in
> > >>>> BigQueryOperator (#[5195](tel:5195))
> > >>>>>>>>>> - [AIRFLOW-[4306](tel:4306)] Global operator extra links
> > >>>> (#[5094](tel:5094))
> > >>>>>>>>>> - [AIRFLOW-[4812](tel:4812)] Add batch images annotation
> > >>>> (#[5433](tel:5433))
> > >>>>>>>>>> - [AIRFLOW-[4135](tel:4135)] Add Google Cloud Build operator
> and
> > >>>> hook (#[5251](tel:5251))
> > >>>>>>>>>> - [AIRFLOW-[4781](tel:4781)] Add the ability to specify ports
> in
> > >>>>>>>>> kubernetesOperator
> > >>>>>>>>>> (#[5410](tel:5410))
> > >>>>>>>>>> - [AIRFLOW-[4521](tel:4521)] Pause dag also pause its subdags
> > >>>> (#[5283](tel:5283))
> > >>>>>>>>>> - [AIRFLOW-[4738](tel:4738)] Enforce exampleinclude for
> example
> > >>>> DAGs (#[5375](tel:5375))
> > >>>>>>>>>> - [AIRFLOW-[4326](tel:4326)] Airflow AWS SQS Operator
> > >>>> (#[5110](tel:5110))
> > >>>>>>>>>> - [AIRFLOW-[3729](tel:3729)] Support "DownwardAPI" in env
> > >> variables
> > >>>> for
> > >>>>>>>>>> KubernetesPodOperator (#[4554](tel:4554))
> > >>>>>>>>>> - [AIRFLOW-[4585](tel:4585)] Implement Kubernetes Pod Mutation
> > >> Hook
> > >>>> (#[5359](tel:5359))
> > >>>>>>>>>> - [AIRFLOW-[161](tel:161)] New redirect route and extra links
> > >>>> (#[5059](tel:5059))
> > >>>>>>>>>> - [AIRFLOW-[4420](tel:4420)] Backfill respects
> task_concurrency
> > >>>> (#[5221](tel:5221))
> > >>>>>>>>>> - [AIRFLOW-[4147](tel:4147)] Add Operator to publish event to
> > >> Redis
> > >>>> (#[4967](tel:4967))
> > >>>>>>>>>> - [AIRFLOW-[3359](tel:3359)] Add option to pass customer
> > >> encryption
> > >>>> keys to
> > >>>>>>>>> Dataproc
> > >>>>>>>>>> (#[4200](tel:4200))
> > >>>>>>>>>> - [AIRFLOW-[4318](tel:4318)] Create Google Cloud Translate
> > Speech
> > >>>> Operator
> > >>>>>>>>> (#[5102](tel:5102))
> > >>>>>>>>>> - [AIRFLOW-[3960](tel:3960)] Adds Google Cloud Speech
> operators
> > >>>> (#[4780](tel:4780))
> > >>>>>>>>>> - [AIRFLOW-[1501](tel:1501)] Add
> > GoogleCloudStorageDeleteOperator
> > >>>> (#[5230](tel:5230))
> > >>>>>>>>>> - [AIRFLOW-[3672](tel:3672)] Add support for Mongo DB DNS
> > Seedlist
> > >>>> Connection
> > >>>>>>>>> Format
> > >>>>>>>>>> (#[4481](tel:4481))
> > >>>>>>>>>> - [AIRFLOW-[4397](tel:4397)] add integrations docs manually
> for
> > >> gcs
> > >>>> sensors
> > >>>>>>>>> (#[5204](tel:5204))
> > >>>>>>>>>> - [AIRFLOW-[4251](tel:4251)] Instrument DagRun schedule delay
> > >>>> (#[5050](tel:5050))
> > >>>>>>>>>> - [AIRFLOW-[4118](tel:4118)] instrument DagRun duration
> > >>>> (#[4946](tel:4946))
> > >>>>>>>>>> - [AIRFLOW-[4361](tel:4361)] Fix flaky
> > >>>>>>>>> test_integration_run_dag_with_scheduler_failure
> > >>>>>>>>>> (#[5182](tel:5182))
> > >>>>>>>>>> - [AIRFLOW-[4361](tel:4361)] Fix flaky
> > >>>>>>>>> test_integration_run_dag_with_scheduler_failure
> > >>>>>>>>>> (#[5140](tel:5140))
> > >>>>>>>>>> - [AIRFLOW-[4168](tel:4168)] Create Google Cloud Video
> > >> Intelligence
> > >>>> Operators
> > >>>>>>>>> (#[4985](tel:4985))
> > >>>>>>>>>> - [AIRFLOW-[4397](tel:4397)] Add
> GCSUploadSessionCompleteSensor
> > >>>> (#[5166](tel:5166))
> > >>>>>>>>>> - [AIRFLOW-[4335](tel:4335)] Add default num_retries to GCP
> > >>>> connection (#[5117](tel:5117))
> > >>>>>>>>>> - [AIRFLOW-[3808](tel:3808)] Add cluster_fields to
> > BigQueryHook's
> > >>>>>>>>> create_empty_table
> > >>>>>>>>>> (#[4654](tel:4654))
> > >>>>>>>>>> - [AIRFLOW-[4362](tel:4362)] Fix
> > >> test_execution_limited_parallelism
> > >>>> (#[5141](tel:5141))
> > >>>>>>>>>> - [AIRFLOW-[4307](tel:4307)] Backfill respects concurrency
> limit
> > >>>> (#[5128](tel:5128))
> > >>>>>>>>>> - [AIRFLOW-[4268](tel:4268)] Add
> > MsSqlToGoogleCloudStorageOperator
> > >>>> (#[5077](tel:5077))
> > >>>>>>>>>> - [AIRFLOW-[4169](tel:4169)] Add Google Cloud Vision Detect
> > >>>> Operators (#[4986](tel:4986))
> > >>>>>>>>>> - [AIRFLOW-XXX] Fix WS-[2019-0032](tel:20190032)
> > >> (#[5384](tel:5384))
> > >>>>>>>>>> - [AIRFLOW-XXX] Fix CVE-[2019-11358](tel:201911358)
> > >>>> (#[5197](tel:5197))
> > >>>>>>>>>> - [AIRFLOW-XXX] Change allowed version of Jinja2 to fix
> > >>>>>>>>> CVE-[2019-10906](tel:201910906)
> > >>>>>>>>>> (#[5075](tel:5075))
> > >>>>>>>>>>
> > >>>>>>>>>> Improvement
> > >>>>>>>>>> """""""""""
> > >>>>>>>>>> - [AIRFLOW-[4797](tel:4797)] Improve performance and behaviour
> > of
> > >>>> zombie
> > >>>>>>>>> detection
> > >>>>>>>>>> (#[5511](tel:5511))
> > >>>>>>>>>> - [AIRFLOW-[4911](tel:4911)] Silence the FORBIDDEN errors from
> > the
> > >>>>>>>>> KubernetesExecutor
> > >>>>>>>>>> (#[5547](tel:5547))
> > >>>>>>>>>> - [AIRFLOW-[3495](tel:3495)] Validate one of query and
> query_uri
> > >>>> passed to
> > >>>>>>>>>> DataProcSparkSqlOperator (#[5510](tel:5510))
> > >>>>>>>>>> - [AIRFLOW-[4925](tel:4925)] Improve css style for Variables
> > >> Import
> > >>>> file field
> > >>>>>>>>> (#[5552](tel:5552))
> > >>>>>>>>>> - [AIRFLOW-[4906](tel:4906)] Improve debugging for the
> > >>>> SparkSubmitHook (#[5542](tel:5542))
> > >>>>>>>>>> - [AIRFLOW-[4904](tel:4904)] unittest.cfg name and path can be
> > >>>> overriden by
> > >>>>>>>>> setting
> > >>>>>>>>>> $AIRFLOW_TEST_CONFIG (#[5540](tel:5540))
> > >>>>>>>>>> - [AIRFLOW-[4920](tel:4920)] Use html.escape instead of
> > cgi.escape
> > >>>> to fix
> > >>>>>>>>>> DeprecationWarning (#[5551](tel:5551))
> > >>>>>>>>>> - [AIRFLOW-[4919](tel:4919)] DataProcJobBaseOperator
> > >>>> dataproc_*_properties
> > >>>>>>>>> templated
> > >>>>>>>>>> (#[5555](tel:5555))
> > >>>>>>>>>> - [AIRFLOW-[4478](tel:4478)] Lazily instantiate default
> > resources
> > >>>> objects.
> > >>>>>>>>> (#[5259](tel:5259))
> > >>>>>>>>>> - [AIRFLOW-[4564](tel:4564)] AzureContainerInstance bugfixes
> and
> > >>>> improvements
> > >>>>>>>>> (#[5319](tel:5319))
> > >>>>>>>>>> - [AIRFLOW-[4237](tel:4237)] Including Try Number of Task in
> > Gantt
> > >>>> Chart (#[5037](tel:5037))
> > >>>>>>>>>> - [AIRFLOW-[4862](tel:4862)] Allow directly using IP address
> as
> > >>>> hostname for
> > >>>>>>>>> webserver
> > >>>>>>>>>> logs (#[5501](tel:5501))
> > >>>>>>>>>> - [AIRFLOW-[4857](tel:4857)] Add templated fields to
> > >>>> SlackWebhookOperator (#[5490](tel:5490))
> > >>>>>>>>>> - [AIRFLOW-[3502](tel:3502)] Add celery config option for
> > setting
> > >>>> "pool" (#[4308](tel:4308))
> > >>>>>>>>>> - [AIRFLOW-[3217](tel:3217)] Button to toggle line wrapping in
> > log
> > >>>> and code views
> > >>>>>>>>>> (#[4277](tel:4277))
> > >>>>>>>>>> - [AIRFLOW-[4491](tel:4491)] Add a "Jump to end" button for
> logs
> > >>>> (#[5266](tel:5266))
> > >>>>>>>>>> - [AIRFLOW-[4422](tel:4422)] Pool utilization stats
> > >>>> (#[5453](tel:5453))
> > >>>>>>>>>> - [AIRFLOW-[4805](tel:4805)] Add py_file as templated field in
> > >>>>>>>>> DataflowPythonOperator
> > >>>>>>>>>> (#[5451](tel:5451))
> > >>>>>>>>>> - [AIRFLOW-[4838](tel:4838)] Surface Athena errors in
> > >>>> AWSAthenaOperator (#[5467](tel:5467))
> > >>>>>>>>>> - [AIRFLOW-[4831](tel:4831)] conf.has_option no longer throws
> if
> > >>>> section is
> > >>>>>>>>> missing.
> > >>>>>>>>>> (#[5455](tel:5455))
> > >>>>>>>>>> - [AIRFLOW-[4829](tel:4829)] More descriptive exceptions for
> EMR
> > >>>> sensors (#[5452](tel:5452))
> > >>>>>>>>>> - [AIRFLOW-[4414](tel:4414)] AWSAthenaOperator: Push
> > >>>> QueryExecutionID to XCom
> > >>>>>>>>> (#[5276](tel:5276))
> > >>>>>>>>>> - [AIRFLOW-[4791](tel:4791)] add "schema" keyword arg to
> > >>>> SnowflakeOperator
> > >>>>>>>>> (#[5415](tel:5415))
> > >>>>>>>>>> - [AIRFLOW-[4759](tel:4759)] Don't error when marking
> sucessful
> > >> run
> > >>>> as failed
> > >>>>>>>>> (#[5435](tel:5435))
> > >>>>>>>>>> - [AIRFLOW-[4716](tel:4716)] Instrument dag loading time
> > duration
> > >>>> (#[5350](tel:5350))
> > >>>>>>>>>> - [AIRFLOW-[3958](tel:3958)] Support list tasks as upstream in
> > >>>> chain (#[4779](tel:4779))
> > >>>>>>>>>> - [AIRFLOW-[4409](tel:4409)] Prevent task duration break by
> null
> > >>>> value (#[5178](tel:5178))
> > >>>>>>>>>> - [AIRFLOW-[4418](tel:4418)] Add "failed only" option to task
> > >> modal
> > >>>> (#[5193](tel:5193))
> > >>>>>>>>>> - [AIRFLOW-[4740](tel:4740)] Accept string ``end_date`` in DAG
> > >>>> default_args
> > >>>>>>>>> (#[5381](tel:5381))
> > >>>>>>>>>> - [AIRFLOW-[4423](tel:4423)] Improve date handling in mysql to
> > gcs
> > >>>> operator.
> > >>>>>>>>> (#[5196](tel:5196))
> > >>>>>>>>>> - [AIRFLOW-[4447](tel:4447)] Display task duration as human
> > >>>> friendly format in UI
> > >>>>>>>>>> (#[5218](tel:5218))
> > >>>>>>>>>> - [AIRFLOW-[4377](tel:4377)] Remove needless object conversion
> > in
> > >>>> DAG.owner()
> > >>>>>>>>> (#[5144](tel:5144))
> > >>>>>>>>>> - [AIRFLOW-[4766](tel:4766)] Add autoscaling option for
> > >>>>>>>>> DataprocClusterCreateOperator
> > >>>>>>>>>> (#[5425](tel:5425))
> > >>>>>>>>>> - [AIRFLOW-[4795](tel:4795)] Upgrade alembic to latest
> release.
> > >>>> (#[5411](tel:5411))
> > >>>>>>>>>> - [AIRFLOW-[4793](tel:4793)] Add signature_name to mlengine
> > >>>> operator (#[5417](tel:5417))
> > >>>>>>>>>> - [AIRFLOW-[3211](tel:3211)] Reattach to GCP Dataproc jobs
> upon
> > >>>> Airflow restart
> > >>>>>>>>>> (#[4083](tel:4083))
> > >>>>>>>>>> - [AIRFLOW-[4750](tel:4750)] Log identified zombie task
> > instances
> > >>>> (#[5389](tel:5389))
> > >>>>>>>>>> - [AIRFLOW-[3870](tel:3870)] STFPOperator: Update log level
> and
> > >>>> return value
> > >>>>>>>>> (#[4355](tel:4355))
> > >>>>>>>>>> - [AIRFLOW-[4759](tel:4759)] Batch queries in set_state API.
> > >>>> (#[5403](tel:5403))
> > >>>>>>>>>> - [AIRFLOW-[2737](tel:2737)] Restore original license header
> to
> > >>>>>>>>>> airflow.api.auth.backend.kerberos_auth
> > >>>>>>>>>> - [AIRFLOW-[3635](tel:3635)] Fix incorrect logic in detele_dag
> > >>>> (introduced in
> > >>>>>>>>> PR#[4406](tel:4406))
> > >>>>>>>>>> (#[4445](tel:4445))
> > >>>>>>>>>> - [AIRFLOW-[3599](tel:3599)] Removed Dagbag from delete dag
> > >>>> (#[4406](tel:4406))
> > >>>>>>>>>> - [AIRFLOW-[4737](tel:4737)] Increase and document celery
> queue
> > >>>> name limit
> > >>>>>>>>> (#[5383](tel:5383))
> > >>>>>>>>>> - [AIRFLOW-[4505](tel:4505)] Correct Tag ALL for PY3
> > >>>> (#[5275](tel:5275))
> > >>>>>>>>>> - [AIRFLOW-[4743](tel:4743)] Add environment variables support
> > to
> > >>>> SSHOperator
> > >>>>>>>>> (#[5385](tel:5385))
> > >>>>>>>>>> - [AIRFLOW-[4725](tel:4725)] Fix setup.py PEP[440](tel:440) &
> > >>>> Sphinx-PyPI-upload dependency
> > >>>>>>>>>> (#[5363](tel:5363))
> > >>>>>>>>>> - [AIRFLOW-[3370](tel:3370)] Add stdout output options to
> > >>>> Elasticsearch task log
> > >>>>>>>>>> handler (#[5048](tel:5048))
> > >>>>>>>>>> - [AIRFLOW-[4396](tel:4396)] Provide a link to external
> > >>>> Elasticsearch logs in UI.
> > >>>>>>>>>> (#[5164](tel:5164))
> > >>>>>>>>>> - [AIRFLOW-[1381](tel:1381)] Allow setting host temporary
> > >> directory
> > >>>> in
> > >>>>>>>>> DockerOperator
> > >>>>>>>>>> (#[5369](tel:5369))
> > >>>>>>>>>> - [AIRFLOW-[4598](tel:4598)] Task retries are not exhausted
> for
> > >> K8s
> > >>>> executor
> > >>>>>>>>> (#[5347](tel:5347))
> > >>>>>>>>>> - [AIRFLOW-[4218](tel:4218)] Support to Provide http args to
> > >>>> K8executor while
> > >>>>>>>>> calling
> > >>>>>>>>>> k8 python client lib apis (#[5060](tel:5060))
> > >>>>>>>>>> - [AIRFLOW-[4159](tel:4159)] Add support for additional static
> > pod
> > >>>> labels for
> > >>>>>>>>>> K8sExecutor (#[5134](tel:5134))
> > >>>>>>>>>> - [AIRFLOW-[4720](tel:4720)] Allow comments in .airflowignore
> > >>>> files. (#[5355](tel:5355))
> > >>>>>>>>>> - [AIRFLOW-[4486](tel:4486)] Add AWS IAM authenication in
> > >> MySqlHook
> > >>>> (#[5334](tel:5334))
> > >>>>>>>>>> - [AIRFLOW-[4417](tel:4417)] Add AWS IAM authenication for
> > >>>> PostgresHook (#[5223](tel:5223))
> > >>>>>>>>>> - [AIRFLOW-[3990](tel:3990)] Compile regular expressions.
> > >>>> (#[4813](tel:4813))
> > >>>>>>>>>> - [AIRFLOW-[4572](tel:4572)] Rename prepare_classpath() to
> > >>>> prepare_syspath()
> > >>>>>>>>> (#[5328](tel:5328))
> > >>>>>>>>>> - [AIRFLOW-[3869](tel:3869)] Raise consistent exception in
> > >>>>>>>>>> AirflowConfigParser.getboolean (#[4692](tel:4692))
> > >>>>>>>>>> - [AIRFLOW-[4571](tel:4571)] Add headers to templated field
> for
> > >>>>>>>>> SimpleHttpOperator
> > >>>>>>>>>> (#[5326](tel:5326))
> > >>>>>>>>>> - [AIRFLOW-[3867](tel:3867)] Rename GCP's subpackage
> > >>>> (#[4690](tel:4690))
> > >>>>>>>>>> - [AIRFLOW-[3725](tel:3725)] Add private_key to bigquery_hook
> > >>>> get_pandas_df
> > >>>>>>>>> (#[4549](tel:4549))
> > >>>>>>>>>> - [AIRFLOW-[4546](tel:4546)] Upgrade google-cloud-bigtable.
> > >>>> (#[5307](tel:5307))
> > >>>>>>>>>> - [AIRFLOW-[4519](tel:4519)] Optimise operator classname
> sorting
> > >> in
> > >>>> views (#[5282](tel:5282))
> > >>>>>>>>>> - [AIRFLOW-[4503](tel:4503)] Support fully pig options
> > >>>> (#[5271](tel:5271))
> > >>>>>>>>>> - [AIRFLOW-[4468](tel:4468)] add sql_alchemy_max_overflow
> > >> parameter
> > >>>> (#[5249](tel:5249))
> > >>>>>>>>>> - [AIRFLOW-[4467](tel:4467)] Add dataproc_jars to templated
> > fields
> > >>>> in Dataproc
> > >>>>>>>>> oper…
> > >>>>>>>>>> (#[5248](tel:5248))
> > >>>>>>>>>> - [AIRFLOW-[4381](tel:4381)] Use get_direct_relative_ids get
> > task
> > >>>> relatives
> > >>>>>>>>> (#[5147](tel:5147))
> > >>>>>>>>>> - [AIRFLOW-[3624](tel:3624)] Add masterType parameter to
> > >>>> MLEngineTrainingOperator
> > >>>>>>>>>> (#[4428](tel:4428))
> > >>>>>>>>>> - [AIRFLOW-[3143](tel:3143)] Support Auto-Zone in
> > >>>> DataprocClusterCreateOperator
> > >>>>>>>>> (#[5169](tel:5169))
> > >>>>>>>>>> - [AIRFLOW-[3874](tel:3874)] Improve
> > >>>> BigQueryHook.run_with_configuration's
> > >>>>>>>>> location
> > >>>>>>>>>> support (#[4695](tel:4695))
> > >>>>>>>>>> - [AIRFLOW-[4399](tel:4399)] Avoid duplicated os.path.isfile()
> > >>>> check in
> > >>>>>>>>> models.dagbag
> > >>>>>>>>>> (#[5165](tel:5165))
> > >>>>>>>>>> - [AIRFLOW-[4031](tel:4031)] Allow for key pair auth in
> > snowflake
> > >>>> hook (#[4875](tel:4875))
> > >>>>>>>>>> - [AIRFLOW-[3901](tel:3901)] add role as optional config
> > parameter
> > >>>> for
> > >>>>>>>>> SnowflakeHook
> > >>>>>>>>>> (#[4721](tel:4721))
> > >>>>>>>>>> - [AIRFLOW-[3455](tel:3455)] add region in snowflake connector
> > >>>> (#[4285](tel:4285))
> > >>>>>>>>>> - [AIRFLOW-[4073](tel:4073)] add template_ext for AWS Athena
> > >>>> operator (#[4907](tel:4907))
> > >>>>>>>>>> - [AIRFLOW-[4093](tel:4093)] AWSAthenaOperator: Throw
> exception
> > if
> > >>>> job
> > >>>>>>>>>> failed/cancelled/reach max retries (#[4919](tel:4919))
> > >>>>>>>>>> - [AIRFLOW-[4356](tel:4356)] Add extra RuntimeEnvironment keys
> > to
> > >>>> DataFlowHook
> > >>>>>>>>> (#[5149](tel:5149))
> > >>>>>>>>>> - [AIRFLOW-[4337](tel:4337)] Fix docker-compose deprecation
> > >> warning
> > >>>> in CI (#[5119](tel:5119))
> > >>>>>>>>>> - [AIRFLOW-[3603](tel:3603)] QuboleOperator: Remove SQLCommand
> > >> from
> > >>>> SparkCmd
> > >>>>>>>>>> documentation (#[4411](tel:4411))
> > >>>>>>>>>> - [AIRFLOW-[4328](tel:4328)] Fix link to task instances from
> > Pool
> > >>>> page (#[5124](tel:5124))
> > >>>>>>>>>> - [AIRFLOW-[4255](tel:4255)] Make GCS Hook Backwards
> compatible
> > >>>> (#[5089](tel:5089))
> > >>>>>>>>>> - [AIRFLOW-[4103](tel:4103)] Allow uppercase letters in
> dataflow
> > >>>> job names
> > >>>>>>>>> (#[4925](tel:4925))
> > >>>>>>>>>> - [AIRFLOW-[4255](tel:4255)] Replace Discovery based api with
> > >>>> client based for
> > >>>>>>>>> GCS
> > >>>>>>>>>> (#[5054](tel:5054))
> > >>>>>>>>>> - [AIRFLOW-[4311](tel:4311)] Remove sleep in localexecutor
> > >>>> (#[5096](tel:5096))
> > >>>>>>>>>> - [AIRFLOW-[2836](tel:2836)] Minor
> > >>>> improvement-contrib.sensors.FileSensor (#[3674](tel:3674))
> > >>>>>>>>>> - [AIRFLOW-[4104](tel:4104)] Add type annotations to common
> > >>>> classes. (#[4926](tel:4926))
> > >>>>>>>>>> - [AIRFLOW-[3910](tel:3910)] Raise exception explicitly in
> > >>>> Connection.get_hook()
> > >>>>>>>>>> (#[4728](tel:4728))
> > >>>>>>>>>> - [AIRFLOW-[3322](tel:3322)] Update QuboleHook to fetch args
> > >>>> dynamically from
> > >>>>>>>>> qds_sdk
> > >>>>>>>>>> (#[4165](tel:4165))
> > >>>>>>>>>> - [AIRFLOW-[4565](tel:4565)] instrument celery executor
> > >>>> (#[5321](tel:5321))
> > >>>>>>>>>> - [AIRFLOW-[4573](tel:4573)] Import airflow_local_settings
> after
> > >>>>>>>>> prepare_classpath
> > >>>>>>>>>> (#[5330](tel:5330))
> > >>>>>>>>>> - [AIRFLOW-[4448](tel:4448)] Don't bake ENV and _cmd into tmp
> > >>>> config for non-sudo
> > >>>>>>>>>> (#[4050](tel:4050))
> > >>>>>>>>>> - [AIRFLOW-[4295](tel:4295)] Make ``method`` attribute case
> > >>>> insensitive in
> > >>>>>>>>> HttpHook
> > >>>>>>>>>> (#[5313](tel:5313))
> > >>>>>>>>>> - [AIRFLOW-[3703](tel:3703)] Add dnsPolicy option for
> > >>>> KubernetesPodOperator
> > >>>>>>>>> (#[4520](tel:4520))
> > >>>>>>>>>> - [AIRFLOW-[3057](tel:3057)] add prev_*_date_success to
> template
> > >>>> context (#[5372](tel:5372))
> > >>>>>>>>>> - [AIRFLOW-[4336](tel:4336)] Stop showing entire GCS files
> bytes
> > >> in
> > >>>> log for
> > >>>>>>>>>> gcs_download_operator (#[5151](tel:5151))
> > >>>>>>>>>> - [AIRFLOW-[4528](tel:4528)] Cancel DataProc task on timeout
> > >>>> (#[5293](tel:5293))
> > >>>>>>>>>>
> > >>>>>>>>>> Bug fixes
> > >>>>>>>>>> """""""""
> > >>>>>>>>>> - [AIRFLOW-[4934](tel:4934)] Fix ProxyFix due to Werkzeug
> > upgrade
> > >>>> (#[5563](tel:5563)) (#[5571](tel:5571))
> > >>>>>>>>>> - [AIRFLOW-[4136](tel:4136)] fix key_file of hook is
> overwritten
> > >> by
> > >>>> SSHHook
> > >>>>>>>>> connection
> > >>>>>>>>>> (#[5558](tel:5558))
> > >>>>>>>>>> - [AIRFLOW-[4587](tel:4587)] Replace self.conn with
> > >> self.get_conn()
> > >>>> in
> > >>>>>>>>> AWSAthenaHook
> > >>>>>>>>>> (#[5545](tel:5545))
> > >>>>>>>>>> - [AIRFLOW-[1740](tel:1740)] Fix xcom creation and update via
> UI
> > >>>> (#[5530](tel:5530)) (#[5531](tel:5531))
> > >>>>>>>>>> - [AIRFLOW-[4900](tel:4900)] Resolve incompatible version of
> > >>>> Werkzeug (#[5535](tel:5535))
> > >>>>>>>>>> - [AIRFLOW-[4510](tel:4510)] Don't mutate default_args during
> > DAG
> > >>>> initialization
> > >>>>>>>>>> (#[5277](tel:5277))
> > >>>>>>>>>> - [AIRFLOW-[3360](tel:3360)] Make the DAGs search respect
> other
> > >>>> querystring
> > >>>>>>>>> parameters
> > >>>>>>>>>> with url-search-params-polyfill for IE support
> > (#[5503](tel:5503))
> > >>>>>>>>>> - [AIRFLOW-[4896](tel:4896)] Make KubernetesExecutorConfig's
> > >>>> default args
> > >>>>>>>>> immutable
> > >>>>>>>>>> (#[5534](tel:5534))
> > >>>>>>>>>> - [AIRFLOW-[4494](tel:4494)] Remove ``shell=True`` in
> > DaskExecutor
> > >>>> (#[5273](tel:5273))
> > >>>>>>>>>> - [AIRFLOW-[4890](tel:4890)] Fix Log link in TaskInstance's
> View
> > >>>> for Non-RBAC
> > >>>>>>>>> (#[5525](tel:5525))
> > >>>>>>>>>> - [AIRFLOW-[4892](tel:4892)] Fix connection creation via UIs
> > >>>> (#[5527](tel:5527))
> > >>>>>>>>>> - [AIRFLOW-[4406](tel:4406)] Fix a method name typo:
> > >>>> NullFernet.decrpyt to
> > >>>>>>>>> decrypt
> > >>>>>>>>>> (#[5509](tel:5509))
> > >>>>>>>>>> - [AIRFLOW-[4849](tel:4849)] Add gcp_conn_id to
> > cloudsqldatabehook
> > >>>> class to use
> > >>>>>>>>>> correctly CloudSqlProxyRunner class (#[5478](tel:5478))
> > >>>>>>>>>> - [AIRFLOW-[4769](tel:4769)] Pass gcp_conn_id to BigtableHook
> > >>>> (#[5445](tel:5445))
> > >>>>>>>>>> - [AIRFLOW-[4524](tel:4524)] Fix incorrect field names in view
> > for
> > >>>> Mark
> > >>>>>>>>>> Success/Failure (#[5486](tel:5486))
> > >>>>>>>>>> - [AIRFLOW-[3671](tel:3671)] Remove arg ``replace`` of
> > >>>> MongoToS3Operator from
> > >>>>>>>>>> ``kwargs`` (#[4480](tel:4480))
> > >>>>>>>>>> - [AIRFLOW-[4845](tel:4845)] Fix bug where runAsUser 0 doesn't
> > get
> > >>>> set in k8s
> > >>>>>>>>> security
> > >>>>>>>>>> context (#[5474](tel:5474))
> > >>>>>>>>>> - [AIRFLOW-[4354](tel:4354)] Fix exception in "between" date
> > >> filter
> > >>>> in classic UI
> > >>>>>>>>>> (#[5480](tel:5480))
> > >>>>>>>>>> - [AIRFLOW-[4587](tel:4587)] Replace self.conn with
> > >> self.get_conn()
> > >>>> in
> > >>>>>>>>> AWSAthenaHook
> > >>>>>>>>>> (#[5462](tel:5462))
> > >>>>>>>>>> - [AIRFLOW-[4516](tel:4516)] K8s runAsUser and fsGroup cannot
> be
> > >>>> strings (#[5429](tel:5429))
> > >>>>>>>>>> - [AIRFLOW-[4298](tel:4298)] Stop Scheduler repeatedly warning
> > >>>> "connection
> > >>>>>>>>>> invalidated" (#[5470](tel:5470))
> > >>>>>>>>>> - [AIRFLOW-[4559](tel:4559)] JenkinsJobTriggerOperator bugfix
> > >>>> (#[5318](tel:5318))
> > >>>>>>>>>> - [AIRFLOW-[4841](tel:4841)] Pin Sphinx AutoApi to
> > >> [1.0.0](tel:100)
> > >>>> (#[5468](tel:5468))
> > >>>>>>>>>> - [AIRFLOW-[4479](tel:4479)] Include s3_overwrite kwarg in
> > >>>> load_bytes method
> > >>>>>>>>> (#[5312](tel:5312))
> > >>>>>>>>>> - [AIRFLOW-[3746](tel:3746)] Fix DockerOperator missing
> > container
> > >>>> exit (#[4583](tel:4583))
> > >>>>>>>>>> - [AIRFLOW-[4233](tel:4233)] Remove Template Extension from Bq
> > to
> > >>>> GCS Operator
> > >>>>>>>>> (#[5456](tel:5456))
> > >>>>>>>>>> -
> > >>>>
> > >>
> >
> [AIRFLOW-[2141](tel:2141)][AIRFLOW-[3157](tel:3157)][AIRFLOW-[4170](tel:4170)]
> > >>>> Serialize non-str value
> > >>>>>>>>> by
> > >>>>>>>>>> JSON when importing Variables (#[4991](tel:4991))
> > >>>>>>>>>> - [AIRFLOW-[4826](tel:4826)] Remove warning from ``airflow
> > >>>> resetdb`` command
> > >>>>>>>>> (#[5447](tel:5447))
> > >>>>>>>>>> - [AIRFLOW-[4148](tel:4148)] Fix editing DagRuns when clicking
> > >>>> state column
> > >>>>>>>>> (#[5436](tel:5436))
> > >>>>>>>>>> - [AIRFLOW-[4455](tel:4455)] dag_details broken for subdags in
> > >> RBAC
> > >>>> UI (#[5234](tel:5234))
> > >>>>>>>>>> - [AIRFLOW-[2955](tel:2955)] Fix kubernetes pod operator to
> set
> > >>>> requests and
> > >>>>>>>>> limits on
> > >>>>>>>>>> task pods (#[4551](tel:4551))
> > >>>>>>>>>> - [AIRFLOW-[4459](tel:4459)] Fix wrong DAG count in /home page
> > >> when
> > >>>> DAG count is
> > >>>>>>>>> zero
> > >>>>>>>>>> (#[5235](tel:5235))
> > >>>>>>>>>> - [AIRFLOW-[3876](tel:3876)] AttributeError: module
> 'distutils'
> > >> has
> > >>>> no attribute
> > >>>>>>>>> 'util'
> > >>>>>>>>>> - [AIRFLOW-[4146](tel:4146)] Fix CgroupTaskRunner errors
> > >>>> (#[5224](tel:5224))
> > >>>>>>>>>> - [AIRFLOW-[4524](tel:4524)] Fix bug with "Ignore \*" toggles
> in
> > >>>> RBAC mode
> > >>>>>>>>> (#[5378](tel:5378))
> > >>>>>>>>>> - [AIRFLOW-[4765](tel:4765)] Fix DataProcPigOperator execute
> > >> method
> > >>>> (#[5426](tel:5426))
> > >>>>>>>>>> - [AIRFLOW-[4798](tel:4798)] obviate interdependencies for
> > dagbag
> > >>>> and TI tests
> > >>>>>>>>> (#[5422](tel:5422))
> > >>>>>>>>>> - [AIRFLOW-[4800](tel:4800)] fix GKEClusterHook ctor calls
> > >>>> (#[5424](tel:5424))
> > >>>>>>>>>> - [AIRFLOW-[4799](tel:4799)] don't mutate self.env in
> > BashOperator
> > >>>> execute method
> > >>>>>>>>>> (#[5421](tel:5421))
> > >>>>>>>>>> - [AIRFLOW-[4393](tel:4393)] Add retry logic when fetching pod
> > >>>> status and/or logs
> > >>>>>>>>> in
> > >>>>>>>>>> KubernetesPodOperator (#[5284](tel:5284))
> > >>>>>>>>>> - [AIRFLOW-[4174](tel:4174)] Fix HttpHook run with backoff
> > >>>> (#[5213](tel:5213))
> > >>>>>>>>>> - [AIRFLOW-[4463](tel:4463)] Handle divide-by-zero errors in
> > short
> > >>>> retry
> > >>>>>>>>> intervals
> > >>>>>>>>>> (#[5243](tel:5243))
> > >>>>>>>>>> - [AIRFLOW-[2614](tel:2614)] Speed up trigger_dag API call
> when
> > >>>> lots of DAGs in
> > >>>>>>>>> system
> > >>>>>>>>>> - [AIRFLOW-[4756](tel:4756)] add ti.state to ti.start_date as
> > >>>> criteria for gantt
> > >>>>>>>>>> (#[5399](tel:5399))
> > >>>>>>>>>> - [AIRFLOW-[4760](tel:4760)] Fix zip-packaged DAGs
> disappearing
> > >>>> from DagBag when
> > >>>>>>>>>> reloaded (#[5404](tel:5404))
> > >>>>>>>>>> - [AIRFLOW-[4731](tel:4731)] Fix GCS hook with
> > >>>> google-storage-client [1.16](tel:116) (#[5368](tel:5368))
> > >>>>>>>>>> - [AIRFLOW-[3506](tel:3506)] use match_phrase to query log_id
> in
> > >>>> elasticsearch
> > >>>>>>>>> (#[4342](tel:4342))
> > >>>>>>>>>> - [AIRFLOW-[4084](tel:4084)] fix ElasticSearch log download
> > >>>> (#[5177](tel:5177))
> > >>>>>>>>>> - [AIRFLOW-[4501](tel:4501)] Register pendulum datetime
> > converter
> > >>>> for
> > >>>>>>>>> sqla+pymysql
> > >>>>>>>>>> (#[5190](tel:5190))
> > >>>>>>>>>> - [AIRFLOW-[986](tel:986)] HiveCliHook ignores 'proxy_user'
> > value
> > >>>> in a
> > >>>>>>>>> connection's
> > >>>>>>>>>> extra parameter (#[5305](tel:5305))
> > >>>>>>>>>> - [AIRFLOW-[4442](tel:4442)] fix hive_tblproperties in
> > >>>> HiveToDruidTransfer
> > >>>>>>>>> (#[5211](tel:5211))
> > >>>>>>>>>> - [AIRFLOW-[4557](tel:4557)] Add gcp_conn_id parameter to
> > >>>> get_sqlproxy_runner()
> > >>>>>>>>> of
> > >>>>>>>>>> CloudSqlDatabaseHook (#[5314](tel:5314))
> > >>>>>>>>>> - [AIRFLOW-[4545](tel:4545)] Upgrade FAB to latest version
> > >>>> (#[4955](tel:4955))
> > >>>>>>>>>> - [AIRFLOW-[4492](tel:4492)] Change Dataproc Cluster operators
> > to
> > >>>> poll Operations
> > >>>>>>>>>> (#[5269](tel:5269))
> > >>>>>>>>>> - [AIRFLOW-[4452](tel:4452)] Webserver and Scheduler keep
> > crashing
> > >>>> because of
> > >>>>>>>>>> slackclient update (#[5225](tel:5225))
> > >>>>>>>>>> - [AIRFLOW-[4450](tel:4450)] Fix request arguments in
> > >>>> has_dag_access (#[5220](tel:5220))
> > >>>>>>>>>> - [AIRFLOW-[4434](tel:4434)] Support Impala with the
> > >>>> HiveServer2Hook (#[5206](tel:5206))
> > >>>>>>>>>> - [AIRFLOW-[3449](tel:3449)] Write local dag parsing logs when
> > >>>> remote logging
> > >>>>>>>>> enabled.
> > >>>>>>>>>> (#[5175](tel:5175))
> > >>>>>>>>>> - [AIRFLOW-[4300](tel:4300)] Fix graph modal call when DAG has
> > not
> > >>>> yet run
> > >>>>>>>>> (#[5185](tel:5185))
> > >>>>>>>>>> - [AIRFLOW-[4401](tel:4401)] Use managers for Queue
> > >> synchronization
> > >>>> (#[5200](tel:5200))
> > >>>>>>>>>> - [AIRFLOW-[3626](tel:3626)] Fixed triggering DAGs contained
> > >> within
> > >>>> zip files
> > >>>>>>>>> (#[4439](tel:4439))
> > >>>>>>>>>> - [AIRFLOW-[3720](tel:3720)] Fix missmatch while comparing GCS
> > and
> > >>>> S3 files
> > >>>>>>>>> (#[4766](tel:4766))
> > >>>>>>>>>> - [AIRFLOW-[4403](tel:4403)] search by ``dag_id`` or
> ``owners``
> > in
> > >>>> UI (#[5184](tel:5184))
> > >>>>>>>>>> - [AIRFLOW-[4308](tel:4308)] Fix TZ-loop around DST on python
> > 3.6+
> > >>>> (#[5095](tel:5095))
> > >>>>>>>>>> - [AIRFLOW-[4324](tel:4324)] fix DAG fuzzy search in RBAC UI
> > >>>> (#[5131](tel:5131))
> > >>>>>>>>>> - [AIRFLOW-[4297](tel:4297)] Temporary hot fix on
> manage_slas()
> > >> for
> > >>>> [1.10.4](tel:1104)
> > >>>>>>>>> release
> > >>>>>>>>>> (#[5150](tel:5150))
> > >>>>>>>>>> - [AIRFLOW-[4299](tel:4299)] Upgrade to Celery
> [4.3.0](tel:430)
> > to
> > >>>> fix crashing workers
> > >>>>>>>>> (#[5116](tel:5116))
> > >>>>>>>>>> - [AIRFLOW-[4291](tel:4291)] Correctly render doc_md in DAG
> > graph
> > >>>> page (#[5121](tel:5121))
> > >>>>>>>>>> - [AIRFLOW-[4310](tel:4310)] Fix incorrect link on Dag Details
> > >> page
> > >>>> (#[5122](tel:5122))
> > >>>>>>>>>> - [AIRFLOW-[4331](tel:4331)] Correct filter for Null-state
> runs
> > >>>> from Dag Detail
> > >>>>>>>>> page
> > >>>>>>>>>> (#[5123](tel:5123))
> > >>>>>>>>>> - [AIRFLOW-[4294](tel:4294)] Fix missing dag & task runs in UI
> > >>>> dag_id contains a
> > >>>>>>>>> dot
> > >>>>>>>>>> (#[5111](tel:5111))
> > >>>>>>>>>> - [AIRFLOW-[4332](tel:4332)] Upgrade sqlalchemy to remove
> > security
> > >>>> Vulnerability
> > >>>>>>>>>> (#[5113](tel:5113))
> > >>>>>>>>>> - [AIRFLOW-[4312](tel:4312)] Add template_fields &
> template_ext
> > to
> > >>>>>>>>> BigQueryCheckO…
> > >>>>>>>>>> (#[5097](tel:5097))
> > >>>>>>>>>> - [AIRFLOW-[4293](tel:4293)] Fix downgrade in
> > >>>>>>>>>> d4ecb8fbee3_add_schedule_interval_to_dag.py
> (#[5086](tel:5086))
> > >>>>>>>>>> - [AIRFLOW-[4267](tel:4267)] Fix TI duration in Graph View
> > >>>> (#[5071](tel:5071))
> > >>>>>>>>>> - [AIRFLOW-[4163](tel:4163)] IntervalCheckOperator supports
> > >>>> relative diff and not
> > >>>>>>>>>> ignore 0 (#[4983](tel:4983))
> > >>>>>>>>>> - [AIRFLOW-[3938](tel:3938)] QuboleOperator Fixes and Support
> > for
> > >>>> SqlCommand
> > >>>>>>>>> (#[4832](tel:4832))
> > >>>>>>>>>> - [AIRFLOW-[2903](tel:2903)] Change default owner to
> ``airflow``
> > >>>> (#[4151](tel:4151))
> > >>>>>>>>>> - [aIRFLOW-[4136](tel:4136)] Fix overwrite of key_file by
> > >>>> constructor (#[5155](tel:5155))
> > >>>>>>>>>> - [AIRFLOW-[3241](tel:3241)] Remove Invalid template ext in
> GCS
> > >>>> Sensors (#[4076](tel:4076))
> > >>>>>>>>>>
> > >>>>>>>>>> Misc/Internal
> > >>>>>>>>>> """""""""""""
> > >>>>>>>>>> - [AIRFLOW-[4937](tel:4937)] Fix lodash security issue with
> > >> version
> > >>>> below [4.17.13](tel:41713)
> > >>>>>>>>>> (#[5572](tel:5572)) (used only in build-pipeline, not runtime)
> > >>>>>>>>>> - [AIRFLOW-[4868](tel:4868)] Fix typo in
> > >> kubernetes/docker/build.sh
> > >>>> (#[5505](tel:5505))
> > >>>>>>>>>> - [AIRFLOW-[4211](tel:4211)] Add tests for WebHDFSHook
> > >>>> (#[5015](tel:5015))
> > >>>>>>>>>> - [AIRFLOW-[4320](tel:4320)] Add tests for
> > >>>> SegmentTrackEventOperator (#[5104](tel:5104))
> > >>>>>>>>>> - [AIRFLOW-[4319](tel:4319)] Add tests for Bigquery related
> > >>>> Operators (#[5101](tel:5101))
> > >>>>>>>>>> - [AIRFLOW-[4014](tel:4014)] Change DatastoreHook and add
> tests
> > >>>> (#[4842](tel:4842))
> > >>>>>>>>>> - [AIRFLOW-[4322](tel:4322)] Add test for VerticaOperator
> > >>>> (#[5107](tel:5107))
> > >>>>>>>>>> - [AIRFLOW-[4323](tel:4323)] Add 2 tests for WinRMOperator
> > >>>> (#[5108](tel:5108))
> > >>>>>>>>>> - [AIRFLOW-[3677](tel:3677)] Improve CheckOperator test
> coverage
> > >>>> (#[4756](tel:4756))
> > >>>>>>>>>> - [AIRFLOW-[4659](tel:4659)] Fix pylint problems for api
> module
> > >>>> (#[5398](tel:5398))
> > >>>>>>>>>> - [AIRFLOW-[4358](tel:4358)] Speed up test_jobs by not running
> > >>>> tasks (#[5162](tel:5162))
> > >>>>>>>>>> - [AIRFLOW-[4394](tel:4394)] Don't test behaviour of
> BackfillJob
> > >>>> from CLI tests
> > >>>>>>>>> (#[5160](tel:5160))
> > >>>>>>>>>> - [AIRFLOW-[3471](tel:3471)] Move XCom out of models.py
> > >>>> (#[4629](tel:4629))
> > >>>>>>>>>> - [AIRFLOW-[4379](tel:4379)] Remove duplicate code & Add
> > >> validation
> > >>>> in
> > >>>>>>>>> gcs_to_gcs.py
> > >>>>>>>>>> (#[5145](tel:5145))
> > >>>>>>>>>> - [AIRFLOW-[4259](tel:4259)] Move models out of models.py
> > >>>> (#[5056](tel:5056))
> > >>>>>>>>>> - [AIRFLOW-XXX] Speed up building of Cassanda module on Travis
> > >>>>>>>>> (#[5233](tel:5233))
> > >>>>>>>>>> - [AIRFLOW-[4535](tel:4535)] Break jobs.py into multiple files
> > >>>> (#[5303](tel:5303))
> > >>>>>>>>>> - [AIRFLOW-[1464](tel:1464)] Batch update task_instance state
> > >>>> (#[5323](tel:5323))
> > >>>>>>>>>> - [AIRFLOW-[4554](tel:4554)] Test for sudo command, add some
> > other
> > >>>> test docs
> > >>>>>>>>> (#[5310](tel:5310))
> > >>>>>>>>>> - [AIRFLOW-[4419](tel:4419)] Refine concurrency check in
> > scheduler
> > >>>> (#[5194](tel:5194))
> > >>>>>>>>>> - [AIRFLOW-[4269](tel:4269)] Minor acceleration of
> > >>>> jobs._process_task_instances()
> > >>>>>>>>>> (#[5076](tel:5076))
> > >>>>>>>>>> - [AIRFLOW-[4341](tel:4341)] Remove ``View.render()`` already
> > >>>> exists in
> > >>>>>>>>> fab.BaseView
> > >>>>>>>>>> (#[5125](tel:5125))
> > >>>>>>>>>> - [AIRFLOW-[4342](tel:4342)] Use @cached_property instead of
> > >>>> re-implementing it
> > >>>>>>>>> each
> > >>>>>>>>>> time (#[5126](tel:5126))
> > >>>>>>>>>> - [AIRFLOW-[4256](tel:4256)] Remove noqa from migrations
> > >>>> (#[5055](tel:5055))
> > >>>>>>>>>> - [AIRFLOW-[4034](tel:4034)] Remove unnecessary string
> > formatting
> > >>>> with
> > >>>>>>>>> ``**locals()``
> > >>>>>>>>>> (#[4861](tel:4861))
> > >>>>>>>>>> - [AIRFLOW-[3944](tel:3944)] Remove code smells
> > >> (#[4762](tel:4762))
> > >>>>>>>>>>
> > >>>>>>>>>> Doc-only changes
> > >>>>>>>>>> """"""""""""""""
> > >>>>>>>>>> - [AIRFLOW-XXX] Correct BaseSensorOperator docs
> > >> (#[5562](tel:5562))
> > >>>>>>>>>> - [AIRFLOW-[4926](tel:4926)] Fix example dags where its
> > start_date
> > >>>> is
> > >>>>>>>>>> datetime.utcnow() (#[5553](tel:5553))
> > >>>>>>>>>> - [AIRFLOW-[4860](tel:4860)] Remove Redundant Information in
> > >>>> Example Dags (#[5497](tel:5497))
> > >>>>>>>>>> - [AIRFLOW-[4767](tel:4767)] Fix errors in the documentation
> of
> > >>>> Dataproc Operator
> > >>>>>>>>>> (#[5487](tel:5487))
> > >>>>>>>>>> - [AIRFLOW-[1684](tel:1684)] Branching based on XCom variable
> > >>>> (Docs) (#[4365](tel:4365))
> > >>>>>>>>>> - [AIRFLOW-[3341](tel:3341)] FAQ return DAG object example
> > >>>> (#[4605](tel:4605))
> > >>>>>>>>>> - [AIRFLOW-[4433](tel:4433)] Add missing type in
> DockerOperator
> > >> doc
> > >>>> string
> > >>>>>>>>> (#[5205](tel:5205))
> > >>>>>>>>>> - [AIRFLOW-[4321](tel:4321)] Replace incorrect info of Max
> Size
> > >>>> limit of GCS
> > >>>>>>>>> Object
> > >>>>>>>>>> Size (#[5106](tel:5106))
> > >>>>>>>>>> - [AIRFLOW-XXX] Add information about user list
> > >> (#[5341](tel:5341))
> > >>>>>>>>>> - [AIRFLOW-XXX] Clarify documentation related to autodetect
> > >>>> parameter
> > >>>>>>>>> in
> > >>>>>>>>>> GCS_to_BQ Op (#[5294](tel:5294))
> > >>>>>>>>>> - [AIRFLOW-XXX] Remove mention of pytz compatibility from
> > timezone
> > >>>>>>>>>> documentation (#[5316](tel:5316))
> > >>>>>>>>>> - [AIRFLOW-XXX] Add missing docs for
> > >>>> GoogleCloudStorageDeleteOperator
> > >>>>>>>>>> (#[5274](tel:5274))
> > >>>>>>>>>> - [AIRFLOW-XXX] Remove incorrect note about Scopes of GCP
> > >> connection
> > >>>>>>>>>> (#[5242](tel:5242))
> > >>>>>>>>>> - [AIRFLOW-XXX] Fix mistakes in docs of Dataproc operators
> > >>>> (#[5192](tel:5192))
> > >>>>>>>>>> - [AIRFLOW-XXX] Link to correct class for timedelta in
> > macros.rst
> > >>>>>>>>> (#[5226](tel:5226))
> > >>>>>>>>>> - [AIRFLOW-XXX] Add Kamil as committer (#[5216](tel:5216))
> > >>>>>>>>>> - [AIRFLOW-XXX] Add Joshua and Kevin as committer
> > >>>> (#[5207](tel:5207))
> > >>>>>>>>>> - [AIRFLOW-XXX] Reduce log spam in tests (#[5174](tel:5174))
> > >>>>>>>>>> - [AIRFLOW-XXX] Speed up tests for PythonSensor
> > >> (#[5158](tel:5158))
> > >>>>>>>>>> - [AIRFLOW-XXX] Add Bas Harenslak to committer list
> > >>>> (#[5157](tel:5157))
> > >>>>>>>>>> - [AIRFLOW-XXX] Add Jarek Potiuk to commiter list
> > >>>> (#[5132](tel:5132))
> > >>>>>>>>>> - [AIRFLOW-XXX] Update docstring for SchedulerJob
> > >>>> (#[5105](tel:5105))
> > >>>>>>>>>> - [AIRFLOW-XXX] Fix docstrings for
> > >>>>>>>>> CassandraToGoogleCloudStorageOperator
> > >>>>>>>>>> (#[5103](tel:5103))
> > >>>>>>>>>> - [AIRFLOW-XXX] update SlackWebhookHook and
> SlackWebhookOperator
> > >>>>>>>>> docstring
> > >>>>>>>>>> (#[5074](tel:5074))
> > >>>>>>>>>> - [AIRFLOW-XXX] Ignore python files under node_modules in docs
> > >>>>>>>>> (#[5063](tel:5063))
> > >>>>>>>>>> - [AIRFLOW-XXX] Build a universal wheel with LICNESE files
> > >>>> (#[5052](tel:5052))
> > >>>>>>>>>> - [AIRFLOW-XXX] Fix docstrings of SQSHook (#[5099](tel:5099))
> > >>>>>>>>>> - [AIRFLOW-XXX] Use Py3.7 on readthedocs
> > >>>>>>>>>> - [AIRFLOW-[4446](tel:4446)] Fix typos (#[5217](tel:5217))
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Best regards,
> > >>>>>>
> > >>>>>> Andrii Soldatenko
> > >>>>>> Python Developer
> > >>>>>>
> > >>>>>> skype: andrii.soldatenko
> > >>>>>> andrii.soldatenko@gmail.com
> > >>>>>> http://asoldatenko.com
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> *Kaxil Naik*
> > >>> *Big Data Consultant | DevOps Data Engineer*
> > >>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> > Neo4j
> > >>> Developer
> > >>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> > >>
> > >>
> >
> >
>


-- 
*Kaxil Naik*
*Big Data Consultant | DevOps Data Engineer*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*LinkedIn*: https://www.linkedin.com/in/kaxil

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message