airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Meickle <jmeic...@quantopian.com.INVALID>
Subject Re: [VOTE] Release Airflow 1.10.4 from RC3
Date Thu, 18 Jul 2019 17:44:41 GMT
Yes, it's zombie processes owned by systemd. Here's a writeup with some
various logs:
https://gist.github.com/Eronarn/c5bc2df607d168d6fda3a70700c941d9

It's reproducible (lmk if you want to co-pilot on debugging), and we're up
to 3 zombies in the hour or two since our most recent reboot. So it won't
immediately take a cluster down, but it is a steady resource leak that
isn't fixed by restarting the scheduler process.

On Thu, Jul 18, 2019 at 12:31 PM Ash Berlin-Taylor <ash@apache.org> wrote:

> Errk, that's not good.
>
> When you say orphaned do you mean zombie processes owned by pid 1, or just
> "process that are hanging around?"
>
> https://github.com/apache/airflow/pull/5605 might help if it's the later
> (but I suspect it isn't)
>
> We've also had reports of the scheduler having a memory leak since ~1.10.2
> https://issues.apache.org/jira/browse/AIRFLOW-4593 so that may be part of
> the OOMing too. (If this is the cause of your problem then .4 isn't worse
> than .3 I don't think?)
>
> -ash
>
> > On 18 Jul 2019, at 15:54, James Meickle <jmeickle@quantopian.com.INVALID>
> wrote:
> >
> > Hi folks,
> >
> > Sorry to throw a wrench into this, but we're found that this release is
> > leaving orphaned processes that eventually OOMing the scheduler. 4
> clusters
> > have this problem post-upgrade, while 1 non-upgraded cluster doesn't have
> > this problem.
> >
> > In our case we're using Supervisor to manage scheduler restarts. I am
> still
> > investigating whether this is an interaction with our unique Supervisor
> > config, or if it's actually purely an Airflow problem. This switch to
> > multiprocessing.Manager looks like a promising avenue of investigation
> > though:
> >
> https://github.com/apache/airflow/compare/1.10.3...1.10.4rc3#diff-ced4fd65ce02db58eed692eef6e01d05L495
> >
> >
> >
> >
> >
> > On Wed, Jul 17, 2019 at 3:35 PM Bolke de Bruin <bdbruin@gmail.com>
> wrote:
> >
> >> +1, binding.
> >>
> >> Thanks ash!
> >>
> >> Verstuurd vanaf mijn iPad
> >>
> >>> Op 17 jul. 2019 om 20:45 heeft Kaxil Naik <kaxilnaik@gmail.com> het
> >> volgende geschreven:
> >>>
> >>> +1 (binding) Ran DAGs in both UIs. LGTM
> >>>
> >>> Great Job Ash. Appreciate all the effort you put into this.
> >>>
> >>> Regards,
> >>> Kaxil
> >>>
> >>> On Wed, Jul 17, 2019 at 10:30 PM Felix Uellendall
> <feluelle@pm.me.invalid
> >>>
> >>> wrote:
> >>>
> >>>> +1 (non-binding) flawlessly passed all my tests via rbac and classic
> ui.
> >>>> Also tested it on production-level dags. Great job Ash, thanks :)
> >>>>
> >>>> Kind regards,
> >>>> Felix
> >>>>
> >>>> Sent from ProtonMail mobile
> >>>>
> >>>> -------- Original Message --------
> >>>>> On Jul 17, 2019, 16:41, Ash Berlin-Taylor wrote:
> >>>>>
> >>>>> Thanks Andrii - I missed it to when reviewing the PR so it's my fault
> >>>> too. Make the PR against the v[1-10](tel:110)-stable branch please.
> >>>>>
> >>>>> -ash
> >>>>>
> >>>>>> On 17 Jul [2019](tel:2019), at 14:25, Andrii Soldatenko <
> >>>> andrii.soldatenko@gmail.com> wrote:
> >>>>>>
> >>>>>> @Ash, i'll fix wrong section bug.
> >>>>>>
> >>>>>> Sorry about that.
> >>>>>>
> >>>>>> On Wed, Jul 17, [2019](tel:2019) at 4:11 PM Robin Edwards <
> >>>> rob@bidnamic.com> wrote:
> >>>>>>
> >>>>>>> +1 (none binding) - been running in production since RC2.
> >>>>>>>
> >>>>>>> Thanks for all your hard work
> >>>>>>>
> >>>>>>> R
> >>>>>>>
> >>>>>>> On Tue, 16 Jul [2019](tel:2019) at 21:15, Ash Berlin-Taylor <
> >>>> ash@apache.org> wrote:
> >>>>>>>>
> >>>>>>>> Thanks for testing.
> >>>>>>>>
> >>>>>>>> On 1) everyone should run upgradedb on every upgrade. The
> behaviour
> >> of
> >>>>>>> not running it wasn't great.
> >>>>>>>>
> >>>>>>>> 2) I thought we set deprecation on the ES logging config vars,
> >> except
> >>>> we
> >>>>>>> put the depreciation under the wrong section:
> >>>>>>>
> >>>>
> >>
> https://github.com/apache/airflow/blob/1.10.4rc3/airflow/configuration.py#L147-L153
> >>>>>>> :(
> >>>>>>>>
> >>>>>>>> On 16 July [2019 21](tel:201921):06:40 BST, James Meickle <
> >>>> jmeickle@quantopian.com.INVALID>
> >>>>>>> wrote:
> >>>>>>>>> +1 (nonbinding) to the release, it fixes a lot of UI issues we've
> >>>> been
> >>>>>>>>> seeing lately. Though two notes:
> >>>>>>>>>
> >>>>>>>>> 1) Tasks were unscheduleable until I ran an upgradedb due the
> >> default
> >>>>>>>>> pool
> >>>>>>>>> change.
> >>>>>>>>>
> >>>>>>>>> 2) I got crash loops because I based our custom logging file off
> of
> >>>> the
> >>>>>>>>> previous version's template. The change to the Elasticsearch conf
> >>>>>>>>> variables
> >>>>>>>>> broke this, even though we don't directly use Elasticsearch,
> since
> >> it
> >>>>>>>>> was
> >>>>>>>>> now looking for undefined conf variables. It was confusing until
> I
> >>>>>>>>> hunted
> >>>>>>>>> around the diff and backported the changes in the default
> template
> >> to
> >>>>>>>>> the
> >>>>>>>>> template that lives in our repo.
> >>>>>>>>>
> >>>>>>>>> On Mon, Jul 15, [2019](tel:2019) at 10:17 AM Ash Berlin-Taylor <
> >>>> ash@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hello Airflow community,
> >>>>>>>>>>
> >>>>>>>>>> This email is calling a vote on the release, which will last for
> >> 72
> >>>>>>>>> hours
> >>>>>>>>>> (2019-07-08 15:15 Z), and until three binding votes have been
> >> cast.
> >>>>>>>>>> Consider this my (binding) +1.
> >>>>>>>>>>
> >>>>>>>>>> Airflow [1.10.4](tel:1104) RC3 is available at:
> >>>>>>>>>> https://dist.apache.org/repos/dist/dev/airflow/1.10.4rc3/
> >>>>>>>>>>
> >>>>>>>>>> *apache-airflow-[1.10.4](tel:1104)rc3-source.tar.gz* is a source
> >>>> release that
> >>>>>>>>> comes
> >>>>>>>>>> with INSTALL instructions.
> >>>>>>>>>> *apache-airflow-[1.10.4](tel:1104)rc3-bin.tar.gz* is the binary
> >>>> Python "sdist"
> >>>>>>>>> release.
> >>>>>>>>>> *apache_airflow-[1.10.4](tel:1104)rc3-py2.py3-none-any.whl* is
> the
> >>>> binary Python
> >>>>>>>>>> "wheel" release.
> >>>>>>>>>>
> >>>>>>>>>> For connivence of testers the RC is on PYPI too. It can be
> >> installed
> >>>>>>>>> with:
> >>>>>>>>>>
> >>>>>>>>>> pip install 'apache-airflow==[1.10.4](tel:1104)rc3'
> >>>>>>>>>>
> >>>>>>>>>> Public keys are available at:
> >>>>>>>>>> https://dist.apache.org/repos/dist/release/airflow/KEYS
> >>>>>>>>>>
> >>>>>>>>>> Only votes from PMC members are binding (sorry committers), but
> >>>>>>>>> members of
> >>>>>>>>>> the community are encouraged to test the release and vote with
> >>>>>>>>>> "(non-binding)".
> >>>>>>>>>>
> >>>>>>>>>> Please note that the version number excludes the `rcX` string,
> so
> >>>>>>>>> it's now
> >>>>>>>>>> simply [1.10.4](tel:1104). This will allow us to rename the
> >>>> artefact without
> >>>>>>>>> modifying
> >>>>>>>>>> the checksums when we actually release.
> >>>>>>>>>>
> >>>>>>>>>> [ ] +1 Release this package as Apache Airflow [1.10.4](tel:1104)
> >>>>>>>>>> [ ] 0 No opinion
> >>>>>>>>>> [ ] -1 Do not release this package because...
> >>>>>>>>>>
> >>>>>>>>>> Changes since RC2:
> >>>>>>>>>>
> >>>>>>>>>> [AIRFLOW-XXX] Update changelog for [1.10.4](tel:1104)rc3 [Ash
> >>>> Berlin-Taylor]
> >>>>>>>>>> [AIRFLOW-XXX] Disable intersphinx loading of `requests` modules
> >>>>>>>>> (#[5590](tel:5590))
> >>>>>>>>>> [Ash Berlin-Taylor]
> >>>>>>>>>> [AIRFLOW-[4797](tel:4797)] Improve performance and behaviour of
> >>>> zombie detection
> >>>>>>>>>> (#[5511](tel:5511)) [Stefan Seelmann]
> >>>>>>>>>> [AIRFLOW-[4911](tel:4911)] Silence the FORBIDDEN errors from the
> >>>>>>>>> KubernetesExecutor
> >>>>>>>>>> (#[5547](tel:5547)) [Andrii Soldatenko]
> >>>>>>>>>> [AIRFLOW-[4884](tel:4884)] Roll up import_errors in RBAC UI
> >>>> (#[5516](tel:5516))
> >>>>>>>>> [shuwen-zhang]
> >>>>>>>>>> [AIRFLOW-[4937](tel:4937)] Fix lodash security issue with
> version
> >>>> below [4.17.13](tel:41713)
> >>>>>>>>>> (#[5572](tel:5572)) [Jarek Potiuk]
> >>>>>>>>>> [AIRFLOW-[4934](tel:4934)] Bump Flask to resolve Werkzeug
> ProxyFix
> >>>> (#[5571](tel:5571))
> >>>>>>>>> [Nathaniel
> >>>>>>>>>> Ritholtz]
> >>>>>>>>>> [AIRFLOW-[4934](tel:4934)] Fix ProxyFix due to Werkzeug upgrade
> >>>> (#[5563](tel:5563))
> >>>>>>>>> [Nathaniel
> >>>>>>>>>> Ritholtz]
> >>>>>>>>>> [AIRFLOW-[4136](tel:4136)] fix key_file of hook is overwritten
> by
> >>>> SSHHook
> >>>>>>>>> connection
> >>>>>>>>>> (#[5558](tel:5558)) [eladkal]
> >>>>>>>>>> [AIRFLOW-XXX] Correct BaseSensorOperator docs
> (#[5562](tel:5562))
> >>>> [Matthew
> >>>>>>>>> Bowden]
> >>>>>>>>>> [AIRFLOW-[3495](tel:3495)] Validate one of query and query_uri
> >>>> passed to
> >>>>>>>>>> DataProcSparkSqlOperator (#[5510](tel:5510)) [eladkal]
> >>>>>>>>>> [AIRFLOW-[4868](tel:4868)] Fix typo in
> kubernetes/docker/build.sh
> >>>> (#[5505](tel:5505)) [Facu]
> >>>>>>>>>> [AIRFLOW-[4925](tel:4925)] Improve css style for Variables
> Import
> >>>> file field
> >>>>>>>>> (#[5552](tel:5552))
> >>>>>>>>>> [Felix Uellendall]
> >>>>>>>>>> [AIRFLOW-[4906](tel:4906)] Improve debugging for the
> >>>> SparkSubmitHook (#[5542](tel:5542))
> >>>>>>>>> [Fokko
> >>>>>>>>>> Driesprong]
> >>>>>>>>>> [AIRFLOW-[4904](tel:4904)] Retrieve test config file from
> >>>> $AIRFLOW_TEST_CONFIG
> >>>>>>>>> (#[5540](tel:5540))
> >>>>>>>>>> [Matthew Rastovac]
> >>>>>>>>>> [AIRFLOW-[4926](tel:4926)] Fix example dags where its start_date
> >> is
> >>>>>>>>> datetime.utcnow()
> >>>>>>>>>> (#[5553](tel:5553)) [Felix Uellendall]
> >>>>>>>>>> [AIRFLOW-[4587](tel:4587)] Replace self.conn with
> self.get_conn()
> >> in
> >>>>>>>>> AWSAthenaHook
> >>>>>>>>>> (#[5545](tel:5545)) [Bhavika Tekwani]
> >>>>>>>>>> [AIRFLOW-[1740](tel:1740)] Fix xcom creation and update via UI
> >>>> (#[5530](tel:5530)) [Felix
> >>>>>>>>>> Uellendall]
> >>>>>>>>>> [AIRFLOW-[1740](tel:1740)] Fix xcom creation and update via RBAC
> >> UI
> >>>> (#[5561](tel:5561))
> >>>>>>>>> [Felix
> >>>>>>>>>> Uellendall]
> >>>>>>>>>> [AIRFLOW-[4920](tel:4920)] Use html.escape instead of cgi.escape
> >> to
> >>>> fix
> >>>>>>>>>> DeprecationWarning (#[5551](tel:5551)) [Xtreak]
> >>>>>>>>>> [AIRFLOW-[4919](tel:4919)] DataProcJobBaseOperator
> >>>> dataproc_*_properties
> >>>>>>>>> templated
> >>>>>>>>>> (#[5555](tel:5555)) [eladkal]
> >>>>>>>>>> [AIRFLOW-[4900](tel:4900)] Resolve incompatible version of
> >> Werkzeug
> >>>> (#[5535](tel:5535))
> >>>>>>>>> [Luqman
> >>>>>>>>>> Abdul Mushawwir]
> >>>>>>>>>> [AIRFLOW-[4478](tel:4478)] Lazily instantiate default resources
> >>>> objects. (#[5259](tel:5259))
> >>>>>>>>>> [Joshua Carp]
> >>>>>>>>>> [AIRFLOW-[4510](tel:4510)] Don't mutate default_args during DAG
> >>>> initialization
> >>>>>>>>> (#[5277](tel:5277))
> >>>>>>>>>> [Abhishek Ray]
> >>>>>>>>>> [AIRFLOW-[3360](tel:3360)] Make the DAGs search respect other
> >>>> querystring
> >>>>>>>>> parameters
> >>>>>>>>>> with url-search-params-polyfill for IE support
> (#[5503](tel:5503))
> >>>> [Alex]
> >>>>>>>>>> [AIRFLOW-[4896](tel:4896)] Make KubernetesExecutorConfig's
> default
> >>>> args immutable
> >>>>>>>>>> (#[5534](tel:5534)) [Philippe Gagnon]
> >>>>>>>>>> [AIRFLOW-[4494](tel:4494)] Remove `shell=True` in DaskExecutor
> >>>> (#[5273](tel:5273)) [MisLink]
> >>>>>>>>>> [AIRFLOW-[4890](tel:4890)] Fix Log link in TaskInstance's View
> for
> >>>> Non-RBAC
> >>>>>>>>> (#[5525](tel:5525))
> >>>>>>>>>> [Felix Uellendall]
> >>>>>>>>>> [AIRFLOW-[4892](tel:4892)] Fix connection creation via UIs
> >>>> (#[5527](tel:5527)) [Ash
> >>>>>>>>> Berlin-Taylor]
> >>>>>>>>>> [AIRFLOW-[4862](tel:4862)] Fix bug for earlier change to allow
> >>>> using IP as
> >>>>>>>>> hostname
> >>>>>>>>>> (#[5513](tel:5513)) [Xiaodong]
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Full Changelog since [1.10.3](tel:1103):
> >>>>>>>>>>
> >>>>>>>>>> Airflow [1.10.4](tel:1104), - [2019-07-18](tel:20190718)
> >>>>>>>>>> ----------------------------
> >>>>>>>>>>
> >>>>>>>>>> New Features
> >>>>>>>>>> """"""""""""
> >>>>>>>>>> - [AIRFLOW-[4884](tel:4884)] Roll up import_errors in RBAC UI
> >>>> (#[5516](tel:5516))
> >>>>>>>>>> - [AIRFLOW-[4871](tel:4871)] Allow creating DagRuns via RBAC UI
> >>>> (#[5507](tel:5507))
> >>>>>>>>>> - [AIRFLOW-[4591](tel:4591)] Make default_pool a real pool
> >>>> (#[5349](tel:5349))
> >>>>>>>>>> - [AIRFLOW-[4844](tel:4844)] Add optional
> is_paused_upon_creation
> >>>> argument to DAG
> >>>>>>>>>> (#[5473](tel:5473))
> >>>>>>>>>> - [AIRFLOW-[4456](tel:4456)] Add sub-classable
> BaseBranchOperator
> >>>> (#[5231](tel:5231))
> >>>>>>>>>> - [AIRFLOW-[4343](tel:4343)] Show warning in UI if scheduler is
> >> not
> >>>> running
> >>>>>>>>> (#[5127](tel:5127))
> >>>>>>>>>> - [AIRFLOW-[4739](tel:4739)] Add ability to arbitrarily define
> >>>> kubernetes worker
> >>>>>>>>> pod
> >>>>>>>>>> labels (#[5376](tel:5376))
> >>>>>>>>>> - [AIRFLOW-[4348](tel:4348)] Add GCP console link in
> >>>> BigQueryOperator (#[5195](tel:5195))
> >>>>>>>>>> - [AIRFLOW-[4306](tel:4306)] Global operator extra links
> >>>> (#[5094](tel:5094))
> >>>>>>>>>> - [AIRFLOW-[4812](tel:4812)] Add batch images annotation
> >>>> (#[5433](tel:5433))
> >>>>>>>>>> - [AIRFLOW-[4135](tel:4135)] Add Google Cloud Build operator and
> >>>> hook (#[5251](tel:5251))
> >>>>>>>>>> - [AIRFLOW-[4781](tel:4781)] Add the ability to specify ports in
> >>>>>>>>> kubernetesOperator
> >>>>>>>>>> (#[5410](tel:5410))
> >>>>>>>>>> - [AIRFLOW-[4521](tel:4521)] Pause dag also pause its subdags
> >>>> (#[5283](tel:5283))
> >>>>>>>>>> - [AIRFLOW-[4738](tel:4738)] Enforce exampleinclude for example
> >>>> DAGs (#[5375](tel:5375))
> >>>>>>>>>> - [AIRFLOW-[4326](tel:4326)] Airflow AWS SQS Operator
> >>>> (#[5110](tel:5110))
> >>>>>>>>>> - [AIRFLOW-[3729](tel:3729)] Support "DownwardAPI" in env
> >> variables
> >>>> for
> >>>>>>>>>> KubernetesPodOperator (#[4554](tel:4554))
> >>>>>>>>>> - [AIRFLOW-[4585](tel:4585)] Implement Kubernetes Pod Mutation
> >> Hook
> >>>> (#[5359](tel:5359))
> >>>>>>>>>> - [AIRFLOW-[161](tel:161)] New redirect route and extra links
> >>>> (#[5059](tel:5059))
> >>>>>>>>>> - [AIRFLOW-[4420](tel:4420)] Backfill respects task_concurrency
> >>>> (#[5221](tel:5221))
> >>>>>>>>>> - [AIRFLOW-[4147](tel:4147)] Add Operator to publish event to
> >> Redis
> >>>> (#[4967](tel:4967))
> >>>>>>>>>> - [AIRFLOW-[3359](tel:3359)] Add option to pass customer
> >> encryption
> >>>> keys to
> >>>>>>>>> Dataproc
> >>>>>>>>>> (#[4200](tel:4200))
> >>>>>>>>>> - [AIRFLOW-[4318](tel:4318)] Create Google Cloud Translate
> Speech
> >>>> Operator
> >>>>>>>>> (#[5102](tel:5102))
> >>>>>>>>>> - [AIRFLOW-[3960](tel:3960)] Adds Google Cloud Speech operators
> >>>> (#[4780](tel:4780))
> >>>>>>>>>> - [AIRFLOW-[1501](tel:1501)] Add
> GoogleCloudStorageDeleteOperator
> >>>> (#[5230](tel:5230))
> >>>>>>>>>> - [AIRFLOW-[3672](tel:3672)] Add support for Mongo DB DNS
> Seedlist
> >>>> Connection
> >>>>>>>>> Format
> >>>>>>>>>> (#[4481](tel:4481))
> >>>>>>>>>> - [AIRFLOW-[4397](tel:4397)] add integrations docs manually for
> >> gcs
> >>>> sensors
> >>>>>>>>> (#[5204](tel:5204))
> >>>>>>>>>> - [AIRFLOW-[4251](tel:4251)] Instrument DagRun schedule delay
> >>>> (#[5050](tel:5050))
> >>>>>>>>>> - [AIRFLOW-[4118](tel:4118)] instrument DagRun duration
> >>>> (#[4946](tel:4946))
> >>>>>>>>>> - [AIRFLOW-[4361](tel:4361)] Fix flaky
> >>>>>>>>> test_integration_run_dag_with_scheduler_failure
> >>>>>>>>>> (#[5182](tel:5182))
> >>>>>>>>>> - [AIRFLOW-[4361](tel:4361)] Fix flaky
> >>>>>>>>> test_integration_run_dag_with_scheduler_failure
> >>>>>>>>>> (#[5140](tel:5140))
> >>>>>>>>>> - [AIRFLOW-[4168](tel:4168)] Create Google Cloud Video
> >> Intelligence
> >>>> Operators
> >>>>>>>>> (#[4985](tel:4985))
> >>>>>>>>>> - [AIRFLOW-[4397](tel:4397)] Add GCSUploadSessionCompleteSensor
> >>>> (#[5166](tel:5166))
> >>>>>>>>>> - [AIRFLOW-[4335](tel:4335)] Add default num_retries to GCP
> >>>> connection (#[5117](tel:5117))
> >>>>>>>>>> - [AIRFLOW-[3808](tel:3808)] Add cluster_fields to
> BigQueryHook's
> >>>>>>>>> create_empty_table
> >>>>>>>>>> (#[4654](tel:4654))
> >>>>>>>>>> - [AIRFLOW-[4362](tel:4362)] Fix
> >> test_execution_limited_parallelism
> >>>> (#[5141](tel:5141))
> >>>>>>>>>> - [AIRFLOW-[4307](tel:4307)] Backfill respects concurrency limit
> >>>> (#[5128](tel:5128))
> >>>>>>>>>> - [AIRFLOW-[4268](tel:4268)] Add
> MsSqlToGoogleCloudStorageOperator
> >>>> (#[5077](tel:5077))
> >>>>>>>>>> - [AIRFLOW-[4169](tel:4169)] Add Google Cloud Vision Detect
> >>>> Operators (#[4986](tel:4986))
> >>>>>>>>>> - [AIRFLOW-XXX] Fix WS-[2019-0032](tel:20190032)
> >> (#[5384](tel:5384))
> >>>>>>>>>> - [AIRFLOW-XXX] Fix CVE-[2019-11358](tel:201911358)
> >>>> (#[5197](tel:5197))
> >>>>>>>>>> - [AIRFLOW-XXX] Change allowed version of Jinja2 to fix
> >>>>>>>>> CVE-[2019-10906](tel:201910906)
> >>>>>>>>>> (#[5075](tel:5075))
> >>>>>>>>>>
> >>>>>>>>>> Improvement
> >>>>>>>>>> """""""""""
> >>>>>>>>>> - [AIRFLOW-[4797](tel:4797)] Improve performance and behaviour
> of
> >>>> zombie
> >>>>>>>>> detection
> >>>>>>>>>> (#[5511](tel:5511))
> >>>>>>>>>> - [AIRFLOW-[4911](tel:4911)] Silence the FORBIDDEN errors from
> the
> >>>>>>>>> KubernetesExecutor
> >>>>>>>>>> (#[5547](tel:5547))
> >>>>>>>>>> - [AIRFLOW-[3495](tel:3495)] Validate one of query and query_uri
> >>>> passed to
> >>>>>>>>>> DataProcSparkSqlOperator (#[5510](tel:5510))
> >>>>>>>>>> - [AIRFLOW-[4925](tel:4925)] Improve css style for Variables
> >> Import
> >>>> file field
> >>>>>>>>> (#[5552](tel:5552))
> >>>>>>>>>> - [AIRFLOW-[4906](tel:4906)] Improve debugging for the
> >>>> SparkSubmitHook (#[5542](tel:5542))
> >>>>>>>>>> - [AIRFLOW-[4904](tel:4904)] unittest.cfg name and path can be
> >>>> overriden by
> >>>>>>>>> setting
> >>>>>>>>>> $AIRFLOW_TEST_CONFIG (#[5540](tel:5540))
> >>>>>>>>>> - [AIRFLOW-[4920](tel:4920)] Use html.escape instead of
> cgi.escape
> >>>> to fix
> >>>>>>>>>> DeprecationWarning (#[5551](tel:5551))
> >>>>>>>>>> - [AIRFLOW-[4919](tel:4919)] DataProcJobBaseOperator
> >>>> dataproc_*_properties
> >>>>>>>>> templated
> >>>>>>>>>> (#[5555](tel:5555))
> >>>>>>>>>> - [AIRFLOW-[4478](tel:4478)] Lazily instantiate default
> resources
> >>>> objects.
> >>>>>>>>> (#[5259](tel:5259))
> >>>>>>>>>> - [AIRFLOW-[4564](tel:4564)] AzureContainerInstance bugfixes and
> >>>> improvements
> >>>>>>>>> (#[5319](tel:5319))
> >>>>>>>>>> - [AIRFLOW-[4237](tel:4237)] Including Try Number of Task in
> Gantt
> >>>> Chart (#[5037](tel:5037))
> >>>>>>>>>> - [AIRFLOW-[4862](tel:4862)] Allow directly using IP address as
> >>>> hostname for
> >>>>>>>>> webserver
> >>>>>>>>>> logs (#[5501](tel:5501))
> >>>>>>>>>> - [AIRFLOW-[4857](tel:4857)] Add templated fields to
> >>>> SlackWebhookOperator (#[5490](tel:5490))
> >>>>>>>>>> - [AIRFLOW-[3502](tel:3502)] Add celery config option for
> setting
> >>>> "pool" (#[4308](tel:4308))
> >>>>>>>>>> - [AIRFLOW-[3217](tel:3217)] Button to toggle line wrapping in
> log
> >>>> and code views
> >>>>>>>>>> (#[4277](tel:4277))
> >>>>>>>>>> - [AIRFLOW-[4491](tel:4491)] Add a "Jump to end" button for logs
> >>>> (#[5266](tel:5266))
> >>>>>>>>>> - [AIRFLOW-[4422](tel:4422)] Pool utilization stats
> >>>> (#[5453](tel:5453))
> >>>>>>>>>> - [AIRFLOW-[4805](tel:4805)] Add py_file as templated field in
> >>>>>>>>> DataflowPythonOperator
> >>>>>>>>>> (#[5451](tel:5451))
> >>>>>>>>>> - [AIRFLOW-[4838](tel:4838)] Surface Athena errors in
> >>>> AWSAthenaOperator (#[5467](tel:5467))
> >>>>>>>>>> - [AIRFLOW-[4831](tel:4831)] conf.has_option no longer throws if
> >>>> section is
> >>>>>>>>> missing.
> >>>>>>>>>> (#[5455](tel:5455))
> >>>>>>>>>> - [AIRFLOW-[4829](tel:4829)] More descriptive exceptions for EMR
> >>>> sensors (#[5452](tel:5452))
> >>>>>>>>>> - [AIRFLOW-[4414](tel:4414)] AWSAthenaOperator: Push
> >>>> QueryExecutionID to XCom
> >>>>>>>>> (#[5276](tel:5276))
> >>>>>>>>>> - [AIRFLOW-[4791](tel:4791)] add "schema" keyword arg to
> >>>> SnowflakeOperator
> >>>>>>>>> (#[5415](tel:5415))
> >>>>>>>>>> - [AIRFLOW-[4759](tel:4759)] Don't error when marking sucessful
> >> run
> >>>> as failed
> >>>>>>>>> (#[5435](tel:5435))
> >>>>>>>>>> - [AIRFLOW-[4716](tel:4716)] Instrument dag loading time
> duration
> >>>> (#[5350](tel:5350))
> >>>>>>>>>> - [AIRFLOW-[3958](tel:3958)] Support list tasks as upstream in
> >>>> chain (#[4779](tel:4779))
> >>>>>>>>>> - [AIRFLOW-[4409](tel:4409)] Prevent task duration break by null
> >>>> value (#[5178](tel:5178))
> >>>>>>>>>> - [AIRFLOW-[4418](tel:4418)] Add "failed only" option to task
> >> modal
> >>>> (#[5193](tel:5193))
> >>>>>>>>>> - [AIRFLOW-[4740](tel:4740)] Accept string ``end_date`` in DAG
> >>>> default_args
> >>>>>>>>> (#[5381](tel:5381))
> >>>>>>>>>> - [AIRFLOW-[4423](tel:4423)] Improve date handling in mysql to
> gcs
> >>>> operator.
> >>>>>>>>> (#[5196](tel:5196))
> >>>>>>>>>> - [AIRFLOW-[4447](tel:4447)] Display task duration as human
> >>>> friendly format in UI
> >>>>>>>>>> (#[5218](tel:5218))
> >>>>>>>>>> - [AIRFLOW-[4377](tel:4377)] Remove needless object conversion
> in
> >>>> DAG.owner()
> >>>>>>>>> (#[5144](tel:5144))
> >>>>>>>>>> - [AIRFLOW-[4766](tel:4766)] Add autoscaling option for
> >>>>>>>>> DataprocClusterCreateOperator
> >>>>>>>>>> (#[5425](tel:5425))
> >>>>>>>>>> - [AIRFLOW-[4795](tel:4795)] Upgrade alembic to latest release.
> >>>> (#[5411](tel:5411))
> >>>>>>>>>> - [AIRFLOW-[4793](tel:4793)] Add signature_name to mlengine
> >>>> operator (#[5417](tel:5417))
> >>>>>>>>>> - [AIRFLOW-[3211](tel:3211)] Reattach to GCP Dataproc jobs upon
> >>>> Airflow restart
> >>>>>>>>>> (#[4083](tel:4083))
> >>>>>>>>>> - [AIRFLOW-[4750](tel:4750)] Log identified zombie task
> instances
> >>>> (#[5389](tel:5389))
> >>>>>>>>>> - [AIRFLOW-[3870](tel:3870)] STFPOperator: Update log level and
> >>>> return value
> >>>>>>>>> (#[4355](tel:4355))
> >>>>>>>>>> - [AIRFLOW-[4759](tel:4759)] Batch queries in set_state API.
> >>>> (#[5403](tel:5403))
> >>>>>>>>>> - [AIRFLOW-[2737](tel:2737)] Restore original license header to
> >>>>>>>>>> airflow.api.auth.backend.kerberos_auth
> >>>>>>>>>> - [AIRFLOW-[3635](tel:3635)] Fix incorrect logic in detele_dag
> >>>> (introduced in
> >>>>>>>>> PR#[4406](tel:4406))
> >>>>>>>>>> (#[4445](tel:4445))
> >>>>>>>>>> - [AIRFLOW-[3599](tel:3599)] Removed Dagbag from delete dag
> >>>> (#[4406](tel:4406))
> >>>>>>>>>> - [AIRFLOW-[4737](tel:4737)] Increase and document celery queue
> >>>> name limit
> >>>>>>>>> (#[5383](tel:5383))
> >>>>>>>>>> - [AIRFLOW-[4505](tel:4505)] Correct Tag ALL for PY3
> >>>> (#[5275](tel:5275))
> >>>>>>>>>> - [AIRFLOW-[4743](tel:4743)] Add environment variables support
> to
> >>>> SSHOperator
> >>>>>>>>> (#[5385](tel:5385))
> >>>>>>>>>> - [AIRFLOW-[4725](tel:4725)] Fix setup.py PEP[440](tel:440) &
> >>>> Sphinx-PyPI-upload dependency
> >>>>>>>>>> (#[5363](tel:5363))
> >>>>>>>>>> - [AIRFLOW-[3370](tel:3370)] Add stdout output options to
> >>>> Elasticsearch task log
> >>>>>>>>>> handler (#[5048](tel:5048))
> >>>>>>>>>> - [AIRFLOW-[4396](tel:4396)] Provide a link to external
> >>>> Elasticsearch logs in UI.
> >>>>>>>>>> (#[5164](tel:5164))
> >>>>>>>>>> - [AIRFLOW-[1381](tel:1381)] Allow setting host temporary
> >> directory
> >>>> in
> >>>>>>>>> DockerOperator
> >>>>>>>>>> (#[5369](tel:5369))
> >>>>>>>>>> - [AIRFLOW-[4598](tel:4598)] Task retries are not exhausted for
> >> K8s
> >>>> executor
> >>>>>>>>> (#[5347](tel:5347))
> >>>>>>>>>> - [AIRFLOW-[4218](tel:4218)] Support to Provide http args to
> >>>> K8executor while
> >>>>>>>>> calling
> >>>>>>>>>> k8 python client lib apis (#[5060](tel:5060))
> >>>>>>>>>> - [AIRFLOW-[4159](tel:4159)] Add support for additional static
> pod
> >>>> labels for
> >>>>>>>>>> K8sExecutor (#[5134](tel:5134))
> >>>>>>>>>> - [AIRFLOW-[4720](tel:4720)] Allow comments in .airflowignore
> >>>> files. (#[5355](tel:5355))
> >>>>>>>>>> - [AIRFLOW-[4486](tel:4486)] Add AWS IAM authenication in
> >> MySqlHook
> >>>> (#[5334](tel:5334))
> >>>>>>>>>> - [AIRFLOW-[4417](tel:4417)] Add AWS IAM authenication for
> >>>> PostgresHook (#[5223](tel:5223))
> >>>>>>>>>> - [AIRFLOW-[3990](tel:3990)] Compile regular expressions.
> >>>> (#[4813](tel:4813))
> >>>>>>>>>> - [AIRFLOW-[4572](tel:4572)] Rename prepare_classpath() to
> >>>> prepare_syspath()
> >>>>>>>>> (#[5328](tel:5328))
> >>>>>>>>>> - [AIRFLOW-[3869](tel:3869)] Raise consistent exception in
> >>>>>>>>>> AirflowConfigParser.getboolean (#[4692](tel:4692))
> >>>>>>>>>> - [AIRFLOW-[4571](tel:4571)] Add headers to templated field for
> >>>>>>>>> SimpleHttpOperator
> >>>>>>>>>> (#[5326](tel:5326))
> >>>>>>>>>> - [AIRFLOW-[3867](tel:3867)] Rename GCP's subpackage
> >>>> (#[4690](tel:4690))
> >>>>>>>>>> - [AIRFLOW-[3725](tel:3725)] Add private_key to bigquery_hook
> >>>> get_pandas_df
> >>>>>>>>> (#[4549](tel:4549))
> >>>>>>>>>> - [AIRFLOW-[4546](tel:4546)] Upgrade google-cloud-bigtable.
> >>>> (#[5307](tel:5307))
> >>>>>>>>>> - [AIRFLOW-[4519](tel:4519)] Optimise operator classname sorting
> >> in
> >>>> views (#[5282](tel:5282))
> >>>>>>>>>> - [AIRFLOW-[4503](tel:4503)] Support fully pig options
> >>>> (#[5271](tel:5271))
> >>>>>>>>>> - [AIRFLOW-[4468](tel:4468)] add sql_alchemy_max_overflow
> >> parameter
> >>>> (#[5249](tel:5249))
> >>>>>>>>>> - [AIRFLOW-[4467](tel:4467)] Add dataproc_jars to templated
> fields
> >>>> in Dataproc
> >>>>>>>>> oper…
> >>>>>>>>>> (#[5248](tel:5248))
> >>>>>>>>>> - [AIRFLOW-[4381](tel:4381)] Use get_direct_relative_ids get
> task
> >>>> relatives
> >>>>>>>>> (#[5147](tel:5147))
> >>>>>>>>>> - [AIRFLOW-[3624](tel:3624)] Add masterType parameter to
> >>>> MLEngineTrainingOperator
> >>>>>>>>>> (#[4428](tel:4428))
> >>>>>>>>>> - [AIRFLOW-[3143](tel:3143)] Support Auto-Zone in
> >>>> DataprocClusterCreateOperator
> >>>>>>>>> (#[5169](tel:5169))
> >>>>>>>>>> - [AIRFLOW-[3874](tel:3874)] Improve
> >>>> BigQueryHook.run_with_configuration's
> >>>>>>>>> location
> >>>>>>>>>> support (#[4695](tel:4695))
> >>>>>>>>>> - [AIRFLOW-[4399](tel:4399)] Avoid duplicated os.path.isfile()
> >>>> check in
> >>>>>>>>> models.dagbag
> >>>>>>>>>> (#[5165](tel:5165))
> >>>>>>>>>> - [AIRFLOW-[4031](tel:4031)] Allow for key pair auth in
> snowflake
> >>>> hook (#[4875](tel:4875))
> >>>>>>>>>> - [AIRFLOW-[3901](tel:3901)] add role as optional config
> parameter
> >>>> for
> >>>>>>>>> SnowflakeHook
> >>>>>>>>>> (#[4721](tel:4721))
> >>>>>>>>>> - [AIRFLOW-[3455](tel:3455)] add region in snowflake connector
> >>>> (#[4285](tel:4285))
> >>>>>>>>>> - [AIRFLOW-[4073](tel:4073)] add template_ext for AWS Athena
> >>>> operator (#[4907](tel:4907))
> >>>>>>>>>> - [AIRFLOW-[4093](tel:4093)] AWSAthenaOperator: Throw exception
> if
> >>>> job
> >>>>>>>>>> failed/cancelled/reach max retries (#[4919](tel:4919))
> >>>>>>>>>> - [AIRFLOW-[4356](tel:4356)] Add extra RuntimeEnvironment keys
> to
> >>>> DataFlowHook
> >>>>>>>>> (#[5149](tel:5149))
> >>>>>>>>>> - [AIRFLOW-[4337](tel:4337)] Fix docker-compose deprecation
> >> warning
> >>>> in CI (#[5119](tel:5119))
> >>>>>>>>>> - [AIRFLOW-[3603](tel:3603)] QuboleOperator: Remove SQLCommand
> >> from
> >>>> SparkCmd
> >>>>>>>>>> documentation (#[4411](tel:4411))
> >>>>>>>>>> - [AIRFLOW-[4328](tel:4328)] Fix link to task instances from
> Pool
> >>>> page (#[5124](tel:5124))
> >>>>>>>>>> - [AIRFLOW-[4255](tel:4255)] Make GCS Hook Backwards compatible
> >>>> (#[5089](tel:5089))
> >>>>>>>>>> - [AIRFLOW-[4103](tel:4103)] Allow uppercase letters in dataflow
> >>>> job names
> >>>>>>>>> (#[4925](tel:4925))
> >>>>>>>>>> - [AIRFLOW-[4255](tel:4255)] Replace Discovery based api with
> >>>> client based for
> >>>>>>>>> GCS
> >>>>>>>>>> (#[5054](tel:5054))
> >>>>>>>>>> - [AIRFLOW-[4311](tel:4311)] Remove sleep in localexecutor
> >>>> (#[5096](tel:5096))
> >>>>>>>>>> - [AIRFLOW-[2836](tel:2836)] Minor
> >>>> improvement-contrib.sensors.FileSensor (#[3674](tel:3674))
> >>>>>>>>>> - [AIRFLOW-[4104](tel:4104)] Add type annotations to common
> >>>> classes. (#[4926](tel:4926))
> >>>>>>>>>> - [AIRFLOW-[3910](tel:3910)] Raise exception explicitly in
> >>>> Connection.get_hook()
> >>>>>>>>>> (#[4728](tel:4728))
> >>>>>>>>>> - [AIRFLOW-[3322](tel:3322)] Update QuboleHook to fetch args
> >>>> dynamically from
> >>>>>>>>> qds_sdk
> >>>>>>>>>> (#[4165](tel:4165))
> >>>>>>>>>> - [AIRFLOW-[4565](tel:4565)] instrument celery executor
> >>>> (#[5321](tel:5321))
> >>>>>>>>>> - [AIRFLOW-[4573](tel:4573)] Import airflow_local_settings after
> >>>>>>>>> prepare_classpath
> >>>>>>>>>> (#[5330](tel:5330))
> >>>>>>>>>> - [AIRFLOW-[4448](tel:4448)] Don't bake ENV and _cmd into tmp
> >>>> config for non-sudo
> >>>>>>>>>> (#[4050](tel:4050))
> >>>>>>>>>> - [AIRFLOW-[4295](tel:4295)] Make ``method`` attribute case
> >>>> insensitive in
> >>>>>>>>> HttpHook
> >>>>>>>>>> (#[5313](tel:5313))
> >>>>>>>>>> - [AIRFLOW-[3703](tel:3703)] Add dnsPolicy option for
> >>>> KubernetesPodOperator
> >>>>>>>>> (#[4520](tel:4520))
> >>>>>>>>>> - [AIRFLOW-[3057](tel:3057)] add prev_*_date_success to template
> >>>> context (#[5372](tel:5372))
> >>>>>>>>>> - [AIRFLOW-[4336](tel:4336)] Stop showing entire GCS files bytes
> >> in
> >>>> log for
> >>>>>>>>>> gcs_download_operator (#[5151](tel:5151))
> >>>>>>>>>> - [AIRFLOW-[4528](tel:4528)] Cancel DataProc task on timeout
> >>>> (#[5293](tel:5293))
> >>>>>>>>>>
> >>>>>>>>>> Bug fixes
> >>>>>>>>>> """""""""
> >>>>>>>>>> - [AIRFLOW-[4934](tel:4934)] Fix ProxyFix due to Werkzeug
> upgrade
> >>>> (#[5563](tel:5563)) (#[5571](tel:5571))
> >>>>>>>>>> - [AIRFLOW-[4136](tel:4136)] fix key_file of hook is overwritten
> >> by
> >>>> SSHHook
> >>>>>>>>> connection
> >>>>>>>>>> (#[5558](tel:5558))
> >>>>>>>>>> - [AIRFLOW-[4587](tel:4587)] Replace self.conn with
> >> self.get_conn()
> >>>> in
> >>>>>>>>> AWSAthenaHook
> >>>>>>>>>> (#[5545](tel:5545))
> >>>>>>>>>> - [AIRFLOW-[1740](tel:1740)] Fix xcom creation and update via UI
> >>>> (#[5530](tel:5530)) (#[5531](tel:5531))
> >>>>>>>>>> - [AIRFLOW-[4900](tel:4900)] Resolve incompatible version of
> >>>> Werkzeug (#[5535](tel:5535))
> >>>>>>>>>> - [AIRFLOW-[4510](tel:4510)] Don't mutate default_args during
> DAG
> >>>> initialization
> >>>>>>>>>> (#[5277](tel:5277))
> >>>>>>>>>> - [AIRFLOW-[3360](tel:3360)] Make the DAGs search respect other
> >>>> querystring
> >>>>>>>>> parameters
> >>>>>>>>>> with url-search-params-polyfill for IE support
> (#[5503](tel:5503))
> >>>>>>>>>> - [AIRFLOW-[4896](tel:4896)] Make KubernetesExecutorConfig's
> >>>> default args
> >>>>>>>>> immutable
> >>>>>>>>>> (#[5534](tel:5534))
> >>>>>>>>>> - [AIRFLOW-[4494](tel:4494)] Remove ``shell=True`` in
> DaskExecutor
> >>>> (#[5273](tel:5273))
> >>>>>>>>>> - [AIRFLOW-[4890](tel:4890)] Fix Log link in TaskInstance's View
> >>>> for Non-RBAC
> >>>>>>>>> (#[5525](tel:5525))
> >>>>>>>>>> - [AIRFLOW-[4892](tel:4892)] Fix connection creation via UIs
> >>>> (#[5527](tel:5527))
> >>>>>>>>>> - [AIRFLOW-[4406](tel:4406)] Fix a method name typo:
> >>>> NullFernet.decrpyt to
> >>>>>>>>> decrypt
> >>>>>>>>>> (#[5509](tel:5509))
> >>>>>>>>>> - [AIRFLOW-[4849](tel:4849)] Add gcp_conn_id to
> cloudsqldatabehook
> >>>> class to use
> >>>>>>>>>> correctly CloudSqlProxyRunner class (#[5478](tel:5478))
> >>>>>>>>>> - [AIRFLOW-[4769](tel:4769)] Pass gcp_conn_id to BigtableHook
> >>>> (#[5445](tel:5445))
> >>>>>>>>>> - [AIRFLOW-[4524](tel:4524)] Fix incorrect field names in view
> for
> >>>> Mark
> >>>>>>>>>> Success/Failure (#[5486](tel:5486))
> >>>>>>>>>> - [AIRFLOW-[3671](tel:3671)] Remove arg ``replace`` of
> >>>> MongoToS3Operator from
> >>>>>>>>>> ``kwargs`` (#[4480](tel:4480))
> >>>>>>>>>> - [AIRFLOW-[4845](tel:4845)] Fix bug where runAsUser 0 doesn't
> get
> >>>> set in k8s
> >>>>>>>>> security
> >>>>>>>>>> context (#[5474](tel:5474))
> >>>>>>>>>> - [AIRFLOW-[4354](tel:4354)] Fix exception in "between" date
> >> filter
> >>>> in classic UI
> >>>>>>>>>> (#[5480](tel:5480))
> >>>>>>>>>> - [AIRFLOW-[4587](tel:4587)] Replace self.conn with
> >> self.get_conn()
> >>>> in
> >>>>>>>>> AWSAthenaHook
> >>>>>>>>>> (#[5462](tel:5462))
> >>>>>>>>>> - [AIRFLOW-[4516](tel:4516)] K8s runAsUser and fsGroup cannot be
> >>>> strings (#[5429](tel:5429))
> >>>>>>>>>> - [AIRFLOW-[4298](tel:4298)] Stop Scheduler repeatedly warning
> >>>> "connection
> >>>>>>>>>> invalidated" (#[5470](tel:5470))
> >>>>>>>>>> - [AIRFLOW-[4559](tel:4559)] JenkinsJobTriggerOperator bugfix
> >>>> (#[5318](tel:5318))
> >>>>>>>>>> - [AIRFLOW-[4841](tel:4841)] Pin Sphinx AutoApi to
> >> [1.0.0](tel:100)
> >>>> (#[5468](tel:5468))
> >>>>>>>>>> - [AIRFLOW-[4479](tel:4479)] Include s3_overwrite kwarg in
> >>>> load_bytes method
> >>>>>>>>> (#[5312](tel:5312))
> >>>>>>>>>> - [AIRFLOW-[3746](tel:3746)] Fix DockerOperator missing
> container
> >>>> exit (#[4583](tel:4583))
> >>>>>>>>>> - [AIRFLOW-[4233](tel:4233)] Remove Template Extension from Bq
> to
> >>>> GCS Operator
> >>>>>>>>> (#[5456](tel:5456))
> >>>>>>>>>> -
> >>>>
> >>
> [AIRFLOW-[2141](tel:2141)][AIRFLOW-[3157](tel:3157)][AIRFLOW-[4170](tel:4170)]
> >>>> Serialize non-str value
> >>>>>>>>> by
> >>>>>>>>>> JSON when importing Variables (#[4991](tel:4991))
> >>>>>>>>>> - [AIRFLOW-[4826](tel:4826)] Remove warning from ``airflow
> >>>> resetdb`` command
> >>>>>>>>> (#[5447](tel:5447))
> >>>>>>>>>> - [AIRFLOW-[4148](tel:4148)] Fix editing DagRuns when clicking
> >>>> state column
> >>>>>>>>> (#[5436](tel:5436))
> >>>>>>>>>> - [AIRFLOW-[4455](tel:4455)] dag_details broken for subdags in
> >> RBAC
> >>>> UI (#[5234](tel:5234))
> >>>>>>>>>> - [AIRFLOW-[2955](tel:2955)] Fix kubernetes pod operator to set
> >>>> requests and
> >>>>>>>>> limits on
> >>>>>>>>>> task pods (#[4551](tel:4551))
> >>>>>>>>>> - [AIRFLOW-[4459](tel:4459)] Fix wrong DAG count in /home page
> >> when
> >>>> DAG count is
> >>>>>>>>> zero
> >>>>>>>>>> (#[5235](tel:5235))
> >>>>>>>>>> - [AIRFLOW-[3876](tel:3876)] AttributeError: module 'distutils'
> >> has
> >>>> no attribute
> >>>>>>>>> 'util'
> >>>>>>>>>> - [AIRFLOW-[4146](tel:4146)] Fix CgroupTaskRunner errors
> >>>> (#[5224](tel:5224))
> >>>>>>>>>> - [AIRFLOW-[4524](tel:4524)] Fix bug with "Ignore \*" toggles in
> >>>> RBAC mode
> >>>>>>>>> (#[5378](tel:5378))
> >>>>>>>>>> - [AIRFLOW-[4765](tel:4765)] Fix DataProcPigOperator execute
> >> method
> >>>> (#[5426](tel:5426))
> >>>>>>>>>> - [AIRFLOW-[4798](tel:4798)] obviate interdependencies for
> dagbag
> >>>> and TI tests
> >>>>>>>>> (#[5422](tel:5422))
> >>>>>>>>>> - [AIRFLOW-[4800](tel:4800)] fix GKEClusterHook ctor calls
> >>>> (#[5424](tel:5424))
> >>>>>>>>>> - [AIRFLOW-[4799](tel:4799)] don't mutate self.env in
> BashOperator
> >>>> execute method
> >>>>>>>>>> (#[5421](tel:5421))
> >>>>>>>>>> - [AIRFLOW-[4393](tel:4393)] Add retry logic when fetching pod
> >>>> status and/or logs
> >>>>>>>>> in
> >>>>>>>>>> KubernetesPodOperator (#[5284](tel:5284))
> >>>>>>>>>> - [AIRFLOW-[4174](tel:4174)] Fix HttpHook run with backoff
> >>>> (#[5213](tel:5213))
> >>>>>>>>>> - [AIRFLOW-[4463](tel:4463)] Handle divide-by-zero errors in
> short
> >>>> retry
> >>>>>>>>> intervals
> >>>>>>>>>> (#[5243](tel:5243))
> >>>>>>>>>> - [AIRFLOW-[2614](tel:2614)] Speed up trigger_dag API call when
> >>>> lots of DAGs in
> >>>>>>>>> system
> >>>>>>>>>> - [AIRFLOW-[4756](tel:4756)] add ti.state to ti.start_date as
> >>>> criteria for gantt
> >>>>>>>>>> (#[5399](tel:5399))
> >>>>>>>>>> - [AIRFLOW-[4760](tel:4760)] Fix zip-packaged DAGs disappearing
> >>>> from DagBag when
> >>>>>>>>>> reloaded (#[5404](tel:5404))
> >>>>>>>>>> - [AIRFLOW-[4731](tel:4731)] Fix GCS hook with
> >>>> google-storage-client [1.16](tel:116) (#[5368](tel:5368))
> >>>>>>>>>> - [AIRFLOW-[3506](tel:3506)] use match_phrase to query log_id in
> >>>> elasticsearch
> >>>>>>>>> (#[4342](tel:4342))
> >>>>>>>>>> - [AIRFLOW-[4084](tel:4084)] fix ElasticSearch log download
> >>>> (#[5177](tel:5177))
> >>>>>>>>>> - [AIRFLOW-[4501](tel:4501)] Register pendulum datetime
> converter
> >>>> for
> >>>>>>>>> sqla+pymysql
> >>>>>>>>>> (#[5190](tel:5190))
> >>>>>>>>>> - [AIRFLOW-[986](tel:986)] HiveCliHook ignores 'proxy_user'
> value
> >>>> in a
> >>>>>>>>> connection's
> >>>>>>>>>> extra parameter (#[5305](tel:5305))
> >>>>>>>>>> - [AIRFLOW-[4442](tel:4442)] fix hive_tblproperties in
> >>>> HiveToDruidTransfer
> >>>>>>>>> (#[5211](tel:5211))
> >>>>>>>>>> - [AIRFLOW-[4557](tel:4557)] Add gcp_conn_id parameter to
> >>>> get_sqlproxy_runner()
> >>>>>>>>> of
> >>>>>>>>>> CloudSqlDatabaseHook (#[5314](tel:5314))
> >>>>>>>>>> - [AIRFLOW-[4545](tel:4545)] Upgrade FAB to latest version
> >>>> (#[4955](tel:4955))
> >>>>>>>>>> - [AIRFLOW-[4492](tel:4492)] Change Dataproc Cluster operators
> to
> >>>> poll Operations
> >>>>>>>>>> (#[5269](tel:5269))
> >>>>>>>>>> - [AIRFLOW-[4452](tel:4452)] Webserver and Scheduler keep
> crashing
> >>>> because of
> >>>>>>>>>> slackclient update (#[5225](tel:5225))
> >>>>>>>>>> - [AIRFLOW-[4450](tel:4450)] Fix request arguments in
> >>>> has_dag_access (#[5220](tel:5220))
> >>>>>>>>>> - [AIRFLOW-[4434](tel:4434)] Support Impala with the
> >>>> HiveServer2Hook (#[5206](tel:5206))
> >>>>>>>>>> - [AIRFLOW-[3449](tel:3449)] Write local dag parsing logs when
> >>>> remote logging
> >>>>>>>>> enabled.
> >>>>>>>>>> (#[5175](tel:5175))
> >>>>>>>>>> - [AIRFLOW-[4300](tel:4300)] Fix graph modal call when DAG has
> not
> >>>> yet run
> >>>>>>>>> (#[5185](tel:5185))
> >>>>>>>>>> - [AIRFLOW-[4401](tel:4401)] Use managers for Queue
> >> synchronization
> >>>> (#[5200](tel:5200))
> >>>>>>>>>> - [AIRFLOW-[3626](tel:3626)] Fixed triggering DAGs contained
> >> within
> >>>> zip files
> >>>>>>>>> (#[4439](tel:4439))
> >>>>>>>>>> - [AIRFLOW-[3720](tel:3720)] Fix missmatch while comparing GCS
> and
> >>>> S3 files
> >>>>>>>>> (#[4766](tel:4766))
> >>>>>>>>>> - [AIRFLOW-[4403](tel:4403)] search by ``dag_id`` or ``owners``
> in
> >>>> UI (#[5184](tel:5184))
> >>>>>>>>>> - [AIRFLOW-[4308](tel:4308)] Fix TZ-loop around DST on python
> 3.6+
> >>>> (#[5095](tel:5095))
> >>>>>>>>>> - [AIRFLOW-[4324](tel:4324)] fix DAG fuzzy search in RBAC UI
> >>>> (#[5131](tel:5131))
> >>>>>>>>>> - [AIRFLOW-[4297](tel:4297)] Temporary hot fix on manage_slas()
> >> for
> >>>> [1.10.4](tel:1104)
> >>>>>>>>> release
> >>>>>>>>>> (#[5150](tel:5150))
> >>>>>>>>>> - [AIRFLOW-[4299](tel:4299)] Upgrade to Celery [4.3.0](tel:430)
> to
> >>>> fix crashing workers
> >>>>>>>>> (#[5116](tel:5116))
> >>>>>>>>>> - [AIRFLOW-[4291](tel:4291)] Correctly render doc_md in DAG
> graph
> >>>> page (#[5121](tel:5121))
> >>>>>>>>>> - [AIRFLOW-[4310](tel:4310)] Fix incorrect link on Dag Details
> >> page
> >>>> (#[5122](tel:5122))
> >>>>>>>>>> - [AIRFLOW-[4331](tel:4331)] Correct filter for Null-state runs
> >>>> from Dag Detail
> >>>>>>>>> page
> >>>>>>>>>> (#[5123](tel:5123))
> >>>>>>>>>> - [AIRFLOW-[4294](tel:4294)] Fix missing dag & task runs in UI
> >>>> dag_id contains a
> >>>>>>>>> dot
> >>>>>>>>>> (#[5111](tel:5111))
> >>>>>>>>>> - [AIRFLOW-[4332](tel:4332)] Upgrade sqlalchemy to remove
> security
> >>>> Vulnerability
> >>>>>>>>>> (#[5113](tel:5113))
> >>>>>>>>>> - [AIRFLOW-[4312](tel:4312)] Add template_fields & template_ext
> to
> >>>>>>>>> BigQueryCheckO…
> >>>>>>>>>> (#[5097](tel:5097))
> >>>>>>>>>> - [AIRFLOW-[4293](tel:4293)] Fix downgrade in
> >>>>>>>>>> d4ecb8fbee3_add_schedule_interval_to_dag.py (#[5086](tel:5086))
> >>>>>>>>>> - [AIRFLOW-[4267](tel:4267)] Fix TI duration in Graph View
> >>>> (#[5071](tel:5071))
> >>>>>>>>>> - [AIRFLOW-[4163](tel:4163)] IntervalCheckOperator supports
> >>>> relative diff and not
> >>>>>>>>>> ignore 0 (#[4983](tel:4983))
> >>>>>>>>>> - [AIRFLOW-[3938](tel:3938)] QuboleOperator Fixes and Support
> for
> >>>> SqlCommand
> >>>>>>>>> (#[4832](tel:4832))
> >>>>>>>>>> - [AIRFLOW-[2903](tel:2903)] Change default owner to ``airflow``
> >>>> (#[4151](tel:4151))
> >>>>>>>>>> - [aIRFLOW-[4136](tel:4136)] Fix overwrite of key_file by
> >>>> constructor (#[5155](tel:5155))
> >>>>>>>>>> - [AIRFLOW-[3241](tel:3241)] Remove Invalid template ext in GCS
> >>>> Sensors (#[4076](tel:4076))
> >>>>>>>>>>
> >>>>>>>>>> Misc/Internal
> >>>>>>>>>> """""""""""""
> >>>>>>>>>> - [AIRFLOW-[4937](tel:4937)] Fix lodash security issue with
> >> version
> >>>> below [4.17.13](tel:41713)
> >>>>>>>>>> (#[5572](tel:5572)) (used only in build-pipeline, not runtime)
> >>>>>>>>>> - [AIRFLOW-[4868](tel:4868)] Fix typo in
> >> kubernetes/docker/build.sh
> >>>> (#[5505](tel:5505))
> >>>>>>>>>> - [AIRFLOW-[4211](tel:4211)] Add tests for WebHDFSHook
> >>>> (#[5015](tel:5015))
> >>>>>>>>>> - [AIRFLOW-[4320](tel:4320)] Add tests for
> >>>> SegmentTrackEventOperator (#[5104](tel:5104))
> >>>>>>>>>> - [AIRFLOW-[4319](tel:4319)] Add tests for Bigquery related
> >>>> Operators (#[5101](tel:5101))
> >>>>>>>>>> - [AIRFLOW-[4014](tel:4014)] Change DatastoreHook and add tests
> >>>> (#[4842](tel:4842))
> >>>>>>>>>> - [AIRFLOW-[4322](tel:4322)] Add test for VerticaOperator
> >>>> (#[5107](tel:5107))
> >>>>>>>>>> - [AIRFLOW-[4323](tel:4323)] Add 2 tests for WinRMOperator
> >>>> (#[5108](tel:5108))
> >>>>>>>>>> - [AIRFLOW-[3677](tel:3677)] Improve CheckOperator test coverage
> >>>> (#[4756](tel:4756))
> >>>>>>>>>> - [AIRFLOW-[4659](tel:4659)] Fix pylint problems for api module
> >>>> (#[5398](tel:5398))
> >>>>>>>>>> - [AIRFLOW-[4358](tel:4358)] Speed up test_jobs by not running
> >>>> tasks (#[5162](tel:5162))
> >>>>>>>>>> - [AIRFLOW-[4394](tel:4394)] Don't test behaviour of BackfillJob
> >>>> from CLI tests
> >>>>>>>>> (#[5160](tel:5160))
> >>>>>>>>>> - [AIRFLOW-[3471](tel:3471)] Move XCom out of models.py
> >>>> (#[4629](tel:4629))
> >>>>>>>>>> - [AIRFLOW-[4379](tel:4379)] Remove duplicate code & Add
> >> validation
> >>>> in
> >>>>>>>>> gcs_to_gcs.py
> >>>>>>>>>> (#[5145](tel:5145))
> >>>>>>>>>> - [AIRFLOW-[4259](tel:4259)] Move models out of models.py
> >>>> (#[5056](tel:5056))
> >>>>>>>>>> - [AIRFLOW-XXX] Speed up building of Cassanda module on Travis
> >>>>>>>>> (#[5233](tel:5233))
> >>>>>>>>>> - [AIRFLOW-[4535](tel:4535)] Break jobs.py into multiple files
> >>>> (#[5303](tel:5303))
> >>>>>>>>>> - [AIRFLOW-[1464](tel:1464)] Batch update task_instance state
> >>>> (#[5323](tel:5323))
> >>>>>>>>>> - [AIRFLOW-[4554](tel:4554)] Test for sudo command, add some
> other
> >>>> test docs
> >>>>>>>>> (#[5310](tel:5310))
> >>>>>>>>>> - [AIRFLOW-[4419](tel:4419)] Refine concurrency check in
> scheduler
> >>>> (#[5194](tel:5194))
> >>>>>>>>>> - [AIRFLOW-[4269](tel:4269)] Minor acceleration of
> >>>> jobs._process_task_instances()
> >>>>>>>>>> (#[5076](tel:5076))
> >>>>>>>>>> - [AIRFLOW-[4341](tel:4341)] Remove ``View.render()`` already
> >>>> exists in
> >>>>>>>>> fab.BaseView
> >>>>>>>>>> (#[5125](tel:5125))
> >>>>>>>>>> - [AIRFLOW-[4342](tel:4342)] Use @cached_property instead of
> >>>> re-implementing it
> >>>>>>>>> each
> >>>>>>>>>> time (#[5126](tel:5126))
> >>>>>>>>>> - [AIRFLOW-[4256](tel:4256)] Remove noqa from migrations
> >>>> (#[5055](tel:5055))
> >>>>>>>>>> - [AIRFLOW-[4034](tel:4034)] Remove unnecessary string
> formatting
> >>>> with
> >>>>>>>>> ``**locals()``
> >>>>>>>>>> (#[4861](tel:4861))
> >>>>>>>>>> - [AIRFLOW-[3944](tel:3944)] Remove code smells
> >> (#[4762](tel:4762))
> >>>>>>>>>>
> >>>>>>>>>> Doc-only changes
> >>>>>>>>>> """"""""""""""""
> >>>>>>>>>> - [AIRFLOW-XXX] Correct BaseSensorOperator docs
> >> (#[5562](tel:5562))
> >>>>>>>>>> - [AIRFLOW-[4926](tel:4926)] Fix example dags where its
> start_date
> >>>> is
> >>>>>>>>>> datetime.utcnow() (#[5553](tel:5553))
> >>>>>>>>>> - [AIRFLOW-[4860](tel:4860)] Remove Redundant Information in
> >>>> Example Dags (#[5497](tel:5497))
> >>>>>>>>>> - [AIRFLOW-[4767](tel:4767)] Fix errors in the documentation of
> >>>> Dataproc Operator
> >>>>>>>>>> (#[5487](tel:5487))
> >>>>>>>>>> - [AIRFLOW-[1684](tel:1684)] Branching based on XCom variable
> >>>> (Docs) (#[4365](tel:4365))
> >>>>>>>>>> - [AIRFLOW-[3341](tel:3341)] FAQ return DAG object example
> >>>> (#[4605](tel:4605))
> >>>>>>>>>> - [AIRFLOW-[4433](tel:4433)] Add missing type in DockerOperator
> >> doc
> >>>> string
> >>>>>>>>> (#[5205](tel:5205))
> >>>>>>>>>> - [AIRFLOW-[4321](tel:4321)] Replace incorrect info of Max Size
> >>>> limit of GCS
> >>>>>>>>> Object
> >>>>>>>>>> Size (#[5106](tel:5106))
> >>>>>>>>>> - [AIRFLOW-XXX] Add information about user list
> >> (#[5341](tel:5341))
> >>>>>>>>>> - [AIRFLOW-XXX] Clarify documentation related to autodetect
> >>>> parameter
> >>>>>>>>> in
> >>>>>>>>>> GCS_to_BQ Op (#[5294](tel:5294))
> >>>>>>>>>> - [AIRFLOW-XXX] Remove mention of pytz compatibility from
> timezone
> >>>>>>>>>> documentation (#[5316](tel:5316))
> >>>>>>>>>> - [AIRFLOW-XXX] Add missing docs for
> >>>> GoogleCloudStorageDeleteOperator
> >>>>>>>>>> (#[5274](tel:5274))
> >>>>>>>>>> - [AIRFLOW-XXX] Remove incorrect note about Scopes of GCP
> >> connection
> >>>>>>>>>> (#[5242](tel:5242))
> >>>>>>>>>> - [AIRFLOW-XXX] Fix mistakes in docs of Dataproc operators
> >>>> (#[5192](tel:5192))
> >>>>>>>>>> - [AIRFLOW-XXX] Link to correct class for timedelta in
> macros.rst
> >>>>>>>>> (#[5226](tel:5226))
> >>>>>>>>>> - [AIRFLOW-XXX] Add Kamil as committer (#[5216](tel:5216))
> >>>>>>>>>> - [AIRFLOW-XXX] Add Joshua and Kevin as committer
> >>>> (#[5207](tel:5207))
> >>>>>>>>>> - [AIRFLOW-XXX] Reduce log spam in tests (#[5174](tel:5174))
> >>>>>>>>>> - [AIRFLOW-XXX] Speed up tests for PythonSensor
> >> (#[5158](tel:5158))
> >>>>>>>>>> - [AIRFLOW-XXX] Add Bas Harenslak to committer list
> >>>> (#[5157](tel:5157))
> >>>>>>>>>> - [AIRFLOW-XXX] Add Jarek Potiuk to commiter list
> >>>> (#[5132](tel:5132))
> >>>>>>>>>> - [AIRFLOW-XXX] Update docstring for SchedulerJob
> >>>> (#[5105](tel:5105))
> >>>>>>>>>> - [AIRFLOW-XXX] Fix docstrings for
> >>>>>>>>> CassandraToGoogleCloudStorageOperator
> >>>>>>>>>> (#[5103](tel:5103))
> >>>>>>>>>> - [AIRFLOW-XXX] update SlackWebhookHook and SlackWebhookOperator
> >>>>>>>>> docstring
> >>>>>>>>>> (#[5074](tel:5074))
> >>>>>>>>>> - [AIRFLOW-XXX] Ignore python files under node_modules in docs
> >>>>>>>>> (#[5063](tel:5063))
> >>>>>>>>>> - [AIRFLOW-XXX] Build a universal wheel with LICNESE files
> >>>> (#[5052](tel:5052))
> >>>>>>>>>> - [AIRFLOW-XXX] Fix docstrings of SQSHook (#[5099](tel:5099))
> >>>>>>>>>> - [AIRFLOW-XXX] Use Py3.7 on readthedocs
> >>>>>>>>>> - [AIRFLOW-[4446](tel:4446)] Fix typos (#[5217](tel:5217))
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best regards,
> >>>>>>
> >>>>>> Andrii Soldatenko
> >>>>>> Python Developer
> >>>>>>
> >>>>>> skype: andrii.soldatenko
> >>>>>> andrii.soldatenko@gmail.com
> >>>>>> http://asoldatenko.com
> >>>
> >>>
> >>>
> >>> --
> >>> *Kaxil Naik*
> >>> *Big Data Consultant | DevOps Data Engineer*
> >>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> Neo4j
> >>> Developer
> >>> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message