From dev-return-8895-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Wed Jul 17 13:25:45 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8DC6718060E for ; Wed, 17 Jul 2019 15:25:44 +0200 (CEST) Received: (qmail 25616 invoked by uid 500); 17 Jul 2019 13:25:42 -0000 Mailing-List: contact dev-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list dev@airflow.apache.org Received: (qmail 25604 invoked by uid 99); 17 Jul 2019 13:25:42 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jul 2019 13:25:42 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id A76191803AF for ; Wed, 17 Jul 2019 13:25:41 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.302 X-Spam-Level: ** X-Spam-Status: No, score=2.302 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_QUOTING=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id DgLVbOQiKO1s for ; Wed, 17 Jul 2019 13:25:36 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::331; helo=mail-ot1-x331.google.com; envelope-from=andrii.soldatenko@gmail.com; receiver= Received: from mail-ot1-x331.google.com (mail-ot1-x331.google.com [IPv6:2607:f8b0:4864:20::331]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id EC9587E221 for ; Wed, 17 Jul 2019 13:25:35 +0000 (UTC) Received: by mail-ot1-x331.google.com with SMTP id x21so24951317otq.12 for ; Wed, 17 Jul 2019 06:25:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=uQr3/1yhCIf2szgvr7EswXIYO+uT1G+J8BzTgGQs2qU=; b=r2h7Z/+zvMPCx5vNlRaTk4Y620y99L7o9UdS0FbBIGYR+YT8EhnruCBxKkHYgBc3/S Ux2TyZ5uctMNmGsuWiWyg4xxzyaF/OJVjnHpwJuKaew3DIdmIjFTDu/13kWAzQ9pwJZe xLT0QJxlitLNVpr0cGaiY0+3B422Cx+47nk+mNRb2RJXfppeIDr3KARMJXAvzNLHt/Bz gxVvq8mo0xLM4CZQaL6Bb1beWwPvwWVuZ5WlWxr8R0h1f93DT9L7jWDT2r5Imub73RLt DuP6ubxctOJTeKSwNLddM9XrtZxsSiQeC+l8xBmhKn3UpBkT0bd+oYxSwFxzx367FTFc 3MUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=uQr3/1yhCIf2szgvr7EswXIYO+uT1G+J8BzTgGQs2qU=; b=iCkvcYiaIcm3uyKNPKCIuqp0ACmA1o8BK9odDynWRYO4LcwzDfuHExrzaJPzAdgxzE YvuPOhvdwvoESmDb3WZ/VTaHLcsHqXiJG8sg9irbOPEUOKNbx4WWWnDwcRWUkEB4P6Yj 7d955qTetbFegxy4I4p0+YEhCNssj+MJ1gU/gK+o4VDshjy/4hEqVJX9/1O6PeKbxqWw +09jdsaVm398xqIZGMBMXbyGpW1Y/TTgbbGv2yNamdj/71xoz7HaXngds/J6NLlxmPdx S5utK4qvVGLp3CAXGAY7XkbWV/WhbTZpNLsjWeKBljgO21DPXDFcQB47zBjnGpdGKvJ6 vTlQ== X-Gm-Message-State: APjAAAVoQXEeLyblKm34wC2WFeSVfTh+tFVvCaounciP1afNhtokFOJu QNEgcW0/JwmqKZso7winF0126YoFJb57ytICGL6GVUII X-Google-Smtp-Source: APXvYqyTu+yxapzfaFc1pNUN7yt77tDkvS4UvU6V5c44su5N/NpBpZtq0kB1Ko+PTbQJvBSjDPpikTthijaZtRwEqmw= X-Received: by 2002:a9d:76ce:: with SMTP id p14mr8663901otl.342.1563369934124; Wed, 17 Jul 2019 06:25:34 -0700 (PDT) MIME-Version: 1.0 References: <8FF14302-46AD-4B30-9D5D-D88C34FA5667@apache.org> In-Reply-To: From: Andrii Soldatenko Date: Wed, 17 Jul 2019 16:25:22 +0300 Message-ID: Subject: Re: [VOTE] Release Airflow 1.10.4 from RC3 To: dev@airflow.apache.org Cc: James Meickle Content-Type: multipart/alternative; boundary="0000000000009379e7058de06d80" --0000000000009379e7058de06d80 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable @Ash, i'll fix wrong section bug. Sorry about that. On Wed, Jul 17, 2019 at 4:11 PM Robin Edwards wrote: > +1 (none binding) - been running in production since RC2. > > Thanks for all your hard work > > R > > On Tue, 16 Jul 2019 at 21:15, Ash Berlin-Taylor wrote: > > > > Thanks for testing. > > > > On 1) everyone should run upgradedb on every upgrade. The behaviour of > not running it wasn't great. > > > > 2) I thought we set deprecation on the ES logging config vars, except w= e > put the depreciation under the wrong section: > https://github.com/apache/airflow/blob/1.10.4rc3/airflow/configuration.py= #L147-L153 > :( > > > > On 16 July 2019 21:06:40 BST, James Meickle > wrote: > > >+1 (nonbinding) to the release, it fixes a lot of UI issues we've been > > >seeing lately. Though two notes: > > > > > >1) Tasks were unscheduleable until I ran an upgradedb due the default > > >pool > > >change. > > > > > >2) I got crash loops because I based our custom logging file off of th= e > > >previous version's template. The change to the Elasticsearch conf > > >variables > > >broke this, even though we don't directly use Elasticsearch, since it > > >was > > >now looking for undefined conf variables. It was confusing until I > > >hunted > > >around the diff and backported the changes in the default template to > > >the > > >template that lives in our repo. > > > > > >On Mon, Jul 15, 2019 at 10:17 AM Ash Berlin-Taylor > > >wrote: > > > > > >> Hello Airflow community, > > >> > > >> This email is calling a vote on the release, which will last for 72 > > >hours > > >> (2019-07-08 15:15 Z), and until three binding votes have been cast. > > >> Consider this my (binding) +1. > > >> > > >> Airflow 1.10.4 RC3 is available at: > > >> https://dist.apache.org/repos/dist/dev/airflow/1.10.4rc3/ > > >> > > >> *apache-airflow-1.10.4rc3-source.tar.gz* is a source release that > > >comes > > >> with INSTALL instructions. > > >> *apache-airflow-1.10.4rc3-bin.tar.gz* is the binary Python "sdist" > > >release. > > >> *apache_airflow-1.10.4rc3-py2.py3-none-any.whl* is the binary Python > > >> "wheel" release. > > >> > > >> For connivence of testers the RC is on PYPI too. It can be installed > > >with: > > >> > > >> pip install 'apache-airflow=3D=3D1.10.4rc3' > > >> > > >> Public keys are available at: > > >> https://dist.apache.org/repos/dist/release/airflow/KEYS > > >> > > >> Only votes from PMC members are binding (sorry committers), but > > >members of > > >> the community are encouraged to test the release and vote with > > >> "(non-binding)". > > >> > > >> Please note that the version number excludes the `rcX` string, so > > >it's now > > >> simply 1.10.4. This will allow us to rename the artefact without > > >modifying > > >> the checksums when we actually release. > > >> > > >> [ ] +1 Release this package as Apache Airflow 1.10.4 > > >> [ ] 0 No opinion > > >> [ ] -1 Do not release this package because... > > >> > > >> Changes since RC2: > > >> > > >> [AIRFLOW-XXX] Update changelog for 1.10.4rc3 [Ash Berlin-Taylor] > > >> [AIRFLOW-XXX] Disable intersphinx loading of `requests` modules > > >(#5590) > > >> [Ash Berlin-Taylor] > > >> [AIRFLOW-4797] Improve performance and behaviour of zombie detection > > >> (#5511) [Stefan Seelmann] > > >> [AIRFLOW-4911] Silence the FORBIDDEN errors from the > > >KubernetesExecutor > > >> (#5547) [Andrii Soldatenko] > > >> [AIRFLOW-4884] Roll up import_errors in RBAC UI (#5516) > > >[shuwen-zhang] > > >> [AIRFLOW-4937] Fix lodash security issue with version below 4.17.13 > > >> (#5572) [Jarek Potiuk] > > >> [AIRFLOW-4934] Bump Flask to resolve Werkzeug ProxyFix (#5571) > > >[Nathaniel > > >> Ritholtz] > > >> [AIRFLOW-4934] Fix ProxyFix due to Werkzeug upgrade (#5563) > > >[Nathaniel > > >> Ritholtz] > > >> [AIRFLOW-4136] fix key_file of hook is overwritten by SSHHook > > >connection > > >> (#5558) [eladkal] > > >> [AIRFLOW-XXX] Correct BaseSensorOperator docs (#5562) [Matthew > > >Bowden] > > >> [AIRFLOW-3495] Validate one of query and query_uri passed to > > >> DataProcSparkSqlOperator (#5510) [eladkal] > > >> [AIRFLOW-4868] Fix typo in kubernetes/docker/build.sh (#5505) [Facu] > > >> [AIRFLOW-4925] Improve css style for Variables Import file field > > >(#5552) > > >> [Felix Uellendall] > > >> [AIRFLOW-4906] Improve debugging for the SparkSubmitHook (#5542) > > >[Fokko > > >> Driesprong] > > >> [AIRFLOW-4904] Retrieve test config file from $AIRFLOW_TEST_CONFIG > > >(#5540) > > >> [Matthew Rastovac] > > >> [AIRFLOW-4926] Fix example dags where its start_date is > > >datetime.utcnow() > > >> (#5553) [Felix Uellendall] > > >> [AIRFLOW-4587] Replace self.conn with self.get_conn() in > > >AWSAthenaHook > > >> (#5545) [Bhavika Tekwani] > > >> [AIRFLOW-1740] Fix xcom creation and update via UI (#5530) [Felix > > >> Uellendall] > > >> [AIRFLOW-1740] Fix xcom creation and update via RBAC UI (#5561) > > >[Felix > > >> Uellendall] > > >> [AIRFLOW-4920] Use html.escape instead of cgi.escape to fix > > >> DeprecationWarning (#5551) [Xtreak] > > >> [AIRFLOW-4919] DataProcJobBaseOperator dataproc_*_properties > > >templated > > >> (#5555) [eladkal] > > >> [AIRFLOW-4900] Resolve incompatible version of Werkzeug (#5535) > > >[Luqman > > >> Abdul Mushawwir] > > >> [AIRFLOW-4478] Lazily instantiate default resources objects. (#5259) > > >> [Joshua Carp] > > >> [AIRFLOW-4510] Don't mutate default_args during DAG initialization > > >(#5277) > > >> [Abhishek Ray] > > >> [AIRFLOW-3360] Make the DAGs search respect other querystring > > >parameters > > >> with url-search-params-polyfill for IE support (#5503) [Alex] > > >> [AIRFLOW-4896] Make KubernetesExecutorConfig's default args immutabl= e > > >> (#5534) [Philippe Gagnon] > > >> [AIRFLOW-4494] Remove `shell=3DTrue` in DaskExecutor (#5273) [MisLin= k] > > >> [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC > > >(#5525) > > >> [Felix Uellendall] > > >> [AIRFLOW-4892] Fix connection creation via UIs (#5527) [Ash > > >Berlin-Taylor] > > >> [AIRFLOW-4862] Fix bug for earlier change to allow using IP as > > >hostname > > >> (#5513) [Xiaodong] > > >> > > >> > > >> > > >> Full Changelog since 1.10.3: > > >> > > >> Airflow 1.10.4, - 2019-07-18 > > >> ---------------------------- > > >> > > >> New Features > > >> """""""""""" > > >> - [AIRFLOW-4884] Roll up import_errors in RBAC UI (#5516) > > >> - [AIRFLOW-4871] Allow creating DagRuns via RBAC UI (#5507) > > >> - [AIRFLOW-4591] Make default_pool a real pool (#5349) > > >> - [AIRFLOW-4844] Add optional is_paused_upon_creation argument to DA= G > > >> (#5473) > > >> - [AIRFLOW-4456] Add sub-classable BaseBranchOperator (#5231) > > >> - [AIRFLOW-4343] Show warning in UI if scheduler is not running > > >(#5127) > > >> - [AIRFLOW-4739] Add ability to arbitrarily define kubernetes worker > > >pod > > >> labels (#5376) > > >> - [AIRFLOW-4348] Add GCP console link in BigQueryOperator (#5195) > > >> - [AIRFLOW-4306] Global operator extra links (#5094) > > >> - [AIRFLOW-4812] Add batch images annotation (#5433) > > >> - [AIRFLOW-4135] Add Google Cloud Build operator and hook (#5251) > > >> - [AIRFLOW-4781] Add the ability to specify ports in > > >kubernetesOperator > > >> (#5410) > > >> - [AIRFLOW-4521] Pause dag also pause its subdags (#5283) > > >> - [AIRFLOW-4738] Enforce exampleinclude for example DAGs (#5375) > > >> - [AIRFLOW-4326] Airflow AWS SQS Operator (#5110) > > >> - [AIRFLOW-3729] Support "DownwardAPI" in env variables for > > >> KubernetesPodOperator (#4554) > > >> - [AIRFLOW-4585] Implement Kubernetes Pod Mutation Hook (#5359) > > >> - [AIRFLOW-161] New redirect route and extra links (#5059) > > >> - [AIRFLOW-4420] Backfill respects task_concurrency (#5221) > > >> - [AIRFLOW-4147] Add Operator to publish event to Redis (#4967) > > >> - [AIRFLOW-3359] Add option to pass customer encryption keys to > > >Dataproc > > >> (#4200) > > >> - [AIRFLOW-4318] Create Google Cloud Translate Speech Operator > > >(#5102) > > >> - [AIRFLOW-3960] Adds Google Cloud Speech operators (#4780) > > >> - [AIRFLOW-1501] Add GoogleCloudStorageDeleteOperator (#5230) > > >> - [AIRFLOW-3672] Add support for Mongo DB DNS Seedlist Connection > > >Format > > >> (#4481) > > >> - [AIRFLOW-4397] add integrations docs manually for gcs sensors > > >(#5204) > > >> - [AIRFLOW-4251] Instrument DagRun schedule delay (#5050) > > >> - [AIRFLOW-4118] instrument DagRun duration (#4946) > > >> - [AIRFLOW-4361] Fix flaky > > >test_integration_run_dag_with_scheduler_failure > > >> (#5182) > > >> - [AIRFLOW-4361] Fix flaky > > >test_integration_run_dag_with_scheduler_failure > > >> (#5140) > > >> - [AIRFLOW-4168] Create Google Cloud Video Intelligence Operators > > >(#4985) > > >> - [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) > > >> - [AIRFLOW-4335] Add default num_retries to GCP connection (#5117) > > >> - [AIRFLOW-3808] Add cluster_fields to BigQueryHook's > > >create_empty_table > > >> (#4654) > > >> - [AIRFLOW-4362] Fix test_execution_limited_parallelism (#5141) > > >> - [AIRFLOW-4307] Backfill respects concurrency limit (#5128) > > >> - [AIRFLOW-4268] Add MsSqlToGoogleCloudStorageOperator (#5077) > > >> - [AIRFLOW-4169] Add Google Cloud Vision Detect Operators (#4986) > > >> - [AIRFLOW-XXX] Fix WS-2019-0032 (#5384) > > >> - [AIRFLOW-XXX] Fix CVE-2019-11358 (#5197) > > >> - [AIRFLOW-XXX] Change allowed version of Jinja2 to fix > > >CVE-2019-10906 > > >> (#5075) > > >> > > >> Improvement > > >> """"""""""" > > >> - [AIRFLOW-4797] Improve performance and behaviour of zombie > > >detection > > >> (#5511) > > >> - [AIRFLOW-4911] Silence the FORBIDDEN errors from the > > >KubernetesExecutor > > >> (#5547) > > >> - [AIRFLOW-3495] Validate one of query and query_uri passed to > > >> DataProcSparkSqlOperator (#5510) > > >> - [AIRFLOW-4925] Improve css style for Variables Import file field > > >(#5552) > > >> - [AIRFLOW-4906] Improve debugging for the SparkSubmitHook (#5542) > > >> - [AIRFLOW-4904] unittest.cfg name and path can be overriden by > > >setting > > >> $AIRFLOW_TEST_CONFIG (#5540) > > >> - [AIRFLOW-4920] Use html.escape instead of cgi.escape to fix > > >> DeprecationWarning (#5551) > > >> - [AIRFLOW-4919] DataProcJobBaseOperator dataproc_*_properties > > >templated > > >> (#5555) > > >> - [AIRFLOW-4478] Lazily instantiate default resources objects. > > >(#5259) > > >> - [AIRFLOW-4564] AzureContainerInstance bugfixes and improvements > > >(#5319) > > >> - [AIRFLOW-4237] Including Try Number of Task in Gantt Chart (#5037) > > >> - [AIRFLOW-4862] Allow directly using IP address as hostname for > > >webserver > > >> logs (#5501) > > >> - [AIRFLOW-4857] Add templated fields to SlackWebhookOperator (#5490= ) > > >> - [AIRFLOW-3502] Add celery config option for setting "pool" (#4308) > > >> - [AIRFLOW-3217] Button to toggle line wrapping in log and code view= s > > >> (#4277) > > >> - [AIRFLOW-4491] Add a "Jump to end" button for logs (#5266) > > >> - [AIRFLOW-4422] Pool utilization stats (#5453) > > >> - [AIRFLOW-4805] Add py_file as templated field in > > >DataflowPythonOperator > > >> (#5451) > > >> - [AIRFLOW-4838] Surface Athena errors in AWSAthenaOperator (#5467) > > >> - [AIRFLOW-4831] conf.has_option no longer throws if section is > > >missing. > > >> (#5455) > > >> - [AIRFLOW-4829] More descriptive exceptions for EMR sensors (#5452) > > >> - [AIRFLOW-4414] AWSAthenaOperator: Push QueryExecutionID to XCom > > >(#5276) > > >> - [AIRFLOW-4791] add "schema" keyword arg to SnowflakeOperator > > >(#5415) > > >> - [AIRFLOW-4759] Don't error when marking sucessful run as failed > > >(#5435) > > >> - [AIRFLOW-4716] Instrument dag loading time duration (#5350) > > >> - [AIRFLOW-3958] Support list tasks as upstream in chain (#4779) > > >> - [AIRFLOW-4409] Prevent task duration break by null value (#5178) > > >> - [AIRFLOW-4418] Add "failed only" option to task modal (#5193) > > >> - [AIRFLOW-4740] Accept string ``end_date`` in DAG default_args > > >(#5381) > > >> - [AIRFLOW-4423] Improve date handling in mysql to gcs operator. > > >(#5196) > > >> - [AIRFLOW-4447] Display task duration as human friendly format in U= I > > >> (#5218) > > >> - [AIRFLOW-4377] Remove needless object conversion in DAG.owner() > > >(#5144) > > >> - [AIRFLOW-4766] Add autoscaling option for > > >DataprocClusterCreateOperator > > >> (#5425) > > >> - [AIRFLOW-4795] Upgrade alembic to latest release. (#5411) > > >> - [AIRFLOW-4793] Add signature_name to mlengine operator (#5417) > > >> - [AIRFLOW-3211] Reattach to GCP Dataproc jobs upon Airflow restart > > >> (#4083) > > >> - [AIRFLOW-4750] Log identified zombie task instances (#5389) > > >> - [AIRFLOW-3870] STFPOperator: Update log level and return value > > >(#4355) > > >> - [AIRFLOW-4759] Batch queries in set_state API. (#5403) > > >> - [AIRFLOW-2737] Restore original license header to > > >> airflow.api.auth.backend.kerberos_auth > > >> - [AIRFLOW-3635] Fix incorrect logic in detele_dag (introduced in > > >PR#4406) > > >> (#4445) > > >> - [AIRFLOW-3599] Removed Dagbag from delete dag (#4406) > > >> - [AIRFLOW-4737] Increase and document celery queue name limit > > >(#5383) > > >> - [AIRFLOW-4505] Correct Tag ALL for PY3 (#5275) > > >> - [AIRFLOW-4743] Add environment variables support to SSHOperator > > >(#5385) > > >> - [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency > > >> (#5363) > > >> - [AIRFLOW-3370] Add stdout output options to Elasticsearch task log > > >> handler (#5048) > > >> - [AIRFLOW-4396] Provide a link to external Elasticsearch logs in UI= . > > >> (#5164) > > >> - [AIRFLOW-1381] Allow setting host temporary directory in > > >DockerOperator > > >> (#5369) > > >> - [AIRFLOW-4598] Task retries are not exhausted for K8s executor > > >(#5347) > > >> - [AIRFLOW-4218] Support to Provide http args to K8executor while > > >calling > > >> k8 python client lib apis (#5060) > > >> - [AIRFLOW-4159] Add support for additional static pod labels for > > >> K8sExecutor (#5134) > > >> - [AIRFLOW-4720] Allow comments in .airflowignore files. (#5355) > > >> - [AIRFLOW-4486] Add AWS IAM authenication in MySqlHook (#5334) > > >> - [AIRFLOW-4417] Add AWS IAM authenication for PostgresHook (#5223) > > >> - [AIRFLOW-3990] Compile regular expressions. (#4813) > > >> - [AIRFLOW-4572] Rename prepare_classpath() to prepare_syspath() > > >(#5328) > > >> - [AIRFLOW-3869] Raise consistent exception in > > >> AirflowConfigParser.getboolean (#4692) > > >> - [AIRFLOW-4571] Add headers to templated field for > > >SimpleHttpOperator > > >> (#5326) > > >> - [AIRFLOW-3867] Rename GCP's subpackage (#4690) > > >> - [AIRFLOW-3725] Add private_key to bigquery_hook get_pandas_df > > >(#4549) > > >> - [AIRFLOW-4546] Upgrade google-cloud-bigtable. (#5307) > > >> - [AIRFLOW-4519] Optimise operator classname sorting in views (#5282= ) > > >> - [AIRFLOW-4503] Support fully pig options (#5271) > > >> - [AIRFLOW-4468] add sql_alchemy_max_overflow parameter (#5249) > > >> - [AIRFLOW-4467] Add dataproc_jars to templated fields in Dataproc > > >oper=E2=80=A6 > > >> (#5248) > > >> - [AIRFLOW-4381] Use get_direct_relative_ids get task relatives > > >(#5147) > > >> - [AIRFLOW-3624] Add masterType parameter to MLEngineTrainingOperato= r > > >> (#4428) > > >> - [AIRFLOW-3143] Support Auto-Zone in DataprocClusterCreateOperator > > >(#5169) > > >> - [AIRFLOW-3874] Improve BigQueryHook.run_with_configuration's > > >location > > >> support (#4695) > > >> - [AIRFLOW-4399] Avoid duplicated os.path.isfile() check in > > >models.dagbag > > >> (#5165) > > >> - [AIRFLOW-4031] Allow for key pair auth in snowflake hook (#4875) > > >> - [AIRFLOW-3901] add role as optional config parameter for > > >SnowflakeHook > > >> (#4721) > > >> - [AIRFLOW-3455] add region in snowflake connector (#4285) > > >> - [AIRFLOW-4073] add template_ext for AWS Athena operator (#4907) > > >> - [AIRFLOW-4093] AWSAthenaOperator: Throw exception if job > > >> failed/cancelled/reach max retries (#4919) > > >> - [AIRFLOW-4356] Add extra RuntimeEnvironment keys to DataFlowHook > > >(#5149) > > >> - [AIRFLOW-4337] Fix docker-compose deprecation warning in CI (#5119= ) > > >> - [AIRFLOW-3603] QuboleOperator: Remove SQLCommand from SparkCmd > > >> documentation (#4411) > > >> - [AIRFLOW-4328] Fix link to task instances from Pool page (#5124) > > >> - [AIRFLOW-4255] Make GCS Hook Backwards compatible (#5089) > > >> - [AIRFLOW-4103] Allow uppercase letters in dataflow job names > > >(#4925) > > >> - [AIRFLOW-4255] Replace Discovery based api with client based for > > >GCS > > >> (#5054) > > >> - [AIRFLOW-4311] Remove sleep in localexecutor (#5096) > > >> - [AIRFLOW-2836] Minor improvement-contrib.sensors.FileSensor (#3674= ) > > >> - [AIRFLOW-4104] Add type annotations to common classes. (#4926) > > >> - [AIRFLOW-3910] Raise exception explicitly in Connection.get_hook() > > >> (#4728) > > >> - [AIRFLOW-3322] Update QuboleHook to fetch args dynamically from > > >qds_sdk > > >> (#4165) > > >> - [AIRFLOW-4565] instrument celery executor (#5321) > > >> - [AIRFLOW-4573] Import airflow_local_settings after > > >prepare_classpath > > >> (#5330) > > >> - [AIRFLOW-4448] Don't bake ENV and _cmd into tmp config for non-sud= o > > >> (#4050) > > >> - [AIRFLOW-4295] Make ``method`` attribute case insensitive in > > >HttpHook > > >> (#5313) > > >> - [AIRFLOW-3703] Add dnsPolicy option for KubernetesPodOperator > > >(#4520) > > >> - [AIRFLOW-3057] add prev_*_date_success to template context (#5372) > > >> - [AIRFLOW-4336] Stop showing entire GCS files bytes in log for > > >> gcs_download_operator (#5151) > > >> - [AIRFLOW-4528] Cancel DataProc task on timeout (#5293) > > >> > > >> Bug fixes > > >> """"""""" > > >> - [AIRFLOW-4934] Fix ProxyFix due to Werkzeug upgrade (#5563) (#5571= ) > > >> - [AIRFLOW-4136] fix key_file of hook is overwritten by SSHHook > > >connection > > >> (#5558) > > >> - [AIRFLOW-4587] Replace self.conn with self.get_conn() in > > >AWSAthenaHook > > >> (#5545) > > >> - [AIRFLOW-1740] Fix xcom creation and update via UI (#5530) (#5531) > > >> - [AIRFLOW-4900] Resolve incompatible version of Werkzeug (#5535) > > >> - [AIRFLOW-4510] Don't mutate default_args during DAG initialization > > >> (#5277) > > >> - [AIRFLOW-3360] Make the DAGs search respect other querystring > > >parameters > > >> with url-search-params-polyfill for IE support (#5503) > > >> - [AIRFLOW-4896] Make KubernetesExecutorConfig's default args > > >immutable > > >> (#5534) > > >> - [AIRFLOW-4494] Remove ``shell=3DTrue`` in DaskExecutor (#5273) > > >> - [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC > > >(#5525) > > >> - [AIRFLOW-4892] Fix connection creation via UIs (#5527) > > >> - [AIRFLOW-4406] Fix a method name typo: NullFernet.decrpyt to > > >decrypt > > >> (#5509) > > >> - [AIRFLOW-4849] Add gcp_conn_id to cloudsqldatabehook class to use > > >> correctly CloudSqlProxyRunner class (#5478) > > >> - [AIRFLOW-4769] Pass gcp_conn_id to BigtableHook (#5445) > > >> - [AIRFLOW-4524] Fix incorrect field names in view for Mark > > >> Success/Failure (#5486) > > >> - [AIRFLOW-3671] Remove arg ``replace`` of MongoToS3Operator from > > >> ``kwargs`` (#4480) > > >> - [AIRFLOW-4845] Fix bug where runAsUser 0 doesn't get set in k8s > > >security > > >> context (#5474) > > >> - [AIRFLOW-4354] Fix exception in "between" date filter in classic U= I > > >> (#5480) > > >> - [AIRFLOW-4587] Replace self.conn with self.get_conn() in > > >AWSAthenaHook > > >> (#5462) > > >> - [AIRFLOW-4516] K8s runAsUser and fsGroup cannot be strings (#5429) > > >> - [AIRFLOW-4298] Stop Scheduler repeatedly warning "connection > > >> invalidated" (#5470) > > >> - [AIRFLOW-4559] JenkinsJobTriggerOperator bugfix (#5318) > > >> - [AIRFLOW-4841] Pin Sphinx AutoApi to 1.0.0 (#5468) > > >> - [AIRFLOW-4479] Include s3_overwrite kwarg in load_bytes method > > >(#5312) > > >> - [AIRFLOW-3746] Fix DockerOperator missing container exit (#4583) > > >> - [AIRFLOW-4233] Remove Template Extension from Bq to GCS Operator > > >(#5456) > > >> - [AIRFLOW-2141][AIRFLOW-3157][AIRFLOW-4170] Serialize non-str value > > >by > > >> JSON when importing Variables (#4991) > > >> - [AIRFLOW-4826] Remove warning from ``airflow resetdb`` command > > >(#5447) > > >> - [AIRFLOW-4148] Fix editing DagRuns when clicking state column > > >(#5436) > > >> - [AIRFLOW-4455] dag_details broken for subdags in RBAC UI (#5234) > > >> - [AIRFLOW-2955] Fix kubernetes pod operator to set requests and > > >limits on > > >> task pods (#4551) > > >> - [AIRFLOW-4459] Fix wrong DAG count in /home page when DAG count is > > >zero > > >> (#5235) > > >> - [AIRFLOW-3876] AttributeError: module 'distutils' has no attribute > > >'util' > > >> - [AIRFLOW-4146] Fix CgroupTaskRunner errors (#5224) > > >> - [AIRFLOW-4524] Fix bug with "Ignore \*" toggles in RBAC mode > > >(#5378) > > >> - [AIRFLOW-4765] Fix DataProcPigOperator execute method (#5426) > > >> - [AIRFLOW-4798] obviate interdependencies for dagbag and TI tests > > >(#5422) > > >> - [AIRFLOW-4800] fix GKEClusterHook ctor calls (#5424) > > >> - [AIRFLOW-4799] don't mutate self.env in BashOperator execute metho= d > > >> (#5421) > > >> - [AIRFLOW-4393] Add retry logic when fetching pod status and/or log= s > > >in > > >> KubernetesPodOperator (#5284) > > >> - [AIRFLOW-4174] Fix HttpHook run with backoff (#5213) > > >> - [AIRFLOW-4463] Handle divide-by-zero errors in short retry > > >intervals > > >> (#5243) > > >> - [AIRFLOW-2614] Speed up trigger_dag API call when lots of DAGs in > > >system > > >> - [AIRFLOW-4756] add ti.state to ti.start_date as criteria for gantt > > >> (#5399) > > >> - [AIRFLOW-4760] Fix zip-packaged DAGs disappearing from DagBag when > > >> reloaded (#5404) > > >> - [AIRFLOW-4731] Fix GCS hook with google-storage-client 1.16 (#5368= ) > > >> - [AIRFLOW-3506] use match_phrase to query log_id in elasticsearch > > >(#4342) > > >> - [AIRFLOW-4084] fix ElasticSearch log download (#5177) > > >> - [AIRFLOW-4501] Register pendulum datetime converter for > > >sqla+pymysql > > >> (#5190) > > >> - [AIRFLOW-986] HiveCliHook ignores 'proxy_user' value in a > > >connection's > > >> extra parameter (#5305) > > >> - [AIRFLOW-4442] fix hive_tblproperties in HiveToDruidTransfer > > >(#5211) > > >> - [AIRFLOW-4557] Add gcp_conn_id parameter to get_sqlproxy_runner() > > >of > > >> CloudSqlDatabaseHook (#5314) > > >> - [AIRFLOW-4545] Upgrade FAB to latest version (#4955) > > >> - [AIRFLOW-4492] Change Dataproc Cluster operators to poll Operation= s > > >> (#5269) > > >> - [AIRFLOW-4452] Webserver and Scheduler keep crashing because of > > >> slackclient update (#5225) > > >> - [AIRFLOW-4450] Fix request arguments in has_dag_access (#5220) > > >> - [AIRFLOW-4434] Support Impala with the HiveServer2Hook (#5206) > > >> - [AIRFLOW-3449] Write local dag parsing logs when remote logging > > >enabled. > > >> (#5175) > > >> - [AIRFLOW-4300] Fix graph modal call when DAG has not yet run > > >(#5185) > > >> - [AIRFLOW-4401] Use managers for Queue synchronization (#5200) > > >> - [AIRFLOW-3626] Fixed triggering DAGs contained within zip files > > >(#4439) > > >> - [AIRFLOW-3720] Fix missmatch while comparing GCS and S3 files > > >(#4766) > > >> - [AIRFLOW-4403] search by ``dag_id`` or ``owners`` in UI (#5184) > > >> - [AIRFLOW-4308] Fix TZ-loop around DST on python 3.6+ (#5095) > > >> - [AIRFLOW-4324] fix DAG fuzzy search in RBAC UI (#5131) > > >> - [AIRFLOW-4297] Temporary hot fix on manage_slas() for 1.10.4 > > >release > > >> (#5150) > > >> - [AIRFLOW-4299] Upgrade to Celery 4.3.0 to fix crashing workers > > >(#5116) > > >> - [AIRFLOW-4291] Correctly render doc_md in DAG graph page (#5121) > > >> - [AIRFLOW-4310] Fix incorrect link on Dag Details page (#5122) > > >> - [AIRFLOW-4331] Correct filter for Null-state runs from Dag Detail > > >page > > >> (#5123) > > >> - [AIRFLOW-4294] Fix missing dag & task runs in UI dag_id contains a > > >dot > > >> (#5111) > > >> - [AIRFLOW-4332] Upgrade sqlalchemy to remove security Vulnerability > > >> (#5113) > > >> - [AIRFLOW-4312] Add template_fields & template_ext to > > >BigQueryCheckO=E2=80=A6 > > >> (#5097) > > >> - [AIRFLOW-4293] Fix downgrade in > > >> d4ecb8fbee3_add_schedule_interval_to_dag.py (#5086) > > >> - [AIRFLOW-4267] Fix TI duration in Graph View (#5071) > > >> - [AIRFLOW-4163] IntervalCheckOperator supports relative diff and no= t > > >> ignore 0 (#4983) > > >> - [AIRFLOW-3938] QuboleOperator Fixes and Support for SqlCommand > > >(#4832) > > >> - [AIRFLOW-2903] Change default owner to ``airflow`` (#4151) > > >> - [aIRFLOW-4136] Fix overwrite of key_file by constructor (#5155) > > >> - [AIRFLOW-3241] Remove Invalid template ext in GCS Sensors (#4076) > > >> > > >> Misc/Internal > > >> """"""""""""" > > >> - [AIRFLOW-4937] Fix lodash security issue with version below 4.17.1= 3 > > >> (#5572) (used only in build-pipeline, not runtime) > > >> - [AIRFLOW-4868] Fix typo in kubernetes/docker/build.sh (#5505) > > >> - [AIRFLOW-4211] Add tests for WebHDFSHook (#5015) > > >> - [AIRFLOW-4320] Add tests for SegmentTrackEventOperator (#5104) > > >> - [AIRFLOW-4319] Add tests for Bigquery related Operators (#5101) > > >> - [AIRFLOW-4014] Change DatastoreHook and add tests (#4842) > > >> - [AIRFLOW-4322] Add test for VerticaOperator (#5107) > > >> - [AIRFLOW-4323] Add 2 tests for WinRMOperator (#5108) > > >> - [AIRFLOW-3677] Improve CheckOperator test coverage (#4756) > > >> - [AIRFLOW-4659] Fix pylint problems for api module (#5398) > > >> - [AIRFLOW-4358] Speed up test_jobs by not running tasks (#5162) > > >> - [AIRFLOW-4394] Don't test behaviour of BackfillJob from CLI tests > > >(#5160) > > >> - [AIRFLOW-3471] Move XCom out of models.py (#4629) > > >> - [AIRFLOW-4379] Remove duplicate code & Add validation in > > >gcs_to_gcs.py > > >> (#5145) > > >> - [AIRFLOW-4259] Move models out of models.py (#5056) > > >> - [AIRFLOW-XXX] Speed up building of Cassanda module on Travis > > >(#5233) > > >> - [AIRFLOW-4535] Break jobs.py into multiple files (#5303) > > >> - [AIRFLOW-1464] Batch update task_instance state (#5323) > > >> - [AIRFLOW-4554] Test for sudo command, add some other test docs > > >(#5310) > > >> - [AIRFLOW-4419] Refine concurrency check in scheduler (#5194) > > >> - [AIRFLOW-4269] Minor acceleration of jobs._process_task_instances(= ) > > >> (#5076) > > >> - [AIRFLOW-4341] Remove ``View.render()`` already exists in > > >fab.BaseView > > >> (#5125) > > >> - [AIRFLOW-4342] Use @cached_property instead of re-implementing it > > >each > > >> time (#5126) > > >> - [AIRFLOW-4256] Remove noqa from migrations (#5055) > > >> - [AIRFLOW-4034] Remove unnecessary string formatting with > > >``**locals()`` > > >> (#4861) > > >> - [AIRFLOW-3944] Remove code smells (#4762) > > >> > > >> Doc-only changes > > >> """""""""""""""" > > >> - [AIRFLOW-XXX] Correct BaseSensorOperator docs (#5562) > > >> - [AIRFLOW-4926] Fix example dags where its start_date is > > >> datetime.utcnow() (#5553) > > >> - [AIRFLOW-4860] Remove Redundant Information in Example Dags (#5497= ) > > >> - [AIRFLOW-4767] Fix errors in the documentation of Dataproc Operato= r > > >> (#5487) > > >> - [AIRFLOW-1684] Branching based on XCom variable (Docs) (#4365) > > >> - [AIRFLOW-3341] FAQ return DAG object example (#4605) > > >> - [AIRFLOW-4433] Add missing type in DockerOperator doc string > > >(#5205) > > >> - [AIRFLOW-4321] Replace incorrect info of Max Size limit of GCS > > >Object > > >> Size (#5106) > > >> - [AIRFLOW-XXX] Add information about user list (#5341) > > >> - [AIRFLOW-XXX] Clarify documentation related to autodetect paramete= r > > >in > > >> GCS_to_BQ Op (#5294) > > >> - [AIRFLOW-XXX] Remove mention of pytz compatibility from timezone > > >> documentation (#5316) > > >> - [AIRFLOW-XXX] Add missing docs for GoogleCloudStorageDeleteOperato= r > > >> (#5274) > > >> - [AIRFLOW-XXX] Remove incorrect note about Scopes of GCP connection > > >> (#5242) > > >> - [AIRFLOW-XXX] Fix mistakes in docs of Dataproc operators (#5192) > > >> - [AIRFLOW-XXX] Link to correct class for timedelta in macros.rst > > >(#5226) > > >> - [AIRFLOW-XXX] Add Kamil as committer (#5216) > > >> - [AIRFLOW-XXX] Add Joshua and Kevin as committer (#5207) > > >> - [AIRFLOW-XXX] Reduce log spam in tests (#5174) > > >> - [AIRFLOW-XXX] Speed up tests for PythonSensor (#5158) > > >> - [AIRFLOW-XXX] Add Bas Harenslak to committer list (#5157) > > >> - [AIRFLOW-XXX] Add Jarek Potiuk to commiter list (#5132) > > >> - [AIRFLOW-XXX] Update docstring for SchedulerJob (#5105) > > >> - [AIRFLOW-XXX] Fix docstrings for > > >CassandraToGoogleCloudStorageOperator > > >> (#5103) > > >> - [AIRFLOW-XXX] update SlackWebhookHook and SlackWebhookOperator > > >docstring > > >> (#5074) > > >> - [AIRFLOW-XXX] Ignore python files under node_modules in docs > > >(#5063) > > >> - [AIRFLOW-XXX] Build a universal wheel with LICNESE files (#5052) > > >> - [AIRFLOW-XXX] Fix docstrings of SQSHook (#5099) > > >> - [AIRFLOW-XXX] Use Py3.7 on readthedocs > > >> - [AIRFLOW-4446] Fix typos (#5217) > > >> > > >> > --=20 Best regards, Andrii Soldatenko Python Developer skype: andrii.soldatenko andrii.soldatenko@gmail.com http://asoldatenko.com --0000000000009379e7058de06d80--