From dev-return-7020-archive-asf-public=cust-asf.ponee.io@airflow.incubator.apache.org Sat Nov 17 12:53:50 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 11ACA180658 for ; Sat, 17 Nov 2018 12:53:49 +0100 (CET) Received: (qmail 77505 invoked by uid 500); 17 Nov 2018 11:53:48 -0000 Mailing-List: contact dev-help@airflow.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.incubator.apache.org Delivered-To: mailing list dev@airflow.incubator.apache.org Received: (qmail 77494 invoked by uid 99); 17 Nov 2018 11:53:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Nov 2018 11:53:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 57DA8180A4B for ; Sat, 17 Nov 2018 11:53:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.574 X-Spam-Level: **** X-Spam-Status: No, score=4.574 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, SPF_SOFTFAIL=0.972, TRACKER_ID=1.102] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id yM_ia5rjLt-D for ; Sat, 17 Nov 2018 11:53:45 +0000 (UTC) Received: from mail.firemirror.com (mail.firemirror.com [178.63.242.223]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id B176A60D20 for ; Sat, 17 Nov 2018 11:43:01 +0000 (UTC) Received: from [217.169.25.231] (helo=themisto.localdomain) by mail.firemirror.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.90_1) (envelope-from ) id 1gNyzw-0002yv-IG for dev@airflow.incubator.apache.org; Sat, 17 Nov 2018 11:42:53 +0000 From: Ash Berlin-Taylor Content-Type: multipart/alternative; boundary="Apple-Mail=_7A9EB2A3-9D6D-4F7E-804B-4DF4480D243C" Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: [VOTE] Airflow 1.10.1 RC2 Date: Sat, 17 Nov 2018 11:42:49 +0000 References: <8366BFC8-F305-43D4-96C1-BBBF6A5DC51A@reply.com> To: dev@airflow.incubator.apache.org In-Reply-To: <8366BFC8-F305-43D4-96C1-BBBF6A5DC51A@reply.com> Message-Id: <42571E48-6800-4B03-9F1C-D90504FC5E12@apache.org> X-Mailer: Apple Mail (2.3273) --Apple-Mail=_7A9EB2A3-9D6D-4F7E-804B-4DF4480D243C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 The RBAC UI is still marked as experimental and this isn't a regression = from 1.10.0, so could you be convinced to change this to a +1? There are other more critical changes I would like to get out, and I = will follow up straight away with a 1.10.2 that addresses this and XD's = points. (I feel Bolke's pain :) I'm now moderately annoyed at the Apache release = process and how long it takes, it means each release ends up getting = big) -ash > On 17 Nov 2018, at 01:01, Naik Kaxil wrote: >=20 > -1 (binding) . Tested it on Python 2.7.14, got expected result but had = 1 security concern that I want to get in the release. >=20 > Even when 'expose_config'=3DFalse, RBAC you still shows the configs = which can contain sensitive information like airflow metadb passwords, = etc. >=20 > If we can get that in +1 from me. The PR with this fixed has been = merged in the master, commit: = https://github.com/apache/incubator-airflow/commit/85abd44e241e17338a800e3= 7a3c2e85ef346898d = >=20 > PR: https://github.com/apache/incubator-airflow/pull/4194 = >=20 > Regards, > Kaxil >=20 > =EF=BB=BFOn 16/11/2018, 13:41, "Deng Xiaodong" > wrote: >=20 > Hi Ash, >=20 > I would like to give -1 (non-binding), due to two reasons we = discussed > earlier on Slack: >=20 > - there is an issue with the new =E2=80=9Cdelete DAG=E2=80=9D = button in UI. It=E2=80=99s a great > feature, so let=E2=80=99s try to release it =E2=80=9Cbug-less=E2=80=9D= . The fix is in PR > https://github.com/apache/incubator-airflow/pull/4069 (But = understand your > concern is that this PR comes with no test yet). >=20 > - it may be good to pin all dependencies to a specific version to = avoid the > incident caused by dependency breaking change (like what happens to = Redis > yesterday) >=20 >=20 > Last but not least: nice job! Thanks for your works! >=20 >=20 > XD >=20 >=20 > On Fri, Nov 16, 2018 at 21:13 Ash Berlin-Taylor = wrote: >=20 >> Friendly reminder for people (and especially committers) to test this = out >> and vote on it please! >>=20 >> -ash >>=20 >>>=20 >=20 > Kaxil Naik=20 >=20 > Data Reply > Nova South > 160 Victoria Street, Westminster > London SW1E 5LB - UK=20 > phone: +44 (0)20 7730 6000 > k.naik@reply.com > www.reply.com > On 14 Nov 2018, at 22:31, Ash Berlin-Taylor > wrote: >>>=20 >>> Hey all, >>>=20 >>> I have cut Airflow 1.10.1 RC2. This email is calling a vote on the >> release, which will last for 72 hours. Consider this my (binding) +1. >>>=20 >>> Airflow 1.10.1 RC2 is available at: >>>=20 >>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.1rc2/ >>>=20 >>> apache-airflow-1.10.1rc2+incubating-source.tar.gz is a source = release >> that comes with INSTALL instructions. >>> apache-airflow-1.10.1rc2+incubating-bin.tar.gz is the binary Python >> "sdist" release. >>>=20 >>> Public keys are available at: >>>=20 >>> https://dist.apache.org/repos/dist/release/incubator/airflow/KEYS >>>=20 >>> This release candidate has been published to PyPi as a convince for >> testing, but the vote is against the published artefacts at the above = URL, >> and not this. To install from PyPI run `pip install --pre = apache-airflow` >>>=20 >>> Only votes from PMC members are binding, but members of the = community >> are encouraged to test the release and vote with "(non-binding)". >>>=20 >>> Changes since 1.10.1rc1: >>>=20 >>> [AIRFLOW-3343] Update DockerOperator for Docker-py 3.0.0 API changes >> (#4187) >>> [AIRFLOW-XXX] Include 3193 in the changelog >>> [AIRFLOW-XXX] Remove duplicated line in Changelog (#4181) >>> [AIRFLOW-3339] Correctly get DAG timezone when start_date in >> default_args (#4186) >>>=20 >>> Changes since 1.10.1b1: >>>=20 >>> [AIRFLOW-XXX] Correct date and version in Changelog >>> [AIRFLOW-2779] Add license headers to doc files (#4178) >>> [AIRFLOW-XXX] Changelog and version for 1.10.1 >>> [AIRFLOW-2779] Add license headers to doc files (#4178) >>> [AIRFLOW-2779] Add project version to license (#4177) >>> [AIRFLOW-XXX] Sync changelog between release and master branch >>> [AIRFLOW-XXX] Add missing docs for SNS classes (#4155) >>> [AIRFLOW-3072] Assign permission get_logs_with_metadata to viewer = role >> (#4175) >>> [AIRFLOW-2723] Update lxml dependancy to >=3D 4.0.0 >>> [AIRFLOW-3325] Fix UI Page DAGs-column 'Recent Tasks' display issue >> (#4173) >>> [AIRFLOW-XXX] Update Updating instructions for changes in 1.10.1 >>> [AIRFLOW-XXX] Fix a few typos in CHANGELOG (#4169) >>>=20 >>>=20 >>> Full changelog is below: >>>=20 >>> New features: >>>=20 >>> [AIRFLOW-2524] Airflow integration with AWS Sagemaker >>> [AIRFLOW-2657] Add ability to delete DAG from web ui >>> [AIRFLOW-2780] Adds IMAP Hook to interact with a mail server >>> [AIRFLOW-2794] Add delete support for Azure blob >>> [AIRFLOW-2912] Add operators for Google Cloud Functions >>> [AIRFLOW-2974] Add Start/Restart/Terminate methods Databricks Hook >>> [AIRFLOW-2989] No Parameter to change bootDiskType for >> DataprocClusterCreateOperator >>> [AIRFLOW-3078] Basic operators for Google Compute Engine >>> [AIRFLOW-3147] Update Flask-AppBuilder version >>> [AIRFLOW-3231] Basic operators for Google Cloud SQL (deploy / patch = / >> delete) >>> [AIRFLOW-3276] Google Cloud SQL database create / patch / delete >> operators >>>=20 >>> Improvements: >>>=20 >>> [AIRFLOW-393] Add progress callbacks for FTP downloads >>> [AIRFLOW-520] Show Airflow version on web page >>> [AIRFLOW-843] Exceptions now available in context during >> on_failure_callback >>> [AIRFLOW-2476] Update tabulate dependency to v0.8.2 >>> [AIRFLOW-2592] Bump Bleach dependency >>> [AIRFLOW-2622] Add "confirm=3DFalse" option to SFTPOperator >>> [AIRFLOW-2662] support affinity & nodeSelector policies for = kubernetes >> executor/operator >>> [AIRFLOW-2709] Improve error handling in Databricks hook >>> [AIRFLOW-2723] Update lxml dependancy to >=3D 4.0. >>> [AIRFLOW-2763] No precheck mechanism in place during worker >> initialisation for the connection to metadata database >>> [AIRFLOW-2789] Add ability to create single node cluster to >> DataprocClusterCreateOperator >>> [AIRFLOW-2797] Add ability to create Google Dataproc cluster with = custom >> image >>> [AIRFLOW-2854] kubernetes_pod_operator add more configuration items >>> [AIRFLOW-2855] Need to Check Validity of Cron Expression When = Process >> DAG File/Zip File >>> [AIRFLOW-2904] Clean an unnecessary line in >> airflow/executors/celery_executor.py >>> [AIRFLOW-2921] A trivial incorrectness in CeleryExecutor() >>> [AIRFLOW-2922] Potential deal-lock bug in CeleryExecutor() >>> [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file >>> [AIRFLOW-2949] Syntax Highlight for Single Quote >>> [AIRFLOW-2951] dag_run end_date Null after a dag is finished >>> [AIRFLOW-2956] Kubernetes tolerations for pod operator >>> [AIRFLOW-2997] Support for clustered tables in Bigquery = hooks/operators >>> [AIRFLOW-3006] Fix error when schedule_interval=3D"None" >>> [AIRFLOW-3008] Move Kubernetes related example DAGs to >> contrib/example_dags >>> [AIRFLOW-3025] Allow to specify dns and dns-search parameters for >> DockerOperator >>> [AIRFLOW-3067] (www_rbac) Flask flash messages are not displayed >> properly (no background color) >>> [AIRFLOW-3069] Decode output of S3 file transform operator >>> [AIRFLOW-3072] Assign permission get_logs_with_metadata to viewer = role >>> [AIRFLOW-3090] INFO logs are too verbose >>> [AIRFLOW-3103] Update Flask-Login >>> [AIRFLOW-3112] Align SFTP hook with SSH hook >>> [AIRFLOW-3119] Enable loglevel on celery worker and inherit from >> airflow.cfg >>> [AIRFLOW-3137] Make ProxyFix middleware optional >>> [AIRFLOW-3173] Add _cmd options for more password config options >>> [AIRFLOW-3177] Change scheduler_heartbeat metric from gauge to = counter >>> [AIRFLOW-3195] Druid Hook: Log ingestion spec and task id >>> [AIRFLOW-3197] EMR Hook is missing some parameters to valid on the = AWS >> API >>> [AIRFLOW-3232] Make documentation for GCF Functions operator more >> readable >>> [AIRFLOW-3262] Can't get log containing Response when using >> SimpleHttpOperator >>> [AIRFLOW-3265] Add support for "unix_socket" in connection extra for >> Mysql Hook >>>=20 >>> Doc-only changes: >>>=20 >>> [AIRFLOW-1441] Tutorial Inconsistencies Between Example Pipeline >> Definition and Recap >>> [AIRFLOW-2682] Add how-to guide(s) for how to use basic operators = like >> BashOperator and PythonOperator >>> [AIRFLOW-3104] .airflowignore feature is not mentioned at all in >> documentation >>> [AIRFLOW-3237] Refactor example DAGs >>> [AIRFLOW-3187] Update airflow.gif file with a slower version >>> [AIRFLOW-3159] Update Airflow documentation on GCP Logging >>> [AIRFLOW-3030] Command Line docs incorrect subdir >>> [AIRFLOW-2990] Docstrings for Hooks/Operators are in incorrect = format >>> [AIRFLOW-3127] Celery SSL Documentation is out-dated >>> [AIRFLOW-2779] Add license headers to doc files >>> [AIRFLOW-2779] Add project version to license >>>=20 >>> Bug fixes: >>>=20 >>> [AIRFLOW-839] docker_operator.py attempts to log status key without >> first checking existence >>> [AIRFLOW-1104] Concurrency check in scheduler should count queued = tasks >> as well as running >>> [AIRFLOW-1163] Add support for x-forwarded-* headers to support = access >> behind AWS ELB >>> [AIRFLOW-1195] Cleared tasks in SubDagOperator do not trigger Parent >> dag_runs >>> [AIRFLOW-1508] Skipped state not part of State.task_states >>> [AIRFLOW-1762] Use key_file in SSHHook.create_tunnel() >>> [AIRFLOW-1837] Differing start_dates on tasks not respected by = scheduler. >>> [AIRFLOW-1874] Support standard SQL in Check, ValueCheck and >> IntervalCheck BigQuery operators >>> [AIRFLOW-1917] print() from python operators end up with extra new = line >>> [AIRFLOW-1970] Database cannot be initialized if an invalid fernet = key >> is provided >>> [AIRFLOW-2145] Deadlock after clearing a running task >>> [AIRFLOW-2216] Cannot specify a profile for AWS Hook to load with s3 >> config file >>> [AIRFLOW-2574] initdb fails when mysql password contains percent = sign >>> [AIRFLOW-2707] Error accessing log files from web UI >>> [AIRFLOW-2716] Replace new Python 3.7 keywords >>> [AIRFLOW-2744] RBAC app doesn't integrate plugins (blueprints etc) >>> [AIRFLOW-2772] BigQuery hook does not allow specifying both the >> partition field name and table name at the same time >>> [AIRFLOW-2778] Bad Import in collect_dag in DagBag >>> [AIRFLOW-2786] Variables view fails to render if a variable has an = empty >> key >>> [AIRFLOW-2799] Filtering UI objects by datetime is broken >>> [AIRFLOW-2800] Remove airflow/ low-hanging linting errors >>> [AIRFLOW-2825] S3ToHiveTransfer operator may not may able to handle = GZIP >> file with uppercase ext in S3 >>> [AIRFLOW-2848] dag_id is missing in metadata table "job" for = LocalTaskJob >>> [AIRFLOW-2860] DruidHook: time variable is not updated correctly = when >> checking for timeout >>> [AIRFLOW-2865] Race condition between on_success_callback and >> LocalTaskJob's cleanup >>> [AIRFLOW-2893] Stuck dataflow job due to jobName mismatch. >>> [AIRFLOW-2895] Prevent scheduler from spamming heartbeats/logs >>> [AIRFLOW-2900] Code not visible for Packaged DAGs >>> [AIRFLOW-2905] Switch to regional dataflow job service. >>> [AIRFLOW-2907] Sendgrid - Attachments - ERROR - Object of type = 'bytes' >> is not JSON serializable >>> [AIRFLOW-2938] Invalid 'extra' field in connection can raise an >> AttributeError when attempting to edit >>> [AIRFLOW-2979] Deprecated Celery Option not in Options list >>> [AIRFLOW-2981] TypeError in dataflow operators when using GCS jar or >> py_file >>> [AIRFLOW-2984] Cannot convert naive_datetime when task has a naive >> start_date/end_date >>> [AIRFLOW-2994] flatten_results in BigQueryOperator/BigQueryHook = should >> default to None >>> [AIRFLOW-3002] ValueError in dataflow operators when using GCS jar = or >> py_file >>> [AIRFLOW-3012] Email on sla miss is send only to first address on = the >> list >>> [AIRFLOW-3046] ECS Operator mistakenly reports success when task is >> killed due to EC2 host termination >>> [AIRFLOW-3064] No output from `airflow test` due to default logging >> config >>> [AIRFLOW-3072] Only admin can view logs in RBAC UI >>> [AIRFLOW-3079] Improve initdb to support MSSQL Server >>> [AIRFLOW-3089] Google auth doesn't work under http >>> [AIRFLOW-3099] Errors raised when some blocs are missing in = airflow.cfg >>> [AIRFLOW-3109] Default user permission should contain 'can_clear' >>> [AIRFLOW-3111] Confusing comments and instructions for log templates = in >> UPDATING.md and default_airflow.cfg >>> [AIRFLOW-3124] Broken webserver debug mode (RBAC) >>> [AIRFLOW-3136] Scheduler Failing the Task retries run while = processing >> Executor Events >>> [AIRFLOW-3138] Migration cc1e65623dc7 creates issues with postgres >>> [AIRFLOW-3161] Log Url link does not link to task instance logs in = RBAC >> UI >>> [AIRFLOW-3162] HttpHook fails to parse URL when port is specified >>> [AIRFLOW-3183] Potential Bug in >> utils/dag_processing/DagFileProcessorManager.max_runs_reached() >>> [AIRFLOW-3203] Bugs in DockerOperator & Some operator test scripts = were >> named incorrectly >>> [AIRFLOW-3238] Dags, removed from the filesystem, are not = deactivated on >> initdb >>> [AIRFLOW-3268] Cannot pass SSL dictionary to mysql connection via = URL >>> [AIRFLOW-3277] Invalid timezone transition handling for cron = schedules >>> [AIRFLOW-3295] Require encryption in DaskExecutor when certificates = are >> configured. >>> [AIRFLOW-3297] EmrStepSensor marks cancelled step as successful --Apple-Mail=_7A9EB2A3-9D6D-4F7E-804B-4DF4480D243C--