airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <ash_airflowl...@firemirror.com>
Subject Re: [VOTE] Airflow 1.9.0rc1
Date Thu, 09 Nov 2017 12:44:31 GMT
That final URL should have been https://issues.apache.org/jira/browse/AIRFLOW-1797 <https://issues.apache.org/jira/browse/AIRFLOW-1797>,
which now has a PR for it https://github.com/apache/incubator-airflow/pull/2771 <https://github.com/apache/incubator-airflow/pull/2771>

There's going to be some more fixes coming around S3 logs.

One thing I have noticed is the switch to per-try logs (which is awesome! So much easier to
view) means I can't read old logs anymore because it isn't split by try, it's just all in
one file.

Is it worth making a change to the log task handlers to try loading under the old pattern
if none are found with the new style? (The other option is that I just run a migration script
to move the old logs into the new place. That sort of only helps me though.)

-ash

> On 9 Nov 2017, at 11:04, Ash Berlin-Taylor <ash_airflowlist@firemirror.com> wrote:
> 
> And one more - on Python3 we can't use S3Hook.load_string due to bytes vs string issue:
https://issues.apache.org/jira/browse/AIRFLOW-1796 <https://issues.apache.org/jira/browse/AIRFLOW-1796>
> 
> I'll try and work on fixes for some/all of these today, including adding and expanding
on the tests for S3Hook which it looks like was kind of lacking.
> 
> -ash
> 
>> On 9 Nov 2017, at 10:54, Ash Berlin-Taylor <ash_airflowlist@firemirror.com>
wrote:
>> 
>> Thanks for picking this up. Your fix should stop the 500 error, but there's another
problem (which is ultimately user misconfiguration about) https://issues.apache.org/jira/browse/AIRFLOW-1796
- the fix for that to update a doc somewhere, and probably validate this settings is correct
at start time.
>> 
>> 
>> I've found another issue related to arg names of S3Hook. In 1.8.2 it was `s3_conn_id`
but the move to boto3/basing off AWSHook now expects `aws_conn_id`, and various places in
Airflow code base (and a few places in our dags/operators code base) still pass it as s3_conn_id.
I've created https://issues.apache.org/jira/browse/AIRFLOW-1795 for that issue.
>> 
>> -ash
>> 
>> 
>>> On 8 Nov 2017, at 18:54, Daniel Huang <dxhuang@gmail.com> wrote:
>>> 
>>> Still testing this out.
>>> 
>>> Put up a small fix for Ash's second exception
>>> https://github.com/apache/incubator-airflow/pull/2766
>>> 
>>> On Wed, Nov 8, 2017 at 10:48 AM, Bolke de Bruin <bdbruin@gmail.com> wrote:
>>> 
>>>> Hi Chris,
>>>> 
>>>> Actively testing here: we found an issue in the SSHOperator introduced in
>>>> 1.9.0 (fix already merged for RC2, but blocking I as it stops us from
>>>> running SSH properly), some minor fixes by Airbnb should also be in RC2.
>>>> There is some logging “weirdness”, that might warrant a small patch here
in
>>>> there and could be squeezed into RC2, but I don’t consider them blocking.
>>>> 
>>>> So almost there, but we need an RC2 imho.
>>>> 
>>>> -1, binding.
>>>> 
>>>> Bolke
>>>> 
>>>>> On 8 Nov 2017, at 19:00, Ash Berlin-Taylor <ash_airflowlist@firemirror.
>>>> com> wrote:
>>>>> 
>>>>> -1 (for now. Non binding. Is that how this process works?)
>>>>> 
>>>>> We've built a test env for this RC and are testing, but have run into
an
>>>> issue reading task logs. (See below)
>>>>> 
>>>>> We haven't gotten very far with this yet, we will dig more tomorrow
>>>> (it's the end of the UK work day now). I suspect this might be how we've
>>>> misconfigured our logging. We will see tomorrow.
>>>>> 
>>>>> -ash
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> File "/usr/local/lib/python3.5/dist-packages/airflow/www/views.py",
>>>> line 712, in log
>>>>> logs = handler.read(ti)
>>>>> AttributeError: 'NoneType' object has no attribute 'read'
>>>>> 
>>>>> During handling of the above exception, another exception occurred:
>>>>> 
>>>>> Traceback (most recent call last):
>>>>> File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1988,
>>>> in wsgi_app
>>>>> response = self.full_dispatch_request()
>>>>> File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1641,
>>>> in full_dispatch_request
>>>>> rv = self.handle_user_exception(e)
>>>>> File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1544,
>>>> in handle_user_exception
>>>>> reraise(exc_type, exc_value, tb)
>>>>> File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line
>>>> 33, in reraise
>>>>> raise value
>>>>> File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1639,
>>>> in full_dispatch_request
>>>>> rv = self.dispatch_request()
>>>>> File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1625,
>>>> in dispatch_request
>>>>> return self.view_functions[rule.endpoint](**req.view_args)
>>>>> File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py",
>>>> line 69, in inner
>>>>> return self._run_view(f, *args, **kwargs)
>>>>> File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py",
>>>> line 368, in _run_view
>>>>> return fn(self, *args, **kwargs)
>>>>> File "/usr/local/lib/python3.5/dist-packages/flask_login.py", line
>>>> 758, in decorated_view
>>>>> return func(*args, **kwargs)
>>>>> File "/usr/local/lib/python3.5/dist-packages/airflow/www/utils.py",
>>>> line 262, in wrapper
>>>>> return f(*args, **kwargs)
>>>>> File "/usr/local/lib/python3.5/dist-packages/airflow/www/views.py",
>>>> line 715, in log
>>>>> .format(task_log_reader, e.message)]
>>>>> AttributeError: 'AttributeError' object has no attribute 'message'
>>>>> 
>>>>> 
>>>>>> On 8 Nov 2017, at 17:46, Chris Riccomini <criccomini@apache.org>
wrote:
>>>>>> 
>>>>>> Anyone? :/
>>>>>> 
>>>>>> On Mon, Nov 6, 2017 at 1:22 PM, Chris Riccomini <criccomini@apache.org>
>>>>>> wrote:
>>>>>> 
>>>>>>> Hey all,
>>>>>>> 
>>>>>>> I have cut Airflow 1.9.0 RC1. This email is calling a vote on
the
>>>> release,
>>>>>>> which will last fo 72 hours. Consider this my (binding) +1.
>>>>>>> 
>>>>>>> Airflow 1.9.0 RC1 is available at:
>>>>>>> 
>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0rc1/
>>>>>>> 
>>>>>>> apache-airflow-1.9.0rc1+incubating-source.tar.gz is a source
release
>>>> that
>>>>>>> comes with INSTALL instructions.
>>>>>>> apache-airflow-1.9.0rc1+incubating-bin.tar.gz is the binary Python
>>>>>>> "sdist" release.
>>>>>>> 
>>>>>>> Public keys are available at:
>>>>>>> 
>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/
>>>>>>> 
>>>>>>> The release contains the following JIRAs:
>>>>>>> 
>>>>>>> ISSUE ID    |DESCRIPTION                                    
  |PR
>>>>>>> |COMMIT
>>>>>>> AIRFLOW-1779|Add keepalive packets to ssh hook              
  |#2749
>>>>>>> |d2f9d1
>>>>>>> AIRFLOW-1776|stdout/stderr logging not captured             
  |#2745
>>>>>>> |590d9f
>>>>>>> AIRFLOW-1771|Change heartbeat text from boom to heartbeat   
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1767|Airflow Scheduler no longer schedules DAGs     
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1765|Default API auth backed should deny all.       
  |#2737
>>>>>>> |6ecdac
>>>>>>> AIRFLOW-1764|Web Interface should not use experimental api  
  |#2738
>>>>>>> |6bed1d
>>>>>>> AIRFLOW-1757|Contrib.SparkSubmitOperator should allow --package|#2725
>>>>>>> |4e06ee
>>>>>>> AIRFLOW-1745|BashOperator ignores SIGPIPE in subprocess     
  |#2714
>>>>>>> |e021c9
>>>>>>> AIRFLOW-1744|task.retries can be False                      
  |#2713
>>>>>>> |6144c6
>>>>>>> AIRFLOW-1743|Default config template should not contain ldap
fi|#2712
>>>>>>> |270684
>>>>>>> AIRFLOW-1741|Task Duration shows two charts on first page load.|#2711
>>>>>>> |974b49
>>>>>>> AIRFLOW-1734|Sqoop Operator contains logic errors & needs
optio|#2703
>>>>>>> |f6810c
>>>>>>> AIRFLOW-1731|Import custom config on PYTHONPATH             
  |#2721
>>>>>>> |f07eb3
>>>>>>> AIRFLOW-1726|Copy Expert command for Postgres Hook          
  |#2698
>>>>>>> |8a4ad3
>>>>>>> AIRFLOW-1719|Fix small typo - your vs you                   
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1712|Log SSHOperator output                         
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1711|Ldap Attributes not always a "list" part 2     
  |#2731
>>>>>>> |40a936
>>>>>>> AIRFLOW-1706|Scheduler is failed on startup with MS SQL Server
|#2733
>>>>>>> |9e209b
>>>>>>> AIRFLOW-1698|Remove confusing SCHEDULER_RUNS env var from syste|#2677
>>>>>>> |00dd06
>>>>>>> AIRFLOW-1695|Redshift Hook using boto3 & AWS Hook       
      |#2717
>>>>>>> |bfddae
>>>>>>> AIRFLOW-1694|Hive Hooks: Python 3 does not have an `itertools.i|#2674
>>>>>>> |c6e5ae
>>>>>>> AIRFLOW-1692|Master cannot be checked out on windows        
  |#2673
>>>>>>> |31805e
>>>>>>> AIRFLOW-1691|Add better documentation for Google cloud storage
|#2671
>>>>>>> |ace2b1
>>>>>>> AIRFLOW-1690|Error messages regarding gcs log commits are spars|#2670
>>>>>>> |5fb5cd
>>>>>>> AIRFLOW-1682|S3 task handler never writes to S3             
  |#2664
>>>>>>> |0080f0
>>>>>>> AIRFLOW-1678|Fix docstring errors for `set_upstream` and `set_d|-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1677|Fix typo in example_qubole_operator            
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1676|GCS task handler never writes to GCS           
  |#2659
>>>>>>> |781fa4
>>>>>>> AIRFLOW-1675|Fix API docstrings to be properly rendered     
  |#2667
>>>>>>> |f12381
>>>>>>> AIRFLOW-1671|Missing @apply_defaults annotation for gcs downloa|#2655
>>>>>>> |97666b
>>>>>>> AIRFLOW-1669|Fix Docker import in Master                    
  |#na
>>>>>>> |f7f2a8
>>>>>>> AIRFLOW-1668|Redhsift requires a keep alive of < 300s    
     |#2650
>>>>>>> |f2bb77
>>>>>>> AIRFLOW-1664|Make MySqlToGoogleCloudStorageOperator support bin|#2649
>>>>>>> |95813d
>>>>>>> AIRFLOW-1660|Change webpage width to full-width             
  |#2646
>>>>>>> |8ee3d9
>>>>>>> AIRFLOW-1659|Fix invalid attribute bug in FileTaskHandler   
  |#2645
>>>>>>> |bee823
>>>>>>> AIRFLOW-1658|Kill (possibly) still running Druid indexing job
a|#2644
>>>>>>> |cbf7ad
>>>>>>> AIRFLOW-1657|Handle failure of Qubole Operator for s3distcp had|-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1654|Show tooltips for link icons in DAGs view      
  |#2642
>>>>>>> |ada7b2
>>>>>>> AIRFLOW-1647|Fix Spark-sql hook                             
  |#2637
>>>>>>> |b1e5c6
>>>>>>> AIRFLOW-1641|Task gets stuck in queued state                
  |#2715
>>>>>>> |735497
>>>>>>> AIRFLOW-1640|Add Qubole default connection in connection table
|-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1639|ValueError does not have .message attribute    
  |#2629
>>>>>>> |87df67
>>>>>>> AIRFLOW-1637|readme not tracking master branch for travis   
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1636|aws and emr connection types get cleared       
  |#2626
>>>>>>> |540e04
>>>>>>> AIRFLOW-1635|Allow creating Google Cloud Platform connection
wi|#2640
>>>>>>> |6dec7a
>>>>>>> AIRFLOW-1629|make extra a textarea in edit connections form 
  |#2623
>>>>>>> |f5d46f
>>>>>>> AIRFLOW-1628|Docstring of sqlsensor is incorrect            
  |#2621
>>>>>>> |9ba73d
>>>>>>> AIRFLOW-1627|SubDagOperator initialization should only query
po|#2620
>>>>>>> |516ace
>>>>>>> AIRFLOW-1621|Add tests for logic added on server side dag list
|#2614
>>>>>>> |8de9fd
>>>>>>> AIRFLOW-1614|Improve performance of DAG parsing when there are
|#2610
>>>>>>> |a95adb
>>>>>>> AIRFLOW-1611|Customize logging in Airflow                   
  |#2631
>>>>>>> |8b4a50
>>>>>>> AIRFLOW-1609|Ignore all venvs in gitignore                  
  |#2608
>>>>>>> |f1f9b4
>>>>>>> AIRFLOW-1608|GCP Dataflow hook missing pending job state    
  |#2607
>>>>>>> |653562
>>>>>>> AIRFLOW-1606|DAG.sync_to_db is static, but takes a DAG as first|#2606
>>>>>>> |6ac296
>>>>>>> AIRFLOW-1605|Fix log source of local loggers                
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1604|Rename the logger to log                       
  |#2604
>>>>>>> |af4050
>>>>>>> AIRFLOW-1602|Use LoggingMixin for the DAG class             
  |#2602
>>>>>>> |956699
>>>>>>> AIRFLOW-1601|Add configurable time between SIGTERM and SIGKILL
|#2601
>>>>>>> |48a95e
>>>>>>> AIRFLOW-1600|Uncaught exceptions in get_fernet if cryptography
|#2600
>>>>>>> |ad963e
>>>>>>> AIRFLOW-1597|Add GameWisp as Airflow user                   
  |#2599
>>>>>>> |26b747
>>>>>>> AIRFLOW-1594|Installing via pip copies test files into python
l|#2597
>>>>>>> |a6b23a
>>>>>>> AIRFLOW-1593|Expose load_string in WasbHook                 
  |#2596
>>>>>>> |7ece95
>>>>>>> AIRFLOW-1591|Exception: 'TaskInstance' object has no attribute
|#2578
>>>>>>> |f4653e
>>>>>>> AIRFLOW-1590|Small fix for dates util                       
  |#2652
>>>>>>> |31946e
>>>>>>> AIRFLOW-1587|fix `ImportError: cannot import name 'CeleryExecut|#2590
>>>>>>> |34c73b
>>>>>>> AIRFLOW-1586|MySQL to GCS to BigQuery fails for tables with dat|#2589
>>>>>>> |e83012
>>>>>>> AIRFLOW-1584|Remove the insecure /headers endpoints         
  |#2588
>>>>>>> |17ac07
>>>>>>> AIRFLOW-1582|Improve logging structure of Airflow           
  |#2592
>>>>>>> |a7a518
>>>>>>> AIRFLOW-1580|Error in string formatter when throwing an excepti|#2583
>>>>>>> |ea9ab9
>>>>>>> AIRFLOW-1579|Allow jagged rows in BQ Hook.                  
  |#2582
>>>>>>> |5b978b
>>>>>>> AIRFLOW-1577|Add token support to DatabricksHook            
  |#2579
>>>>>>> |c2c515
>>>>>>> AIRFLOW-1573|Remove `thrift < 0.10.0` requirement        
     |#2574
>>>>>>> |aa95f2
>>>>>>> AIRFLOW-1571|Add AWS Lambda Hook for invoking Lambda Function
 |#2718
>>>>>>> |017f18
>>>>>>> AIRFLOW-1568|Add datastore import/export operator           
  |#2568
>>>>>>> |86063b
>>>>>>> AIRFLOW-1567|Clean up ML Engine operators                   
  |#2567
>>>>>>> |af91e2
>>>>>>> AIRFLOW-1564|Default logging filename contains a colon      
  |#2565
>>>>>>> |4c674c
>>>>>>> AIRFLOW-1560|Add AWS DynamoDB hook for inserting batch items
  |#2587
>>>>>>> |71400b
>>>>>>> AIRFLOW-1556|BigQueryBaseCursor should support SQL parameters
 |#2557
>>>>>>> |9df0ac
>>>>>>> AIRFLOW-1546| add Zymergen to org list in README            
  |#2512
>>>>>>> |7cc346
>>>>>>> AIRFLOW-1535|Add support for Dataproc serviceAccountScopes in
D|#2546
>>>>>>> |b1f902
>>>>>>> AIRFLOW-1529|Support quoted newlines in Google BigQuery load
jo|#2545
>>>>>>> |4a4b02
>>>>>>> AIRFLOW-1527|Refactor celery config to make use of template 
  |#2542
>>>>>>> |f4437b
>>>>>>> AIRFLOW-1522|Increase size of val column for variable table in
|#2535
>>>>>>> |8a2d24
>>>>>>> AIRFLOW-1521|Template fields definition for bigquery_table_dele|#2534
>>>>>>> |f1a7c0
>>>>>>> AIRFLOW-1520|S3Hook uses boto2                              
  |#2532
>>>>>>> |386583
>>>>>>> AIRFLOW-1519|Main DAG list page does not scale using client sid|#2531
>>>>>>> |d7d7ce
>>>>>>> AIRFLOW-1512|Add operator for running Python functions in a vir|#2446
>>>>>>> |14e6d7
>>>>>>> AIRFLOW-1507|Make src, dst and bucket parameters as templated
i|#2516
>>>>>>> |d295cf
>>>>>>> AIRFLOW-1505|Document when Jinja substitution occurs        
  |#2523
>>>>>>> |984a87
>>>>>>> AIRFLOW-1504|Log Cluster Name on Dataproc Operator When Execute|#2517
>>>>>>> |1cd6c4
>>>>>>> AIRFLOW-1499s|Eliminate duplicate and unneeded code         
   |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1497|Hidden fields in connection form aren't reset when|#2507
>>>>>>> |d8da8b
>>>>>>> AIRFLOW-1493|Fix race condition with airflow run            
  |#2505
>>>>>>> |b2e175
>>>>>>> AIRFLOW-1492|Add metric for task success/failure            
  |#2504
>>>>>>> |fa84d4
>>>>>>> AIRFLOW-1489|Docs: Typo in BigQueryCheckOperator            
  |#2501
>>>>>>> |111ce5
>>>>>>> AIRFLOW-1483|Page size on model views is to large to render qui|#2497
>>>>>>> |04bfba
>>>>>>> AIRFLOW-1478|Chart -> Owner column should be sortable    
     |#2493
>>>>>>> |651e60
>>>>>>> AIRFLOW-1476|Add INSTALL file for source releases           
  |#2492
>>>>>>> |da76ac
>>>>>>> AIRFLOW-1474|Add dag_id regex for 'airflow clear' CLI command
 |#2486
>>>>>>> |18f849
>>>>>>> AIRFLOW-1470s|BashSensor Implementation                     
   |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1459|integration rst doc is broken in github view   
  |#2481
>>>>>>> |322ec9
>>>>>>> AIRFLOW-1438|Scheduler batch queries should have a limit    
  |#2462
>>>>>>> |3547cb
>>>>>>> AIRFLOW-1437|BigQueryTableDeleteOperator should define deletion|#2459
>>>>>>> |b87903
>>>>>>> AIRFLOW-1432|NVD3 Charts do not have labeled axes and units cha|#2710
>>>>>>> |70ffa4
>>>>>>> AIRFLOW-1402|Cleanup SafeConfigParser DeprecationWarning    
  |#2435
>>>>>>> |38c86b
>>>>>>> AIRFLOW-1401|Standardize GCP project, region, and zone argument|#2439
>>>>>>> |b6d363
>>>>>>> AIRFLOW-1397|Airflow 1.8.1 - No data displays in Last Run Colum|-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1394|Add quote_character parameter to GoogleCloudStorag|#2428
>>>>>>> |9fd0be
>>>>>>> AIRFLOW-1389|BigQueryOperator should support `createDisposition|#2470
>>>>>>> |6e2640
>>>>>>> AIRFLOW-1384|Add ARGO/CaDC                                  
  |#2434
>>>>>>> |715947
>>>>>>> AIRFLOW-1368|Automatically remove the container when it exits
 |#2653
>>>>>>> |d42d23
>>>>>>> AIRFLOW-1359|Provide GoogleCloudML operator for model evaluatio|#2407
>>>>>>> |194d1d
>>>>>>> AIRFLOW-1356|add `--celery_hostname` to `airflow worker`    
  |#2405
>>>>>>> |b9d7d1
>>>>>>> AIRFLOW-1352|Revert bad logging Handler                     
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1350|Add "query_uri" parameter for Google DataProc oper|#2402
>>>>>>> |d32c72
>>>>>>> AIRFLOW-1348|Paginated UI has broken toggles after first page
 |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1345|Don't commit on each loop                      
  |#2397
>>>>>>> |0dd002
>>>>>>> AIRFLOW-1344|Builds failing on Python 3.5 with AttributeError
 |#2394
>>>>>>> |2a5883
>>>>>>> AIRFLOW-1343|Add airflow default label to the dataproc operator|#2396
>>>>>>> |e4b240
>>>>>>> AIRFLOW-1338|gcp_dataflow_hook is incompatible with the recent
|#2388
>>>>>>> |cf2605
>>>>>>> AIRFLOW-1337|Customize log format via config file           
  |#2392
>>>>>>> |4841e3
>>>>>>> AIRFLOW-1335|Use buffered logger                            
  |#2386
>>>>>>> |0d23d3
>>>>>>> AIRFLOW-1333|Enable copy function for Google Cloud Storage Hook|#2385
>>>>>>> |e2c383
>>>>>>> AIRFLOW-1331|Contrib.SparkSubmitOperator should allow --package|#2622
>>>>>>> |fbca8f
>>>>>>> AIRFLOW-1330|Connection.parse_from_uri doesn't work for google_|#2525
>>>>>>> |6e5e9d
>>>>>>> AIRFLOW-1324|Make the Druid operator/hook more general      
  |#2378
>>>>>>> |de99aa
>>>>>>> AIRFLOW-1323|Operators related to Dataproc should keep some par|#2636
>>>>>>> |ed248d
>>>>>>> AIRFLOW-1315|Add Qubole File and Partition Sensors          
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1309|Add optional hive_tblproperties in HiveToDruidTran|-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1301|Add New Relic to Airflow user list             
  |#2359
>>>>>>> |355fc9
>>>>>>> AIRFLOW-1299|Google Dataproc cluster creation operator should
s|#2358
>>>>>>> |c2b80e
>>>>>>> AIRFLOW-1289|Don't restrict scheduler threads to CPU cores  
  |#2353
>>>>>>> |8e23d2
>>>>>>> AIRFLOW-1286|BaseTaskRunner - Exception TypeError: a bytes-like|#2363
>>>>>>> |d8891d
>>>>>>> AIRFLOW-1277|Forbid creation of a known event with empty fields|#na
>>>>>>> |65184a
>>>>>>> AIRFLOW-1276|Forbid event creation with end_data earlier than
s|#na
>>>>>>> |d5d02f
>>>>>>> AIRFLOW-1275|Fix `airflow pool` command exception           
  |#2346
>>>>>>> |9958aa
>>>>>>> AIRFLOW-1273|Google Cloud ML Version and Model CRUD Operator
  |#2379
>>>>>>> |534a0e
>>>>>>> AIRFLOW-1272|Google Cloud ML Batch Prediction Operator      
  |#2390
>>>>>>> |e92d6b
>>>>>>> AIRFLOW-1271|Google Cloud ML Training Operator              
  |#2408
>>>>>>> |0fc450
>>>>>>> AIRFLOW-1256|Add United Airlines as Airflow user            
  |#2332
>>>>>>> |d3484a
>>>>>>> AIRFLOW-1251|Add eRevalue as an Airflow user                
  |#2331
>>>>>>> |8d5160
>>>>>>> AIRFLOW-1248|Fix inconsistent configuration name for worker tim|#2328
>>>>>>> |92314f
>>>>>>> AIRFLOW-1247|CLI: ignore all dependencies argument ignored  
  |#2441
>>>>>>> |e88ecf
>>>>>>> AIRFLOW-1245|Fix random failure of test_trigger_dag_for_date
un|#2325
>>>>>>> |cef01b
>>>>>>> AIRFLOW-1244|Forbid creation of a pool with empty name      
  |#2324
>>>>>>> |df9a10
>>>>>>> AIRFLOW-1242|BigQueryHook assumes that a valid project_id can't|#2335
>>>>>>> |ffe616
>>>>>>> AIRFLOW-1237|Fix IN-predicate sqlalchemy warning            
  |#2320
>>>>>>> |a1f422
>>>>>>> AIRFLOW-1234|Cover utils.operator_helpers with unit tests   
  |#2317
>>>>>>> |d16537
>>>>>>> AIRFLOW-1233|Cover utils.json with unit tests               
  |#2316
>>>>>>> |502410
>>>>>>> AIRFLOW-1232|Remove deprecated readfp warning               
  |#2315
>>>>>>> |6ffaaf
>>>>>>> AIRFLOW-1231|Use flask_wtf.CSRFProtect instead of flask_wtf.Csr|#2313
>>>>>>> |cac49e
>>>>>>> AIRFLOW-1221|Fix DatabricksSubmitRunOperator Templating     
  |#2308
>>>>>>> |0fa104
>>>>>>> AIRFLOW-1217|Enable logging in Sqoop hook                   
  |#2307
>>>>>>> |4f459b
>>>>>>> AIRFLOW-1213|Add hcatalog parameters to the sqoop operator/hook|#2305
>>>>>>> |857850
>>>>>>> AIRFLOW-1208|Speed-up cli tests                             
  |#2301
>>>>>>> |21c142
>>>>>>> AIRFLOW-1207|Enable utils.helpers unit tests                
  |#2300
>>>>>>> |8ac87b
>>>>>>> AIRFLOW-1203|Tests failing after oauth upgrade              
  |#2296
>>>>>>> |3e9c66
>>>>>>> AIRFLOW-1201|Update deprecated 'nose-parameterized' library to
|#2298
>>>>>>> |d2d3e4
>>>>>>> AIRFLOW-1193|Add Checkr to Airflow user list                
  |#2276
>>>>>>> |707238
>>>>>>> AIRFLOW-1189|Get pandas DataFrame using BigQueryHook fails  
  |#2287
>>>>>>> |93666f
>>>>>>> AIRFLOW-1188|Add max_bad_records param to GoogleCloudStorageToB|#2286
>>>>>>> |443e6b
>>>>>>> AIRFLOW-1187|Obsolete package names in documentation        
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1185|Incorrect url to PyPi                          
  |#2283
>>>>>>> |829755
>>>>>>> AIRFLOW-1181|Enable delete and list function for Google Cloud
S|#2281
>>>>>>> |24f73c
>>>>>>> AIRFLOW-1179|Pandas 0.20 broke Google BigQuery hook         
  |#2279
>>>>>>> |ac9ccb
>>>>>>> AIRFLOW-1177|variable json deserialize does not work at set def|#2540
>>>>>>> |65319a
>>>>>>> AIRFLOW-1175|Add Pronto Tools to Airflow user list          
  |#2277
>>>>>>> |86aafa
>>>>>>> AIRFLOW-1173|Add Robinhood to list of Airflow users         
  |#2271
>>>>>>> |379115
>>>>>>> AIRFLOW-1165|airflow webservice crashes on ubuntu16 - python3
 |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1160|Upadte SparkSubmitOperator parameters          
  |#2265
>>>>>>> |2e3f07
>>>>>>> AIRFLOW-1155|Add Tails.com to community                     
  |#2261
>>>>>>> |2fa690
>>>>>>> AIRFLOW-1149|Allow custom filters to be added to jinja2     
  |#2258
>>>>>>> |48135a
>>>>>>> AIRFLOW-1141|Remove DAG.crawl_for_tasks method              
  |#2275
>>>>>>> |a30fee
>>>>>>> AIRFLOW-1140|DatabricksSubmitRunOperator should template the
"j|#2255
>>>>>>> |e6d316
>>>>>>> AIRFLOW-1136|Invalid parameters are not captured for Sqoop oper|#2252
>>>>>>> |2ef4db
>>>>>>> AIRFLOW-1125|Clarify documentation regarding fernet_key     
  |#2251
>>>>>>> |831f8d
>>>>>>> AIRFLOW-1122|Node strokes are too thin for people with color
vi|#2246
>>>>>>> |a08761
>>>>>>> AIRFLOW-1121|airflow webserver --pid no longer write out pid
fi|-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1118|Add evo.company to Airflow users               
  |#2243
>>>>>>> |f16914
>>>>>>> AIRFLOW-1112|Log which pool is full in scheduler when pool slot|#2242
>>>>>>> |74c1ce
>>>>>>> AIRFLOW-1107|Add support for ftps non-default port          
  |#2240
>>>>>>> |4d0c2f
>>>>>>> AIRFLOW-1106|Add Groupalia/Letsbonus                        
  |#2239
>>>>>>> |945b42
>>>>>>> AIRFLOW-1095|ldap_auth memberOf should come from configuration
|#2232
>>>>>>> |6b1c32
>>>>>>> AIRFLOW-1094|Invalid unit tests under `contrib/`            
  |#2234
>>>>>>> |219c50
>>>>>>> AIRFLOW-1091|As a release manager I want to be able to compare
|#2231
>>>>>>> |bfae42
>>>>>>> AIRFLOW-1090|Add HBO                                        
  |#2230
>>>>>>> |177d34
>>>>>>> AIRFLOW-1089|Add Spark application arguments to SparkSubmitOper|#2229
>>>>>>> |e5b914
>>>>>>> AIRFLOW-1081|Task duration page is slow                     
  |#2226
>>>>>>> |0da512
>>>>>>> AIRFLOW-1075|Cleanup security docs                          
  |#2222
>>>>>>> |5a6f18
>>>>>>> AIRFLOW-1065|Add functionality for Azure Blob Storage       
  |#2216
>>>>>>> |f1bc5f
>>>>>>> AIRFLOW-1059|Reset_state_for_orphaned_task should operate in
ba|#2205
>>>>>>> |e05d3b
>>>>>>> AIRFLOW-1058|Improvements for SparkSubmitOperator           
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1051|Add a test for resetdb to CliTests             
  |#2198
>>>>>>> |15aee0
>>>>>>> AIRFLOW-1047|Airflow logs vulnerable to XSS                 
  |#2193
>>>>>>> |fe9ebe
>>>>>>> AIRFLOW-1045|Make log level configurable via airflow.cfg    
  |#2191
>>>>>>> |e739a5
>>>>>>> AIRFLOW-1043|Documentation issues for operators             
  |#2188
>>>>>>> |b55f41
>>>>>>> AIRFLOW-1041|DockerOperator replaces its xcom_push method with
|#2274
>>>>>>> |03704c
>>>>>>> AIRFLOW-1040|Fix typos in comments/docstrings in models.py  
  |#2174
>>>>>>> |d8c0f5
>>>>>>> AIRFLOW-1036|Exponential backoff should use randomization   
  |#2262
>>>>>>> |66168e
>>>>>>> AIRFLOW-1035|Exponential backoff retry logic should use 2 as
ba|#2196
>>>>>>> |4ec932
>>>>>>> AIRFLOW-1034|Make it possible to connect to S3 in sigv4 regions|#2181
>>>>>>> |4c0905
>>>>>>> AIRFLOW-1031|'scheduled__' may replace with DagRun.ID_PREFIX
in|#2613
>>>>>>> |aa3844
>>>>>>> AIRFLOW-1030|HttpHook error when creating HttpSensor        
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-1028|Databricks Operator for Airflow                
  |#2202
>>>>>>> |53ca50
>>>>>>> AIRFLOW-1024|Handle CeleryExecutor errors gracefully        
  |#2355
>>>>>>> |7af20f
>>>>>>> AIRFLOW-1018|Scheduler DAG processes can not log to stdout  
  |#2728
>>>>>>> |ef775d
>>>>>>> AIRFLOW-1016|Allow HTTP HEAD request method on HTTPSensor   
  |#2175
>>>>>>> |4c41f6
>>>>>>> AIRFLOW-1010|Add a convenience script for signing           
  |#2169
>>>>>>> |a2b65a
>>>>>>> AIRFLOW-1009|Remove SQLOperator from Concepts page          
  |#2168
>>>>>>> |7d1144
>>>>>>> AIRFLOW-1007|Jinja sandbox is vulnerable to RCE             
  |#2184
>>>>>>> |daa281
>>>>>>> AIRFLOW-1005|Speed up Airflow startup time                  
  |#na
>>>>>>> |996dd3
>>>>>>> AIRFLOW-999 |Support for Redis database                     
  |#2165
>>>>>>> |8de850
>>>>>>> AIRFLOW-997 |Change setup.cfg to point to Apache instead of Max|#na
>>>>>>> |75cd46
>>>>>>> AIRFLOW-995 |Update Github PR template                      
  |#2163
>>>>>>> |b62485
>>>>>>> AIRFLOW-994 |Add MiNODES to the AIRFLOW Active Users List   
  |#2159
>>>>>>> |ca1623
>>>>>>> AIRFLOW-991 |Mark_success while a task is running leads to fail|-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-990 |DockerOperator fails when logging unicode string
 |#2155
>>>>>>> |6bbf54
>>>>>>> AIRFLOW-988 |SLA Miss Callbacks Are Repeated if Email is Not
be|#2415
>>>>>>> |6e74d4
>>>>>>> AIRFLOW-985 |Extend the sqoop operator/hook with additional par|#2177
>>>>>>> |82eb20
>>>>>>> AIRFLOW-984 |Subdags unrecognized when subclassing SubDagOperat|#2152
>>>>>>> |a8bd16
>>>>>>> AIRFLOW-979 |Add GovTech GDS                                
  |#2149
>>>>>>> |b17bd3
>>>>>>> AIRFLOW-976 |Mark success running task causes it to fail    
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-969 |Catch bad python_callable argument at DAG construc|#2142
>>>>>>> |12901d
>>>>>>> AIRFLOW-963 |Some code examples are not rendered in the airflow|#2139
>>>>>>> |f69c1b
>>>>>>> AIRFLOW-960 |Add support for .editorconfig                  
  |#na
>>>>>>> |f5cacc
>>>>>>> AIRFLOW-959 |.gitignore file is disorganized and incomplete 
  |#na
>>>>>>> |3d3c14
>>>>>>> AIRFLOW-958 |Improve tooltip readability                    
  |#2134
>>>>>>> |b3c3eb
>>>>>>> AIRFLOW-950 |Missing AWS integrations on documentation::integra|#2552
>>>>>>> |01be02
>>>>>>> AIRFLOW-947 |Make PrestoHook surface better messages when the
P|#na
>>>>>>> |6dd4b3
>>>>>>> AIRFLOW-945 |Revert psycopg2 workaround when psycopg2 2.7.1 is
|-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-943 |Add Digital First Media to the Airflow users list
|#2115
>>>>>>> |2cfe28
>>>>>>> AIRFLOW-942 |Add mytaxi to Airflow Users                    
  |#2111
>>>>>>> |d579e6
>>>>>>> AIRFLOW-935 |Impossible to use plugin executors             
  |#2120
>>>>>>> |08a784
>>>>>>> AIRFLOW-926 |jdbc connector is broken due to jaydebeapi api upd|#2651
>>>>>>> |07ed29
>>>>>>> AIRFLOW-917 |Incorrectly formatted failure status message   
  |#2109
>>>>>>> |b8164c
>>>>>>> AIRFLOW-916 |Fix ConfigParser deprecation warning           
  |#2108
>>>>>>> |ef6dd1
>>>>>>> AIRFLOW-911 |Add colouring and profiling info on tests      
  |#2106
>>>>>>> |4f52db
>>>>>>> AIRFLOW-903 |Add configuration setting for default DAG view.
  |#2103
>>>>>>> |cadfae
>>>>>>> AIRFLOW-896 |BigQueryOperator fails to execute with certain inp|#2097
>>>>>>> |2bceee
>>>>>>> AIRFLOW-891 |Webserver Clock Should Include Day             
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-889 |Minor error in the docstrings for BaseOperator.
  |#2084
>>>>>>> |50702d
>>>>>>> AIRFLOW-887 |Add compatibility with future v0.16            
  |#na
>>>>>>> |50902d
>>>>>>> AIRFLOW-886 |Pass Operator result to post_execute hook      
  |#na
>>>>>>> |4da361
>>>>>>> AIRFLOW-885 |Add Change.org to the list of Airflow users    
  |#2089
>>>>>>> |a279be
>>>>>>> AIRFLOW-882 |Code example in docs has unnecessary DAG>>Operator|#2088
>>>>>>> |baa4cd
>>>>>>> AIRFLOW-881 |Create SubDagOperator within DAG context manager
w|#2087
>>>>>>> |0ed608
>>>>>>> AIRFLOW-880 |Fix remote log functionality inconsistencies for
W|#2086
>>>>>>> |974b75
>>>>>>> AIRFLOW-877 |GoogleCloudStorageDownloadOperator: template_ext
c|#2083
>>>>>>> |debc69
>>>>>>> AIRFLOW-875 |Allow HttpSensor params to be templated        
  |#2080
>>>>>>> |62f503
>>>>>>> AIRFLOW-871 |multiple places use logging.warn() instead of warn|#2082
>>>>>>> |21d775
>>>>>>> AIRFLOW-866 |Add FTPSensor                                  
  |#2070
>>>>>>> |5f87f8
>>>>>>> AIRFLOW-863 |Example DAG start dates should be recent to avoid
|#2068
>>>>>>> |bbfd43
>>>>>>> AIRFLOW-862 |Add DaskExecutor                               
  |#2067
>>>>>>> |6e2210
>>>>>>> AIRFLOW-860 |Circular module dependency prevents loading of cus|-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-854 |Add Open Knowledge International to Airflow users
|#2061
>>>>>>> |51a311
>>>>>>> AIRFLOW-842 |scheduler.clean_dirty raises warning: SAWarning:
T|#2072
>>>>>>> |485280
>>>>>>> AIRFLOW-840 |Python3 encoding issue in Kerberos             
  |#2158
>>>>>>> |639336
>>>>>>> AIRFLOW-836 |The paused and queryview endpoints are vulnerable
|#2054
>>>>>>> |6aca2c
>>>>>>> AIRFLOW-831 |Fix broken unit tests                          
  |#2050
>>>>>>> |b86194
>>>>>>> AIRFLOW-830 |Plugin manager should log to debug, not info   
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-829 |Reduce verbosity of successful Travis unit tests
 |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-826 |Add Zendesk Hook                               
  |#2066
>>>>>>> |a09762
>>>>>>> AIRFLOW-823 |Make task instance details available via API   
  |#2045
>>>>>>> |3f546e
>>>>>>> AIRFLOW-822 |Close the connection before throwing exception in
|#2038
>>>>>>> |4b6c38
>>>>>>> AIRFLOW-821 |Scheduler dagbag importing not Py3 compatible  
  |#2039
>>>>>>> |fbb59b
>>>>>>> AIRFLOW-809 |SqlAlchemy is_ ColumnOperator Causing Errors in
MS|-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-802 |Integration of spark-submit                    
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-781 |Allow DataFlowJavaOperator to accept jar file stor|#2037
>>>>>>> |259c86
>>>>>>> AIRFLOW-770 |HDFS hooks should support alternative ways of gett|#2056
>>>>>>> |261b65
>>>>>>> AIRFLOW-756 |Refactor ssh_hook and ssh_operator             
  |-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-751 |SFTP file transfer functionality               
  |#1999
>>>>>>> |fe0ede
>>>>>>> AIRFLOW-725 |Make merge tool use OS' keyring for password stora|#1966
>>>>>>> |8c1695
>>>>>>> AIRFLOW-706 |Configuration shell commands are not split properl|#2053
>>>>>>> |0bb6f2
>>>>>>> AIRFLOW-705 |airflow.configuration.run_command output does not
|-
>>>> |-
>>>>>>> 
>>>>>>> AIRFLOW-681 |homepage doc link should pointing to apache's repo|#2164
>>>>>>> |a8027a
>>>>>>> AIRFLOW-654 |SSL for AMQP w/ Celery(Executor)               
  |#2333
>>>>>>> |868bfe
>>>>>>> AIRFLOW-645 |HttpHook ignores https                         
  |#2311
>>>>>>> |fd381a
>>>>>>> AIRFLOW-365 |Code view in subdag trigger exception          
  |#2043
>>>>>>> |cf102c
>>>>>>> AIRFLOW-300 |Add Google Pubsub hook and operator            
  |#2036
>>>>>>> |d231dc
>>>>>>> AIRFLOW-289 |Use datetime.utcnow() to keep airflow system indep|#2618
>>>>>>> |20c83e
>>>>>>> AIRFLOW-71  |docker_operator - pulling from private repositorie|#na
>>>>>>> |d4406c
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> Chris
>>>>>>> 
>>>>> 
>>>> 
>>>> 
>> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message