airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxime Beauchemin <maximebeauche...@gmail.com>
Subject Re: Release Manager for 1.8.2?
Date Wed, 14 Jun 2017 04:26:44 GMT
A quick update on my progress.

I ran through the "Releasing Airflow" wiki playbook only to realize I had
published a version that failed the Travis build
<https://travis-ci.org/mistercrunch/incubator-airflow/jobs/242532965>
afterwards. I've
been updating the wiki as I go and it seems like it will be better the next
time around. I'm getting "Permission denied" on the Hive hook subprocess if
anyone can shed some light on that.

So I need to fix the unit tests at this point, and probably go straight to
`rc2` to avoid changing the files out there for `rc1`.

Max

On Thu, Jun 8, 2017 at 4:54 PM, Ruslan Dautkhanov <dautkhanov@gmail.com>
wrote:

> It would be great if somebody would have a look at following 3 jiras
>
> I've flagged
> https://issues.apache.org/jira/browse/AIRFLOW-1013
> https://issues.apache.org/jira/browse/AIRFLOW-1178
> https://issues.apache.org/jira/browse/AIRFLOW-1055
>
> Two of them were Blockers for one of previous 1.8 releases but were
> de-escalated because no resources to fix them.
>
> I'm currently Assignee on AIRFLOW-1055 - can't unassign myself - feel free
> to scratch my name there.
>
>
>
>
> --
> Ruslan Dautkhanov
>
> On Fri, Jun 9, 2017 at 12:23 AM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > Cool. I'll let everyone know when RC1 is out.
> >
> > @community, if you think anything else should be a blocker for 1.8.2
> please
> > flag it in Jira very soon or it will miss the release train!
> >
> > Max
> >
> > On Thu, Jun 8, 2017 at 9:06 AM, Chris Riccomini <criccomini@apache.org>
> > wrote:
> >
> > > Thanks for doing all this, Max! When you're ready, I can deploy to some
> > of
> > > our environments to verify. Just let me know.
> > >
> > > On Thu, Jun 8, 2017 at 8:42 AM, Maxime Beauchemin <
> > > maximebeauchemin@gmail.com> wrote:
> > >
> > > > I branched off `v1-8-test` so all should be good. I just didn't know
> > if I
> > > > could move ahead with that branch just yet so I branched off. I just
> > got
> > > > back on `v1-8-test` and pushed what I have to Apache [somehow I had
> to
> > > > rebase meaning someone added something over the past 24 hours].
> > > >
> > > > I just set `Fix Version` of AIRFLOW-1294 to 1.8.2 and set it to
> > blocker.
> > > >
> > > > I cherry-picked all the commits targeting 1.8.2 I could without
> getting
> > > > into conflicts. I'd only work at resolving conflicts on blockers. My
> > plan
> > > > is to add only the 2 rows in red for RC1.
> > > >
> > > > Here's the current output of `airflow-jira 1.8.2`:
> > > >
> > > > ISSUE ID          |TYPE        ||PRIORITY  ||STATUS    |DESCRIPTION
> > > >                                    |MERGED|PR    |COMMITAIRFLOW-1294
> > > >    |Bug         ||Blocker   ||Open      |Backfills can loose tasks to
> > > > execute due to tasks |0     |-     |-
> > > > AIRFLOW-1291      |Bug         ||Blocker     ||Open      |Update
> > > > NOTICE and LICENSE files to meet ASF specs |0     |-     |-
> > > > AIRFLOW-1290      |Bug         ||Major     ||Resolved  |Change docs
> > > > author from "Maxime Beauchemin" to "Ap|1     |#na
> > > > |fc33c040605e7b0121739af841790336bdfa3948
> > > > AIRFLOW-1282      |Bug         ||Major     ||Resolved  |Fix known
> > > > event column sorting                    |1     |#2350
> > > > |2e9eee3697f6261de61c77ba584ace14b22751f0
> > > > AIRFLOW-1281      |Bug         ||Minor     ||Resolved  |Sort
> variables
> > > > by key field by default            |1     |#2347
> > > > |57d5bcdad4202fd8a94be07b4aeacee065801481
> > > > AIRFLOW-1275      |Bug         ||Major     ||In Progress|Fix `airflow
> > > > pool` command exception              |0     |-     |-
> > > > AIRFLOW-1274      |Bug         ||Major     ||Open      |HttpSensor
> > > > parameter params is overriding BaseOper|0     |-     |-
> > > > AIRFLOW-1266      |Bug         ||Major     ||Open      |Long task
> > > > names are truncated in gannt view       |1     |#2345
> > > > |7f33f6e2fe4149e547860fdb237da26a5b7023ec
> > > > AIRFLOW-1263      |Bug         ||Major     ||Resolved  |Airflow
> Charts
> > > > Pages Should Have Dynamic Heights  |1     |#2344
> > > > |acd0166d8c7f5639898791848d985f6441b3b5b5
> > > > AIRFLOW-1244      |Bug         ||Major     ||Resolved  |Forbid
> > > > creation of a pool with empty name         |1     |#2324
> > > > |802fc1549d0f173c646794f2476156d34383adc0
> > > > AIRFLOW-1243      |Bug         ||Minor     ||Resolved  |DAGs table
> has
> > > > no default entries to show         |1     |#2323
> > > > |1232b6a4fd691e0faf081d4cf792a6d96d33b595
> > > > AIRFLOW-1227      |Bug         ||Minor     ||Resolved  |Remove empty
> > > > column on the Logs view              |1     |#2310
> > > > |b0ba3c91044b2abd92289d9b3f7d913060ef6c31
> > > > AIRFLOW-1226      |Bug         ||Minor     ||Resolved  |Remove empty
> > > > column on the Jobs view              |1     |#2309
> > > > |c406652bd4c8f8099841c69f2d1ccc9678d77bfd
> > > > AIRFLOW-1221      |Bug         ||Major     ||Resolved  |Fix
> > > > DatabricksSubmitRunOperator Templating        |0     |-     |-
> > > > AIRFLOW-1217      |Bug         ||Major     ||Resolved  |Enable
> logging
> > > > in Sqoop hook                      |0     |-     |-
> > > > AIRFLOW-1213      |Bug         ||Major     ||Resolved  |Add hcatalog
> > > > parameters to the sqoop operator/hook|0     |-     |-
> > > > AIRFLOW-1203      |Bug         ||Major     ||Closed    |Tests failing
> > > > after oauth upgrade                 |0     |-     |-
> > > > AIRFLOW-1201      |Bug         ||Major     ||Resolved  |Update
> > > > deprecated 'nose-parameterized' library to |0     |-     |-
> > > > AIRFLOW-1200      |Bug         ||Major     ||Resolved  |Forbid
> > > > creation of a variable with an empty key   |1     |#2299
> > > > |cac7d4cb571a4c1f7ba53360765ea6c640c719df
> > > > AIRFLOW-1199      |Bug         ||Major     ||Resolved  |Create modal
> > > > looks like edit modal (edit tab selec|1     |#2293
> > > > |51a83cce96d41eb1b9df9f82188c36c7e12061df
> > > > AIRFLOW-1197      |Bug         ||Major     ||Open      |Contrib Spark
> > > > Submit Hook on_kill error           |0     |-     |-
> > > > AIRFLOW-1192      |New Feature ||Major     ||Resolved  |Add support
> to
> > > > run Zeppelin Notebooks with Qubole |1     |#2322
> > > > |bf966e63be2deb19fa2824efdfe854736318f1b5
> > > > AIRFLOW-1191      |Improvement ||Major     ||Open      |Contrib Spark
> > > > Submit hook should permit override o|0     |-     |-
> > > > AIRFLOW-1186      |Bug         ||Major     ||Resolved  |Task duration
> > > > shows weird graph for some dags     |1     |#2284
> > > > |5f3ee52da7ca4f1fca8a84b3281b31aa6241514e
> > > > AIRFLOW-1180      |Bug         ||Major     ||Resolved  |Fix flask-wtf
> > > > version in setup.py for test_csrf_re|1     |#2280
> > > > |93b8e9643cc273ec0ecc063b9f851d9a8012cd03
> > > > AIRFLOW-1179      |Bug         ||Major     ||Closed    |Pandas 0.20
> > > > broke Google BigQuery hook            |1     |#2279
> > > > |c7c0b72d934fec10d890f7a3bd8fd06c0e2f44bd
> > > > AIRFLOW-1170      |Bug         ||Major     ||Resolved  |DbApiHook
> > > > insert rows does not use prepared statem|1     |#2270
> > > > |bb56805e3801c7f8d49acd8095fd3099609d6fe5
> > > > AIRFLOW-1168      |Bug         ||Major     ||Resolved  |DbApiHook
> does
> > > > not close connections when an excep|1     |#2269
> > > > |777f181f81fa6a383a79b71fdcb9c4f4e77c785a
> > > > AIRFLOW-1167      |Bug         ||Major     ||Resolved  |FTPHook file
> > > > modification date does not support op|1     |#2268
> > > > |272c2f52142c167685646169431eb9dab85b7810
> > > > AIRFLOW-1166      |Bug         ||Major     ||Resolved
> > > > |_change_state_for_tis_without_dagrun can take very|1     |#2267
> > > > |2389a8a7e89ea949e8f9674a3d3b18d2142dfece
> > > > AIRFLOW-1160      |Bug         ||Major     ||Resolved  |Upadte
> > > > SparkSubmitOperator parameters             |1     |#2265
> > > > |69bd26913824c5c796a0a828a5c43ab39d0902f8
> > > > AIRFLOW-1150      |Bug         ||Major     ||Resolved  |Fix scripts
> > > > execution in SparkSql hook            |1     |#2259
> > > > |093b2f04f7a010107fe6dfbc0658a53a1b8467e6
> > > > AIRFLOW-1149      |Bug         ||Major     ||Resolved  |Allow custom
> > > > filters to be added to jinja2        |1     |#2258
> > > > |9692510c4e067d0e81fd8d5893bfb4a0e53941e2
> > > > AIRFLOW-1145      |Bug         ||Minor     ||Resolved
> > > > |Closest_date_partition not working with before = T|1     |#2257
> > > > |f446c0883cab30482436c47b867c4a440412f1ff
> > > > AIRFLOW-1136      |Bug         ||Minor     ||Resolved  |Invalid
> > > > parameters are not captured for Sqoop oper|0     |-     |-
> > > > AIRFLOW-1119      |Bug         ||Major     ||Resolved  |Redshift to
> S3
> > > > operator - headers not on first row|1     |#2245
> > > > |6de5330d5336fb2455eea75c78e95f126320a5b0
> > > > AIRFLOW-1089      |Improvement ||Major     ||Resolved  |Add Spark
> > > > application arguments to SparkSubmitOper|1     |#2229
> > > > |b4e9eb8310957cac47f6964662ce16496061db2c
> > > > AIRFLOW-1081      |Bug         ||Major     ||Resolved  |Task duration
> > > > page is slow                        |0     |-     |-
> > > > AIRFLOW-1078      |Bug         ||Major     ||Resolved  |Latest_runs
> > > > endpoint broken in old flask versions |1     |#2224
> > > > |a4083f367b1f0f6525bcd011122d854626ea6768
> > > > AIRFLOW-1074      |Bug         ||Major     ||Open      |Do not count
> > > > queued tasks in scheduler concurrency|1     |#2221
> > > > |7a0284149a1dde4dc1d65a1d016eadf619e5320e
> > > > AIRFLOW-1064      |Improvement ||Major     ||Resolved
> > > > |TaskInstanceModelView is slow                     |1     |#2215
> > > > |a2c18a58d0ba14101ddb70403fb95feda73cf21a
> > > > AIRFLOW-1047      |Bug         ||Major     ||Resolved  |Airflow logs
> > > > vulnerable to XSS                    |0     |-     |-
> > > > AIRFLOW-1041      |Bug         ||Major     ||Resolved
> |DockerOperator
> > > > replaces its xcom_push method with |0     |-     |-
> > > > AIRFLOW-1038      |Bug         ||Major     ||Resolved  |Specify
> celery
> > > > serializers explicitly and pin vers|1     |#na
> > > > |d1c64ab7b4ee6215900a15849e890ed1b408f5f5
> > > > AIRFLOW-1036      |Improvement ||Major     ||Resolved  |Exponential
> > > > backoff should use randomization      |1     |#2262
> > > > |b4ee88aa6d71145daf91fc71763950ede16f8383
> > > > AIRFLOW-1007      |Bug         ||Major     ||Resolved  |Jinja sandbox
> > > > is vulnerable to RCE                |0     |-     |-
> > > > AIRFLOW-993       |Bug         ||Minor     ||Resolved  |Dags should
> > > > modify the start date and end date of |1     |#2157
> > > > |9fca409cebfd04ce8580c4657099a6db340eb7eb
> > > > AIRFLOW-945       |Bug         ||Major     ||Resolved  |Revert
> > > > psycopg2 workaround when psycopg2 2.7.1 is |0     |-     |-
> > > > AIRFLOW-935       |Bug         ||Major     ||Resolved  |Impossible to
> > > > use plugin executors                |0     |-     |-
> > > > AIRFLOW-860       |Bug         ||Major     ||Resolved  |Circular
> > > > module dependency prevents loading of cus|0     |-     |-
> > > >
> > > >
> > > > Max
> > > >
> > > > On Thu, Jun 8, 2017 at 12:18 AM, Bolke de Bruin <bdbruin@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Max,
> > > > >
> > > > > Sounds good. Couple of things:
> > > > >
> > > > > * Can I suggest using the v1-8-test branch as the branch to be used
> > for
> > > > > preparing the rc? If we hit RC then move it over to v1-8-stable?
> > > > V1-8-test
> > > > > already had some fixes in that should land in 1.8.2 and the RC
> should
> > > be
> > > > > tagged in the stable branch. That also reduces to amount of merge
> > > > conflicts
> > > > > probably as many have been merged. Where did you branch off from?
> > > > Anyways,
> > > > > see also the release management thing on the wiki.
> > > > >
> > > > > Blocker(!)
> > > > > * In the backfills we can loose tasks to execute due to a task
> > setting
> > > > its
> > > > > own state to NONE if concurrency limits are reached, this makes
> them
> > > fall
> > > > > outside of the scope the backfill is managing hence they will not
> be
> > > > > executed (https://issues.apache.org/jira/browse/AIRFLOW-1294 <
> > > > > https://issues.apache.org/jira/browse/AIRFLOW-1294>). Setting
> itself
> > > to
> > > > > NONE should probably be “CONCURRENCY_REACHED” (new state). I
have
> > > marked
> > > > it
> > > > > as a blocker as we had multiple people hitting the issue, but I
> need
> > > 1-2
> > > > > days to get a patch. Feel free to downgrade to critical if you like
> > :).
> > > > >
> > > > > Cheers
> > > > > Bolke
> > > > >
> > > > >
> > > > > > On 8 Jun 2017, at 02:35, Maxime Beauchemin <
> > > maximebeauchemin@gmail.com
> > > > >
> > > > > wrote:
> > > > > >
> > > > > > What a pleasant, mind numbing afternoon doing some release
> > management
> > > > > >
> > > > > > Notes:
> > > > > > * Added a warning that the package name has changed on Pypi
> > > > > > <https://pypi.python.org/pypi/airflow>
> > > > > > * Removed references to my name here
> > > > > > <https://github.com/apache/incubator-airflow/pull/2352>
and
> merged
> > > > > > * Addressed John D. Ament's concerns here
> > > > > > <https://github.com/apache/incubator-airflow/pull/2354>,
please
> > > > review!
> > > > > > * "footable" appears to have been removed, not a problem anymore
> > > > > > * that `airflow-jira is a god send! thanks Bolke.
> > > > > > * reviewed list of Airbnb's production cherries and flagged
those
> > as
> > > > `Fix
> > > > > > Version == 1.8.2`
> > > > > > * Started branch v1-8-2.rc1 and started picking cherries using
> > > > > > `airflow-jira compare 1.8.2`
> > > > > >
> > > > > > I'll finish going through picking everything that targeted 1.8.2
> > that
> > > > > does
> > > > > > not create merge conflict.
> > > > > >
> > > > > > If there's anything flagged as "blocker" that generates merge
> > > conflict,
> > > > > > I'll go case by case about it.
> > > > > >
> > > > > > Soon after, I should be able to announce 1.8.2 RC1, hopefully
> > > sometime
> > > > > > tomorrow or Friday.
> > > > > >
> > > > > > Let me know if there's anything else I'm missing that I should
> > > > consider.
> > > > > >
> > > > > > Cheers!
> > > > > >
> > > > > > Max
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message