aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ASF IRC Bot <asf...@urd.zones.apache.org>
Subject Summary of IRC Meeting in #aurora
Date Mon, 05 Oct 2015 18:39:48 GMT
Summary of IRC Meeting in #aurora at Mon Oct  5 18:05:43 2015:

Attendees: dchung, wickman, jcohen, kts, mkhutornenko, rdelvalle, zmanji, dlester

- Preface
- AURORA-1503
- AURORA-1504
  - Action: wickman to propose json input format on dev list
- AURORA-1376


IRC log follows:

## Preface ##
[Mon Oct  5 18:06:14 2015] <zmanji>: I have a topic if no one else has one
[Mon Oct  5 18:06:19 2015] <jcohen>: Meeting time everyone. As always, everyone is welcome
to participate.
[Mon Oct  5 18:06:25 2015] <jcohen>: Let's start with roll call.
[Mon Oct  5 18:06:32 2015] <dlester>: present :D
[Mon Oct  5 18:06:33 2015] <jcohen>: here.
[Mon Oct  5 18:06:37 2015] <rdelvalle>: present
[Mon Oct  5 18:06:40 2015] <zmanji>: here
[Mon Oct  5 18:06:58 2015] <dchung>: David Chung here.
[Mon Oct  5 18:07:36 2015] <kts>: here
[Mon Oct  5 18:07:40 2015] <mkhutornenko>: here
[Mon Oct  5 18:07:50 2015] <wickman>: here
[Mon Oct  5 18:08:08 2015] <jcohen>: zmanji: you said you had a topic to kick us off?
## AURORA-1503 ##
[Mon Oct  5 18:08:22 2015] <jcohen>: AURORA-1503
[Mon Oct  5 18:08:35 2015] <jcohen>: excellent topic!
[Mon Oct  5 18:08:44 2015] <zmanji>: as some of us might be aware of the Mesos release
frequency has increased to about one release a month
[Mon Oct  5 18:09:02 2015] <zmanji>: but the deprecation policy has not changed
[Mon Oct  5 18:09:21 2015] <zmanji>: This means Aurora 0.9.0 (linked against Mesos 0.22)
does not work against a Mesos master 0.24
[Mon Oct  5 18:09:30 2015] <jcohen>: can you clarify what the deprecation policy is
for those not aware
[Mon Oct  5 18:09:53 2015] <jcohen>: I believe it's +/- 1 version?
[Mon Oct  5 18:10:00 2015] <zmanji>: Yes, it is +/- 1 version
[Mon Oct  5 18:10:13 2015] <jcohen>: but they're considering moving to a time-based
deprecation policy instead?
[Mon Oct  5 18:10:34 2015] <zmanji>: There is some discussion around a time based deprecation
policy, but there is no clarity on when that will occur
[Mon Oct  5 18:10:55 2015] <zmanji>: Going back to the problem in AURORA-1503, we ideally
solve two problems.
[Mon Oct  5 18:11:08 2015] <zmanji>: 1. Allowing our users to upgrade mesos installations
while still running 0.9.x
[Mon Oct  5 18:11:16 2015] <zmanji>: 2. Figure out an upgrade story for 0.10.0
[Mon Oct  5 18:11:33 2015] <zmanji>: I currently have a plan that I would like to share.
I will put up an email on the mailing list later on this week.
[Mon Oct  5 18:12:00 2015] <zmanji>: I think we can release 0.9.1 linked against mesos
0.23. And then release 0.10.0 against mesos 0.24.
[Mon Oct  5 18:12:38 2015] <zmanji>: AFAIK, 0.9.1 (with 0.23) will not disrupt existing
installations running 0.22 and provides a clean upgrade path for our users to 0.24
[Mon Oct  5 18:12:43 2015] <zmanji>: That's all I got
[Mon Oct  5 18:13:29 2015] <jcohen>: I think this probably warrants discussion on the
list as I'm not sure what we consider worthy of a full version bump
[Mon Oct  5 18:13:41 2015] <jcohen>: (i.e. should we release 0.10.0 that does nothing
other than bump mesos version)
[Mon Oct  5 18:13:55 2015] <kts>: i feel like we should err on the side of a full version
bump here
[Mon Oct  5 18:14:01 2015] <mkhutornenko>: +1 to dev list, I also have concerns about
feature fragmentation due to that
[Mon Oct  5 18:14:33 2015] <zmanji>: Anyways, more discussion to come on the dev list
this week, be ready for that
[Mon Oct  5 18:14:42 2015] <jcohen>: thanks zmanji
[Mon Oct  5 18:14:51 2015] <jcohen>: anyone else have a topic for discussion?
[Mon Oct  5 18:14:53 2015] <wickman>: AURORA-1504
[Mon Oct  5 18:15:07 2015] <wickman>: does anybody use any of the --read-json/--write-json
client feautres?
## AURORA-1504 ##
[Mon Oct  5 18:15:29 2015] <wickman>: or specifically, does anyone write configs using
json?
[Mon Oct  5 18:16:29 2015] <wickman>: if the answer is "no", then i'd like to make the
client better at going between .aurora<->json
[Mon Oct  5 18:16:30 2015] <jcohen>: to be clear, this is distinct from the option of
the client outputting json for the sake of parseability in some other orchestrating toolchain?
[Mon Oct  5 18:16:35 2015] <wickman>: maybe through a separate subcommand
[Mon Oct  5 18:16:41 2015] <wickman>: or through the config subcommand
[Mon Oct  5 18:16:46 2015] <wickman>: right
[Mon Oct  5 18:16:49 2015] <wickman>: this is mostly for configs
[Mon Oct  5 18:17:04 2015] <kts>: wickman: iiuc the schema to do this already exists
and is protected by backwards-compatibility guarantees
[Mon Oct  5 18:17:10 2015] <kts>: i.e. the pystachio schema
[Mon Oct  5 18:17:29 2015] <kts>: so allowing json as an input format is uncontroversial
to me
[Mon Oct  5 18:17:31 2015] <wickman>: 'schema' you mean the client contract?
[Mon Oct  5 18:17:54 2015] <wickman>: but i'm proposing that we change the input format
of --read-json
[Mon Oct  5 18:17:57 2015] <kts>: the .aurora file format has a schema
[Mon Oct  5 18:18:16 2015] <kts>: oh how so?
[Mon Oct  5 18:18:17 2015] <wickman>: since it needs to read more than just the information
defined by the Job schema
[Mon Oct  5 18:18:23 2015] <wickman>: specifically metadata fields for JobConfiguration
[Mon Oct  5 18:18:38 2015] <wickman>: like those generated by binding helpers
[Mon Oct  5 18:18:52 2015] <wickman>: either that or we just *add* metadata Map(String,String)
to the Job schema
[Mon Oct  5 18:18:57 2015] <wickman>: and have binding helpers mutate it
[Mon Oct  5 18:19:05 2015] <wickman>: that may be the best option
[Mon Oct  5 18:19:11 2015] <kts>: adding that map seems like a good option
[Mon Oct  5 18:19:19 2015] <kts>: then the json input format can also use binding helpers
[Mon Oct  5 18:19:42 2015] <kts>: for example the (currently future hypothetical) docker-resolve-to-sha
one
[Mon Oct  5 18:20:13 2015] <wickman>: right.
[Mon Oct  5 18:20:40 2015] <kts>: does --read-json currently bypass binding helpers?
[Mon Oct  5 18:20:50 2015] <wickman>: i don't even know if --read-json works at all
tbh
[Mon Oct  5 18:21:09 2015] <kts>: and if nobody uses it anyway then it's a moot discussion
[Mon Oct  5 18:21:11 2015] <wickman>: it's unclear to me that tests even exist either
[Mon Oct  5 18:21:41 2015] <zmanji>: We can just remove --read-json
[Mon Oct  5 18:22:08 2015] <kts>: well it sounds like wickman is a user wanting to use
it
[Mon Oct  5 18:22:12 2015] <wickman>: i want to use it
[Mon Oct  5 18:22:27 2015] <wickman>: or something that allows me to input a job as
job.json instead of <jobkey> <aurorafile>
[Mon Oct  5 18:23:09 2015] <rdelvalle>: this sounds in line with the REST API changes
that have been proposed
[Mon Oct  5 18:23:18 2015] <wickman>: this is ubiquitous with all other cloud offerings,
e.g. kubernetes, cloudformation, hashicorp's new thing etc
[Mon Oct  5 18:23:48 2015] <jcohen>: Yeah, it would definitely be nice for the REST
api to be able to deal with a JSON representation of a job config (and have it be the client's
responsibility to translate from .aurora to JSON)
[Mon Oct  5 18:23:52 2015] <wickman>: yes, being able to use the client to suck in .aurora
+ jobkey and spit out json will be essential as we move towards a REST API
[Mon Oct  5 18:24:04 2015] <wickman>: the 'config' subcommand seems appropriate for
this
[Mon Oct  5 18:24:30 2015] <jcohen>: Nothing here sounds particularly controversial
to me.
[Mon Oct  5 18:24:38 2015] <wickman>: ok.
[Mon Oct  5 18:25:10 2015] <kts>: so we have discussed a need for "strict mode"
[Mon Oct  5 18:25:55 2015] <kts>: it seems like we might want a way to take a json file
with unbound bindings and return a static one
[Mon Oct  5 18:26:32 2015] <wickman>: yes, --bind is still relevant for reading json
blobs
[Mon Oct  5 18:26:50 2015] <wickman>: the client would still do Job.reads_json(open(filename))
[Mon Oct  5 18:27:02 2015] <wickman>: meaning that bindings can still be made, and you
can still reference {{mustache}} variables
[Mon Oct  5 18:27:31 2015] <wickman>: in fact, .json is, for all intents and purposes,
a manifestation of strictm ode
[Mon Oct  5 18:27:42 2015] <kts>: so im proposing we have a mode that will take a json
with {{package.version[example]}} and return one with 42
[Mon Oct  5 18:28:02 2015] <wickman>: yes, binding helpers should also still work in
this model
[Mon Oct  5 18:28:06 2015] <wickman>: assuming they're linked into the client
[Mon Oct  5 18:28:15 2015] <kts>: so input is a json file or a .aurora file
[Mon Oct  5 18:28:23 2015] <wickman>: (as an aside, we should make it easier to add
binding helpers as plugins from the environment)
[Mon Oct  5 18:28:27 2015] <wickman>: input is .json
[Mon Oct  5 18:28:59 2015] <kts>: i think compiling an .aurora file to a .json file
to be deployed later is a use case we should consider here as well
[Mon Oct  5 18:29:12 2015] <zmanji>: +1 to making binding helpers as plugins for the
environment
[Mon Oct  5 18:29:19 2015] <zmanji>: s/environment/client
[Mon Oct  5 18:29:29 2015] <wickman>: that's exactly what i'm interested in -- because
i'd like to save compiled .json files and use them for rollbacks / rollforwards
[Mon Oct  5 18:29:30 2015] <kts>: it sounds like this proposed .json input format is
equivalent to the .aurora input file (less the python eval())
[Mon Oct  5 18:29:57 2015] <kts>: both of those should be compilable to static things
that facilitate roll-{forward,back}
[Mon Oct  5 18:30:03 2015] <jcohen>: it's a JSON representation of a reified Aurora
config, right?
[Mon Oct  5 18:30:13 2015] <kts>: how do you mean reified?
[Mon Oct  5 18:30:21 2015] <jcohen>: everything is bound?
[Mon Oct  5 18:30:25 2015] <kts>: it's not reified, it still supports binding helpers
[Mon Oct  5 18:30:31 2015] <wickman>: yes.  we already have this, except that the Job
schema does not completely encapsulate everything in the JobConfiguration thrift struct.
[Mon Oct  5 18:30:36 2015] <kts>: a json file with {{package.version}} is a valid input
format
[Mon Oct  5 18:30:41 2015] <kts>: and will still run the binding helper for that
[Mon Oct  5 18:30:48 2015] <jcohen>: wickman if you want to use it for rollbacks wouldn't
you want it to be fully reified?
[Mon Oct  5 18:30:54 2015] <kts>: but im proposing we have an output that's 0 binding
helpers
[Mon Oct  5 18:30:59 2015] <kts>: 2 different json formats
[Mon Oct  5 18:31:04 2015] <wickman>: yes, but you can still do 'config read < foo.json
> foo-compiled.json'
[Mon Oct  5 18:31:10 2015] <kts>: yeah
[Mon Oct  5 18:31:14 2015] <jcohen>: ahh
[Mon Oct  5 18:31:18 2015] <kts>: or config read < foo.aurora > foo-compiled.json
[Mon Oct  5 18:31:25 2015] <wickman>: yes, either way
[Mon Oct  5 18:31:28 2015] <kts>: +1
[Mon Oct  5 18:31:35 2015] <jcohen>: +1, sounds great to me.
[Mon Oct  5 18:31:37 2015] <wickman>: well it'd be 'config read jobkey < foo.aurora
> foo-compiled.json' technically
[Mon Oct  5 18:32:06 2015] <wickman>: then 'update start' would need to accept these
compiled json blobs
[Mon Oct  5 18:32:12 2015] <wickman>: via --read-json or whatever
[Mon Oct  5 18:32:57 2015] <mkhutornenko>: wickman: it’s not just update start though,
it’s all commands that currently take aurora config, right?
[Mon Oct  5 18:33:16 2015] <wickman>: yes, but for me the important ones are related
to updates
[Mon Oct  5 18:33:38 2015] <kts>: so i'd propose that we auto-detect json and have --strict
flag to disallow --bind and disable binding helpers
[Mon Oct  5 18:34:04 2015] <wickman>: i don't think we should go so far as to have --strict
disable binding helpers
[Mon Oct  5 18:34:12 2015] <kts>: so update start jobkey file.{aurora,json} allows --bind
[Mon Oct  5 18:34:26 2015] <wickman>: 'jobkey file.json' is redundant though
[Mon Oct  5 18:34:28 2015] <kts>: update start jobkey file.json --strict needs json
input
[Mon Oct  5 18:34:48 2015] <wickman>: perhaps we're getting too much into the weeds
here?  i think we can table this for after the meeting.
[Mon Oct  5 18:34:57 2015] <kts>: yeah we don't need to design it here
[Mon Oct  5 18:35:00 2015] <jcohen>: yeah
[Mon Oct  5 18:35:06 2015] <mkhutornenko>: wickman: agree, mind sending this to dev
list?
[Mon Oct  5 18:35:11 2015] <jcohen>: can discuss in jira or in a mailing list thread
[Mon Oct  5 18:35:14 2015] <wickman>: ok
[Mon Oct  5 18:35:14 2015] <kts>: #action wickman to propose json input format on dev
list
[Mon Oct  5 18:35:33 2015] <jcohen>: any other topics?
[Mon Oct  5 18:35:39 2015] <rdelvalle>: AURORA-1376
## AURORA-1376 ##
[Mon Oct  5 18:36:05 2015] <jcohen>: The floor is yours rdelvalle
[Mon Oct  5 18:36:34 2015] <rdelvalle>: Just real quick, finished switching to the newest
version of Jackson and could use a code review so we can get this feature in before the next
release like Bill wanted to
[Mon Oct  5 18:37:53 2015] <jcohen>: Sounds good. I believe wfarner is back imminently,
so he can hopefully pick the review back up.
[Mon Oct  5 18:38:12 2015] <rdelvalle>: Awesome
[Mon Oct  5 18:38:15 2015] <rdelvalle>: That's all from me
[Mon Oct  5 18:38:25 2015] <jcohen>: thanks rdelvalle
[Mon Oct  5 18:38:33 2015] <jcohen>: Last call for topics...
[Mon Oct  5 18:39:11 2015] <jcohen>: Ok, folks, that'll do it then!
[Mon Oct  5 18:39:34 2015] <kts>: ASFBot: meeting stop


Meeting ended at Mon Oct  5 18:39:34 2015

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message