aurora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Chu-Carroll <mchucarr...@apache.org>
Subject Re: One more try at defaults and aliases
Date Thu, 19 Jun 2014 20:48:03 GMT
Sorry, meant gradle. Back when I worked at Google, I wrote part of blaze,
so it's easy to slip and say that when I'm talking about build tools.

   -Mark


On Thu, Jun 19, 2014 at 4:45 PM, Kevin Sweeney <kevints@apache.org> wrote:

> What's blaze ;-p?
>
>
> On Thu, Jun 19, 2014 at 1:44 PM, Mark Chu-Carroll <mchucarroll@apache.org>
> wrote:
>
>> That's what I thought - but in pants/blaze/maven/..., I thought most
>> people
>> normally left their command-line in the project root. I certainly do. If
>> you're doing that - and a lot of people do - then you can't easily
>> disambiguate between different services living in the same repos.
>>
>>     -Mark
>>
>>
>> On Thu, Jun 19, 2014 at 4:40 PM, Bill Farner <wfarner@apache.org> wrote:
>>
>> > On Thu, Jun 19, 2014 at 1:36 PM, Mark Chu-Carroll <
>> mchucarroll@apache.org>
>> > wrote:
>> >
>> > > Maybe I'm misunderstanding you?
>> > >
>> > > Where else should you look for a default configuration file, where it
>> can
>> > > automatically differentiate between different services stored in the
>> same
>> > > repository?
>> > >
>> >
>> > Working directory would be most intuitive to me.
>> >
>> >
>> > >
>> > >   -Mark
>> > >
>> > >
>> > > On Thu, Jun 19, 2014 at 4:35 PM, Bill Farner <wfarner@apache.org>
>> wrote:
>> > >
>> > > > > in the root of the project directory
>> > > >
>> > > > FWIW i don't see any particular reason why it needs to be in the
>> root
>> > of
>> > > a
>> > > > repo/project dir.
>> > > >
>> > > > -=Bill
>> > > >
>> > > >
>> > > > On Thu, Jun 19, 2014 at 1:31 PM, Mark Chu-Carroll <
>> > > mchucarroll@apache.org>
>> > > > wrote:
>> > > >
>> > > > > I actually don't agree with standardizing the config file name.
>> > > > >
>> > > > > Here's the issue: if you're working with typical modern tools
-
>> > gradle,
>> > > > > maven, pants, sbt, ..., you're doing your work in the root of
a
>> > fairly
>> > > > deep
>> > > > > source hierarchy. Living in that hierarchy, there are likely
>> multiple
>> > > > > aurora job definitions.
>> > > > >
>> > > > > There's no way of just picking a default config file name that's
>> not
>> > > > going
>> > > > > to cause confusion and disruption, because the config file
>> sitting in
>> > > the
>> > > > > root of the project directory is going to be wrong for many users.
>> > > > >
>> > > > > That was the point of the home-directory fallback thing; it
>> provided
>> > > > users
>> > > > > with a way to define a default config file from outside of the
>> > > workspace,
>> > > > > so that different users could set it differently.
>> > > > >
>> > > > > In any case: I'm punting on this for now. It doesn't seem like
>> > there's
>> > > > > anything close to a consensus that this is worth doing, so I'm
>> going
>> > to
>> > > > > shelve it, and we can come back to it if the consensus changes
>> later.
>> > > > >
>> > > > >     -Mark
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Thu, Jun 19, 2014 at 4:08 PM, James Oliver <jdo500@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > +1 for standardizing the config file name
>> > > > > > On Jun 19, 2014 1:01 PM, "Bill Farner" <wfarner@apache.org>
>> wrote:
>> > > > > >
>> > > > > > > I agree with Joe, and tend to disagree with the motivation
>> (for
>> > > > > aliases,
>> > > > > > at
>> > > > > > > least).  The end-result seems to be obfuscation (i'd
likely
>> need
>> > to
>> > > > > peek
>> > > > > > in
>> > > > > > > the init file to figure out what a command does, or
what
>> command
>> > to
>> > > > > run),
>> > > > > > > and i'm doubtful it will result in converting users
away from
>> > > wrapper
>> > > > > > > scripts.
>> > > > > > >
>> > > > > > > I also feel like our configuration files are already
very
>> complex
>> > > > > (nature
>> > > > > > > of the beast), and aliases make them even more so.
>> > > > > > >
>> > > > > > > I am, however, more comfortable with the standard/assumed
>> config
>> > > file
>> > > > > > > naming (i.e. like Vagrantfile).
>> > > > > > >
>> > > > > > >
>> > > > > > > -=Bill
>> > > > > > >
>> > > > > > >
>> > > > > > > On Fri, Jun 13, 2014 at 11:27 AM, Joseph Smith <
>> > > yasumoto7@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > "to set up a configuration file”
>> > > > > > > >
>> > > > > > > > This line to me just says “we’re reinventing
wrapper
>> scripts”
>> > > where
>> > > > > > > > someone could create an “alias” like ./east.sh
and
>> ./west.sh to
>> > > do
>> > > > > > > deploys
>> > > > > > > > to various clusters. In that sense, these aliases
etc are
>> also
>> > > > > “hiding”
>> > > > > > > bad
>> > > > > > > > interfaces, just in a different way- and one which
isn’t as
>> > > > > > > > well-tested/understood as shell scripts.
>> > > > > > > >
>> > > > > > > > It seems like we’re set on putting time into
this, so if we
>> > are,
>> > > > then
>> > > > > > > > ignore me and move onward :). IF we were going
to go with an
>> > > > > approach,
>> > > > > > > this
>> > > > > > > > seems reasonable based on ways that I’ve seen
people wrap
>> their
>> > > > > > > > configs/invocations of the client.
>> > > > > > > >
>> > > > > > > > On Jun 12, 2014, at 7:50 AM, Mark Chu-Carroll
<
>> > > > > mchucarroll@apache.org>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > I'm nothing if not persistent! After feedback
from my
>> second
>> > > try
>> > > > at
>> > > > > > > > > proposing a way of doing defaults and aliases,
I've got a
>> > third
>> > > > > > draft.
>> > > > > > > > >
>> > > > > > > > > Feedback please!
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > # Defaults and Shorthands in the Aurora Client
>> > > > > > > > >
>> > > > > > > > > ## Motivation
>> > > > > > > > >
>> > > > > > > > > Most of the time, users are doing the same
thing, over and
>> > over
>> > > > > > again.
>> > > > > > > > > They're
>> > > > > > > > > working mainly on one particular service,
in one
>> particular
>> > > > > > workspace.
>> > > > > > > > But
>> > > > > > > > > they
>> > > > > > > > > need to repeat the same parameters, over
and over again,
>> and
>> > > they
>> > > > > > need
>> > > > > > > to
>> > > > > > > > > remember what those parameters are, what
order they occur
>> in,
>> > > > what
>> > > > > > > format
>> > > > > > > > > they
>> > > > > > > > > use, and other details that can be difficult
to remember.
>> > > > > > > > >
>> > > > > > > > > In order to avoid these problems, users set
up custom
>> scripts
>> > > > that
>> > > > > > make
>> > > > > > > > it
>> > > > > > > > > easy
>> > > > > > > > > for them to run commands like "`myservice
start`",
>> instead of
>> > > > > > "`aurora
>> > > > > > > > job
>> > > > > > > > >  create west/markcc/prod/myserver
>> > > > > > > > > src/main/aurora/myservice/myservice.aurora`"
>> > > > > > > > >
>> > > > > > > > > Scripts aren't necessarily a bad thing. We've
designed the
>> > > aurora
>> > > > > > > client
>> > > > > > > > to
>> > > > > > > > > be script friendly: every command has at
least an option
>> > > > supporting
>> > > > > > > > > easy-to-parse
>> > > > > > > > > output. But scripts also often cover for
problems that
>> really
>> > > > > > shouldn't
>> > > > > > > > > exist.
>> > > > > > > > >
>> > > > > > > > > Scripts get written for a variety of reasons.
Among the
>> many
>> > > > > reasons
>> > > > > > > that
>> > > > > > > > > people implement
>> > > > > > > > > scripts, there are scripts that add custom
functionality
>> for
>> > a
>> > > > > > specific
>> > > > > > > > > group of users;
>> > > > > > > > > scripts that can cover up for inadequate
or missing core
>> > > > > > functionality;
>> > > > > > > > and
>> > > > > > > > > there are
>> > > > > > > > > scripts that cover up for poor user interfaces.
>> > > > > > > > >
>> > > > > > > > > In aurora, we've worked hard to eliminate
cases where the
>> > core
>> > > > > > > > > functionality of the
>> > > > > > > > > command-line client is inadequate. But our
user interface
>> > still
>> > > > > > needs a
>> > > > > > > > lot
>> > > > > > > > > of work.
>> > > > > > > > > In particular, commands in Aurora often require
very long
>> > > > > parameters,
>> > > > > > > > which
>> > > > > > > > > users have
>> > > > > > > > > a hard time remembering. Look at the example
above - it's
>> a
>> > > > pretty
>> > > > > > > > typical
>> > > > > > > > > case. The
>> > > > > > > > > user wants to create a job. They're going
to be creating
>> the
>> > > same
>> > > > > > job,
>> > > > > > > > over
>> > > > > > > > > and over again.
>> > > > > > > > > But to run it, they need to type out 68 characters
for the
>> > two
>> > > > > > > > parameters!
>> > > > > > > > > This is a
>> > > > > > > > > prime example of the kind of case where users
will write
>> > > scripts,
>> > > > > not
>> > > > > > > to
>> > > > > > > > > provide new/special
>> > > > > > > > > functionality, nor to cover for inadequate
functionality,
>> but
>> > > > just
>> > > > > > > > because
>> > > > > > > > > it's so
>> > > > > > > > > painful to remember and correctly type out
an overly long
>> > > string
>> > > > of
>> > > > > > > > > parameters.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > To reduce this, we'd like to support a way
for users to
>> set
>> > up
>> > > a
>> > > > > > > > > configuration
>> > > > > > > > > file that defines defaults and shorthands
for their
>> everyday
>> > > > work.
>> > > > > > With
>> > > > > > > > > shorthands, a user that only works with a
single service
>> > could
>> > > > say
>> > > > > > > > "aurora
>> > > > > > > > > job
>> > > > > > > > > create", instead of needing to spell out
the full jobkey
>> and
>> > > > > > > > configuration
>> > > > > > > > > file
>> > > > > > > > > location; a user working with multiple datacenters
could
>> say
>> > > > > "`aurora
>> > > > > > > job
>> > > > > > > > > create @east`" or "`aurora job create @west`"
to select
>> the
>> > > > correct
>> > > > > > > > jobkey.
>> > > > > > > > >
>> > > > > > > > > ## Proposal: Aurora Init Files
>> > > > > > > > >
>> > > > > > > > > To allow users to customize shorthands, we'll
provide a
>> > > > > > > > > builtin capability to allow users to provide
a
>> configuration
>> > > > > > > > > file, from which their customizations will
be loaded.
>> > > > > > > > > Many applications use a simple pattern to
solve similar
>> > > problems.
>> > > > > > > > > Vagrant uses a file named "Vagrantfile";
when you run
>> > vagrant,
>> > > if
>> > > > > you
>> > > > > > > > > don't specifically tell the tool where to
find a
>> > configuration,
>> > > > it
>> > > > > > > looks
>> > > > > > > > > in the current directory or one of its parents
to find a
>> file
>> > > > named
>> > > > > > > > > "Vagrantfile".
>> > > > > > > > >
>> > > > > > > > > We'd like to follow a similar pattern, and
create an
>> > > "AuroraInit"
>> > > > > > file.
>> > > > > > > > > The aurora init file is found by searching
the following
>> > > > locations,
>> > > > > > in
>> > > > > > > > > order:
>> > > > > > > > >
>> > > > > > > > >   * the contents of the "--init-file" parameter.
>> > > > > > > > >   * if the "--init-file" parameter is unspecified,
then
>> look
>> > in
>> > > > the
>> > > > > > > > current
>> > > > > > > > >     directory for a file named "AuroraInit".
>> > > > > > > > >   * if no "AuroraInit" file exists in the
current
>> directory,
>> > > then
>> > > > > > look
>> > > > > > > in
>> > > > > > > > > the users
>> > > > > > > > >     home directory for an init file named
".aurora/init".
>> > > > > > > > >
>> > > > > > > > >> **Sidebar: Why fallback to the users
home directory?**
>> > > > > > > > >>
>> > > > > > > > >> Codebases are often shared between multiple
projects,
>> each
>> > of
>> > > > > which
>> > > > > > > > lives
>> > > > > > > > > in
>> > > > > > > > >> a set of subdirectories within the codebase.
For example,
>> > just
>> > > > > look
>> > > > > > at
>> > > > > > > > > the aurora
>> > > > > > > > >> sourcecode, which includes the aurora
scheduler, the
>> aurora
>> > > > > > executor,
>> > > > > > > > > thermos,
>> > > > > > > > >> and the aurora client.
>> > > > > > > > >>
>> > > > > > > > >> If multiple services live in a codebase,
then users can't
>> > put
>> > > an
>> > > > > > > > > AuroraInit file in
>> > > > > > > > >> the root directory of the codebase, because
it would
>> > interfere
>> > > > > with
>> > > > > > > > other
>> > > > > > > > > users'
>> > > > > > > > >> work.
>> > > > > > > > >>
>> > > > > > > > >> The home directory fallback provides
an easy way for
>> users
>> > to
>> > > > set
>> > > > > > up a
>> > > > > > > > > pointer
>> > > > > > > > >> to the correct init file, which won't
be removed by
>> > operations
>> > > > > like
>> > > > > > > > > switching branches
>> > > > > > > > >> in a source repository. Users write shared
init files,
>> which
>> > > are
>> > > > > > > located
>> > > > > > > > > in their projects
>> > > > > > > > >> in the codebase, and create a symbolic
link from their
>> > project
>> > > > > init
>> > > > > > > file
>> > > > > > > > > to their home
>> > > > > > > > >> directory.
>> > > > > > > > >
>> > > > > > > > > ### What goes into an init file?
>> > > > > > > > >
>> > > > > > > > > We should support the following kinds of
things:
>> > > > > > > > >
>> > > > > > > > >   1. Universal defaults - user-defined default
settings
>> that
>> > > will
>> > > > > be
>> > > > > > > > > applied to
>> > > > > > > > >     all commands. For example, if there is
a default
>> config
>> > > file
>> > > > > that
>> > > > > > > > > should always
>> > > > > > > > >     be used if the user doesn't specify one,
that would
>> be a
>> > > > > > universal
>> > > > > > > > > default.
>> > > > > > > > >   2. Command specific defaults - users should
be able to
>> > > specify
>> > > > > that
>> > > > > > > > they
>> > > > > > > > > always want
>> > > > > > > > >     to use certain parameter settings for
a specific
>> command.
>> > > For
>> > > > > > > > example,
>> > > > > > > > > if they
>> > > > > > > > >     want to always use a default batch size
of 10 for
>> > updates,
>> > > > but
>> > > > > > > don't
>> > > > > > > > > want to affect
>> > > > > > > > >     batch sizes for other commands like kill,
they could
>> use
>> > a
>> > > > > > command
>> > > > > > > > > specific default.
>> > > > > > > > >    3. Aliases - shorthand names for longer
parameters. A
>> user
>> > > > could
>> > > > > > > > > specify shorthands
>> > > > > > > > >      "east" and "west" for full jobkeys in
two different
>> > > > > datacenters.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > ### Defaults
>> > > > > > > > >
>> > > > > > > > > A default specifies a set of _bindings_.
If a parameter is
>> > > > omitted
>> > > > > > > > > from a command, and there's a binding for
that parameter,
>> it
>> > > will
>> > > > > be
>> > > > > > > > > automatically inserted into the command as
if the user had
>> > > typed
>> > > > > it.
>> > > > > > > > > The binding is specified in the configuration
file using a
>> > > Python
>> > > > > > > > > dictionary.
>> > > > > > > > >
>> > > > > > > > > For example, if the defaults included  `{'jobspec':
>> > > > > > > > > 'devcluster/me/prod/service'}`,
>> > > > > > > > > then if you ran `aurora job create` without
specifying any
>> > > > > > parameters,
>> > > > > > > > the
>> > > > > > > > > command-line
>> > > > > > > > > would automatically substitute
>> `devcluster/me/prod/service`
>> > for
>> > > > the
>> > > > > > > > > `jobspec` parameter.
>> > > > > > > > >
>> > > > > > > > > Defaults can be declared either globally
(in which case
>> > they'll
>> > > > be
>> > > > > > > > inserted
>> > > > > > > > > as parameters
>> > > > > > > > > for all commands), or for specific commands
(in which case
>> > > > they'll
>> > > > > > only
>> > > > > > > > be
>> > > > > > > > > inserted for a
>> > > > > > > > > single command).
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > ### Aliases
>> > > > > > > > >
>> > > > > > > > > An alias is a short equivalent for a parameter.
When a
>> > command
>> > > > line
>> > > > > > is
>> > > > > > > > > provided by a user, aliases will be expanded
inline.  A
>> user
>> > > can
>> > > > > > > > > specifically
>> > > > > > > > > mark an alias for expansion by prefixing
it with "@"; if
>> an
>> > > alias
>> > > > > > > > > appears on the command-line surrounded by
whitespace, it
>> will
>> > > be
>> > > > > > > > > replaced even if it isn't marked with an
"@".
>> > > > > > > > >
>> > > > > > > > >> **Sidebar: why not just use shell substitutions?**
>> > > > > > > > >>
>> > > > > > > > >> The @-substitution model proposed here
is very similar to
>> > unix
>> > > > > > > variable
>> > > > > > > > >> substitution. So why implement the same
thing all over
>> > again?
>> > > > Why
>> > > > > > not
>> > > > > > > > > just tell
>> > > > > > > > >> users to use their shell?
>> > > > > > > > >>
>> > > > > > > > >> There are several reasons.
>> > > > > > > > >>
>> > > > > > > > >> 1. All of the options and aliases that
affect aurora
>> command
>> > > > > > > invocations
>> > > > > > > > > can be specified
>> > > > > > > > >> in one place: the AuroraInit file.
>> > > > > > > > >> 2. Command shell substitution is complex
- the ways in
>> which
>> > > the
>> > > > > > > command
>> > > > > > > > > shell does
>> > > > > > > > >>  expansion, tokenization, substitution,
and parsing can
>> be
>> > > very
>> > > > > hard
>> > > > > > > to
>> > > > > > > > > follow. Adding
>> > > > > > > > >>  the aurora aliases to that process just
adds another
>> layer
>> > of
>> > > > > > > confusion
>> > > > > > > > > to the process.
>> > > > > > > > >>  With internal alias substitution, we
can avoid the
>> > confusions
>> > > > of
>> > > > > > > > > expansions and
>> > > > > > > > >>  tokenization for aurora aliases.
>> > > > > > > > >> 3. We can provide better debugging and
error messages
>> with
>> > our
>> > > > own
>> > > > > > > alias
>> > > > > > > > > substitution.
>> > > > > > > > >>  For example, if a user specifies a job
using aliases
>> like
>> > > > > "`aurora
>> > > > > > > job
>> > > > > > > > > create @c/@me/test/myservice`",
>> > > > > > > > >>  we can create an error message that
shows exactly what
>> they
>> > > > > typed,
>> > > > > > > and
>> > > > > > > > > how it expanded:
>> > > > > > > > >> <pre>
>> > > > > > > > >> Job "west/markcc/test/myservice" not
found in
>> configuration
>> > > file
>> > > > > > > > > config.aurora.
>> > > > > > > > >> Jobkey was generated by alias expansion:
original key was
>> > > > > > > > > "@c/@me/test/myservice", where:
>> > > > > > > > >>   - "@c" was expanded to "west"
>> > > > > > > > >>   - "@me" was expanded to "markcc"
>> > > > > > > > >>  config_file parameter "config.aurora"
was specified from
>> > > > > defaults.
>> > > > > > > > >> </pre>
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > ### Defining Shorthands and Defaults: Syntax
and Examples
>> > > > > > > > >
>> > > > > > > > > The init file is, like aurora job configurations,
a python
>> > > source
>> > > > > > file
>> > > > > > > > > using a Pystachio
>> > > > > > > > > schema. The schema is loaded, and an `Init`
object is
>> > retrieved
>> > > > > from
>> > > > > > > the
>> > > > > > > > > top-level
>> > > > > > > > > variable `init` in that file.
>> > > > > > > > >
>> > > > > > > > > The pystachio schema for init files is:
>> > > > > > > > >
>> > > > > > > > >    class CommandDefaults(Struct)
>> > > > > > > > >      command = Required(String)
>> > > > > > > > >      defaults=Map(String, String)
>> > > > > > > > >
>> > > > > > > > >    class Init(Struct):
>> > > > > > > > >      defaults=Map(String, String)
>> > > > > > > > >      command_defaults = Optional(CommandDefaults)
>> > > > > > > > >      aliases=Map(String, String)
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > For example, an init file could contain the
following:
>> > > > > > > > >
>> > > > > > > > >    init=Init(
>> > > > > > > > >      defaults={
>> > > > > > > > >        "jobspec": "west/markcc/test/myservice",
>> > > > > > > > >        "config_file": "./src/aurora/myservice.aurora"
>> > > > > > > > >      },
>> > > > > > > > >      aliases={
>> > > > > > > > >        "east": "east/markcc/test/myservice",
>> > > > > > > > >        "c": "east",
>> > > > > > > > >        "me": "markchucarroll"
>> > > > > > > > >      },
>> > > > > > > > >      command_defaults(command="job update",
>> > > > > > > > >        defaults={"--batch-size": 10}
>> > > > > > > > >      ))
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > With this configuration file, if the user
ran "`aurora job
>> > > > create`"
>> > > > > > > > without
>> > > > > > > > > any parameters,
>> > > > > > > > > it would automatically be expanded to "`aurora
job create
>> > > > > > > > > west/markcc/test/myservice
>> ./src/aurora/myservice.aurora`".
>> > > > > > > > >
>> > > > > > > > > If a user ran "`aurora job create east`",
the alias `east`
>> > > would
>> > > > be
>> > > > > > > > > expanded to "east/markcc/test/myservice",
and the missing
>> > > config
>> > > > > file
>> > > > > > > > > parameter would
>> > > > > > > > > be instantiated using the default, to create
a
>> command-line:
>> > > > > > > > > "`aurora job create east/markcc/test/myservice
>> > > > > > > > > ./src/aurora/myservice.aurora`".
>> > > > > > > > >
>> > > > > > > > > If a user ran "`aurora job create @c/@me/test/myservice`",
>> > the
>> > > > two
>> > > > > > > > aliases
>> > > > > > > > > would
>> > > > > > > > > be expanded, and the omitted configuration
parameter
>> would be
>> > > > > added:
>> > > > > > > > > "`aurora job create east/markchucarroll/test/myservice
>> > > > > > > > > ./src/aurora/myservice.aurora`".
>> > > > > > > > >
>> > > > > > > > > If a user ran "`aurora job update`", then
the `jobspec`
>> and
>> > > > > > > `config_file`
>> > > > > > > > > parameters would
>> > > > > > > > > get inserted from the global defaults, and
the
>> > > "--batch-size=10"
>> > > > > > would
>> > > > > > > be
>> > > > > > > > > inserted from
>> > > > > > > > > the command defaults for "aurora job update".
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message