incubator-etch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Dixson" <dixs...@gmail.com>
Subject Re: Etch Name Service Proposal
Date Tue, 06 Jan 2009 15:08:51 GMT
A couple of call flow diagram would be useful.

Also a re-enforcement of how having the name service defined in terms
of an Etch API allows the service to be easily consumed by all
Etch-enabled languages.


--
james


On Mon, Dec 22, 2008 at 2:53 PM, scott comer (sccomer)
<sccomer@cisco.com> wrote:
> this is a proposal for a name service for etch. there is a text document
> describing the
> name service and rationale, and a proposed etch service idl.
>
> the intention is to define the service interface, implement an etch scheme
> which uses it,
> and produce a reference implementation of the service which can support
> simple
> deployments and also test the etch scheme.
>
> i will also post this to the wiki. please review and comment here.
>
> scott out
>
>
> Introduction
>
> When an client is connecting to an etch service, a uri is used to specify
> the
> details of the connection. Typically this code looks something like this:
>
>        RemoteNameServiceServer server = NameServiceHelper.newServer(
>
>  "tcp://host:4001?filter=KeepAlive&TcpTransport.reconnectDelay=4000",
>                null, "factory" );
>
> The uri specifies all the details of the etch connection in a convenient
> form.
> Yet the uri is still hard to manage. With it embedded in code we must
> recompile
> the client to change the connection details. We can load the uri from some
> other place, say a configuration file, environment variable, or the command
> line, but all we've done is move the problem to yet another hard to manage
> place. (Here 'manage' specifically refers to the process of obtaining the
> uri or
> updating it after a change.)
>
> What can we do to make the uri easier to manage? The main issue is that the
> service implementation and deployment specifies what the uri should be. Any
> change to the service implementation or deployment could trigger a need to
> update the uri that clients use to connect to it. So, one fact (connection
> uri),
> one source (service implementation and deployment), but a delivery path
> which is
> not automatic and often includes a human. Can you imagine trying to update
> 30,000 clients with a new connection uri?
>
> The classic solution applies here and represents the start of the next phase
> in
> etch development. Etch needs a Name Service.
>
> Basics
>
> At its heart a name service is easy. We use them every day without really
> thinking about them. Given a simple but abstract identifier a more
> complicated
> but concrete identifier is produced by some sort of lookup. In exchange for
> the
> abstraction, we obtain some independence from the complicated details of the
> concrete identifier, allowing the concrete identifier to change as needed by
> the environment. For example, apache.org is translated into the internet
> protocol v4 address 140.211.11.130 for us by dns. My name, sccomer, is
> translated into a uid (1079) by my nearby linux box and used for all sorts
> of
> evil purposes. In both cases I can use the simple name as a substitute for
> the
> more complicated name seamlessly in the environments I work in. Let's call
> the
> abstract identifier the source. Let's call the concrete identifier the
> target.
>
> In order for the name service to be useful to me, I have to be able to
> depend
> upon and trust it. That is, the translation process needs to be available
> when
> I need it, and the translations need to be accurate and, most importantly,
> secure. By secure I mean that if I'm going to connect to a service and
> supply
> some credentials and use it to do my work, I have an expectation that the
> service itself is reliable and trustworthy. That it is not being spoofed.
> This
> translates into access controls limiting who can make changes to the
> underlying
> data used to implement the name service translation.
>
> Requirements
>
> The name service should be defined using etch. Perhaps this is obvious but
> I'll
> say it here for completeness.
>
> The source should be specified in a uri. This allows us to exchange the
> target
> uri for a source uri + name service api to achieve our goal.
>
> To implement the functionality of the name service we need to have these
> elements:
>
> 1) an api to access the name translations, update them, etc.
> 2) a mechanism to protect the name service api from unauthorized access.
> 3) an etch connection scheme which uses the name service api to automate the
>   translation process for the client.
>
> First the api elements:
>
> 1a) format for source.
> 1b) a method to translate source to target.
> 1c) a method to list translations matching a query string.
> 1d) a method to add translation of a source.
> 1e) a method to remove translation of a source.
> 1f) a method to associate additional qualities with a particular
> translation,
>    which might be used to further qualify the translation for a particular
>    purpose.
>
> In order to implement replicated name services, we need an ability to
> receive
> notification of changes to translations at a name service instance:
>
> 1f) a method to subscribe to changes.
> 1g) a method to unsubscribe to changes.
> 1h) a method to receive notification of changes.
> 1i) a method to differentiate translations from primary or secondary
> sources.
>
> These mechanisms protect access:
>
> 2a) methods that read need to check authorization for reading (1b, 1c, 1f).
> 2b) methods that update need to check authorization for updating (1d, 1e).
>
> Etch client access to name service:
>
> 3a) Etch clients use a uri to specify etch connection parameters, so it is
>    natural that the way to specify a source requiring name service
> translation
>    would be with a uri which uses an alternate scheme along with the
> requested
>    source. Some elements of this source uri must be copied over to the
>    resulting target uri after translation, perhaps overriding elements
> there.
>
> Source Format
>
> Within a given name service database may be many entries offering the same
> (or, essentially the same) api, that is, etch service name (e.g.,
> etch.examples.perf.Perf). This is just as it is for any other service
> available over the network (nfs file servers, smtp mail servers, jabber im
> servers, etc.). This suggests that the api might be useful as an
> organizational
> concept for the name service database.
>
> Suppose we partition the name space into domains based on the api being
> offered.
> Within a given api domain, there may be a number of named instances. The
> instance name is used to uniquely identify a running instance of the service
> offering the api. There may be several ways to access an instance (called
> schemes in etch). The instance might offer tls and soap schemes, for
> example.
> Finally, a given scheme may be available in more than one flavor, depending
> upon the capabilities of the client. The preferred scheme should be tried
> first,
> and if it isn't supported a second scheme is tried, third, etc.
>
> This suggests a four part name: api, instance, scheme, and priority. We can
> combine the four parts into a single name by using the slash character as a
> separator. This gives the name a path-like quality and also allows it to be
> easily embedded in a uri:
>
>        api/instance/scheme/priority
>
> One obvious way to express api is to use the fully qualified service name of
> the
> etch idl. These names are composed of standard identifiers separated by
> periods
> (e.g., etch.examples.perf.Perf).
>
> The instance name should not contain the slash character. Since we want to
> embed
> these in a uri, the instance name should not contain any other uri
> significant
> characters either. If we stick to the same format as the api, we still have
> a
> large and interesting name space to work with.
>
> Since the scheme corresponds to a uri scheme name, then the same uri scheme
> syntax is required. This is pretty much a standard identifier.
>
> Priority is most easily a positive non-zero integer, with 1 being highest
> priority.
>
> So, here is a fully specified source for the Perf service named foo with tcp
> scheme:
>
>        etch.examples.perf.Perf/foo/tcp/1
>
> Since the sequence number is not often needed, it could be defaulted to 1 if
> missing. This would be the same as the previous:
>
>        etch.examples.perf.Perf/foo/tcp
>
> Name Service Api Details
>
> Please see the accompanying ns.etch file for specific api details and
> documentation.
>
> Etch Scheme
>
> An etch scheme is introduced to gain automatic access to the name service.
> This
> removes from most clients any burden relating to name service, and makes
> using
> name server essentially transparent (i.e., no program changes are required).
>
> Here's an example etch scheme:
>
>        etch://etch.examples.perf.Perf/foo/tcp
>
> This would connect to a name service, lookup the source, and then connect to
> the returned target. The highest priority entry matching the other three
> parameters would be used.
>
> Here is another example, this time with modification to the target uri:
>
>        etch://etch.examples.perf.Perf/foo/tcp?add:filter=Logger
>
> Logger would be added to the target filter chain.
>
> Less specific source specifications might be used to request alternate
> capabilities. Here we request a connection, tls would be preferred but we
> will
> accept tcp:
>
>        etch://etch.examples.perf.Perf/foo/tls,tcp
>
> Here we request the nearest service to 78749 within 100 miles:
>
>        etch://etch.examples.perf.Perf/./tcp?distance_le=100mi&zip=78749
>
> (to implement this, the name service would need to have location of
> services and some sort of function calculating distance based on zip codes.)
>
> Q/A:
>
> What other target uri edits can you make?
>
>        This is all kinda twitchy and not well-baked. Speculation. The
> problem is,
>        without edits there is clearly a gap in capability. But edits are
> messy
>        and too many puts us back to first base.
>
>        set:option=value (adds/replaces the query term option=value)
>        del:option       (deletes the query term option)
>        rem:option=value (removes value from option's comma separated list)
>
> Which name service is used?
>
>        The name service might be configured in a number of ways:
>                Environment varible
>                Via resources
>                Dhcp option (configures all clients within a subnet)
>
> Can name service be partitioned or federated?
>
>        Yes. The instance could have structure which suggests a hierarchy of
> name
>        services serving various domains. A request to lookup or register an
>        instance not in the local domain could be routed to another name
> service for
>        processing.
>
> Can name service be replicated?
>
>        Yes. Using the subscribe feature, one name service could replicate
> the
>        contents of another (in a publisher/subscriber fashion). Lookup could
> be
>        satisfied locally, while register could flow through to the publisher
> or
>        the relationship could be more symmetric.
>
> If I connect to a service using the uri "etch://Foo/bar/tcp" and the
> connection
> goes down, how do I reconnect?
>
>        This is tricky. I don't know yet. Originally I was thinking that the
> name
>        would be looked up and then the connection made to the first target
> which
>        worked. That is, the actual transport stack would reflect the matched
> target.
>        But if the target uri for that name changes, or a secondary
> translation is
>        used, then the bottom layers of the current  stack would have to be
>        discarded and rebuilt from the new target uri.
>
> module etch.services.ns
>
> /**
>  * The NameService provides translation from an abstract name for a service
> to a
>  * uri that may be used to contact the service. A given service may offer
>  * several different connection schemes, and a client may only support a
> subset
>  * of those. So, when a client wishes to connect to a service, it may query
>  * the NameService with both abstract name and desired schemes in order to
> find
>  * a suitable match.
>  *
>  * Some example queries:
>  *
>  * All services with servicename='Foo' and instancename='bar' and
> scheme='tcp'
>  *
>  *   serviceUri='Foo/bar/tcp'
>  *
>  * Same as the query above, but written out longhand:
>  *
>  *   servicename='Foo' and instancename='bar' and scheme='tcp'
>  *
>  * Service Foo instance bar with tls or tcp (prefer tls):
>  *
>  *   serviceUri='Foo/bar/tls,tcp'
>  *
>  * or:
>  *
>  *   servicename='Foo' and instancename='bar' and scheme='tls,tcp'
>  *
>  * Fully qualified serviceUri:
>  *
>  *   serviceUri='Foo/bar/tcp/1'
>  *
>  * Any instance of service Foo located in Austin, TX:
>  *
>  *   servicename='Foo' and qualities.location='Austin, TX'
>  *
>  * Instances of service Foo with capacity >= 400:
>  *
>  *   servicename='Foo' and qualities.capacity>=400
>  *
>  * Relative operations on qualities require that the quality be present and
> be
>  * comparable (with mixed types some type promotion is supported). But you
> might
>  * be surprised, qualities which are absent will give a negative comparison
> no
>  * matter which sense is used: both qualities.x > 50 and qualities.x < 50
> are
>  * false whenever qualities.x is null. Consider this test: not qualities.x <
> 50.
>  * This is not the same as qualities.x >= 50. It is the same as qualities.x
> =
>  * null or qualities.x >= 50.
>  */
> @Timeout( 30000 )
> service NameService
> {
>        /** An entry describing a service. */
>        struct Entry
>        (
>                /** A service description uri, composed of
>                 * servicename/instancename/scheme[/priority]. */
>                string serviceUri,
>
>                /** Qualities of this service instance. */
>                Map qualities,
>
>                /** Etch service connection uri, for example
>                 * tcp://localhost:9000?filter=KeepAlive */
>                string targetUri,
>
>                /** Lifetime in seconds from last update. */
>                int ttl,
>
>                /** Who created or last update. */
>                string who,
>
>                /** Date / time of create or last update. */
>                Datetime lastUpdate,
>
>                /** Flag indicating entry has been removed. */
>                boolean removed
>        )
>
>        /**
>         *
>         */
>        @Authorize( canLookup, source )
>        Entry lookup( string source )
>
>        /**
>         * Looks up entries by matching them against the query string.
> Entries are
>         * returned in a stable and consistent order, increasing alphabetical
> by
>         * servicename, instancename, and scheme, and then increasing
> numerical by
>         * priority. If a scheme search term appears with a list, schemes are
>         * returned in the specified order (e.g., scheme='tls,tcp').
>         *
>         * @param query a sql-like query expression using the elements of the
>         * serviceUri, targetUri, and qualities. If query is null or the
> empty
>         * string, all entries are matched.
>         *
>         * @param offset offset in the list of matched items of the first
> item to
>         * return. This parameter and the count parameter are used to step
> through
>         * the result set when there are many results.
>         *
>         * @param count number of items to return.
>         *
>         * @return entries matching query. If you requested 10 items and 10
> are
>         * returned, there could be more. To get the next batch, add
> results.length
>         * to offset and call find again:
>         *
>         * int i = 0;
>         * int n = 10;
>         * Entry[] results;
>         * while ((results = service.find( null, i, n )).length > 0)
>         * {
>         *   for (Entry e: results)
>         *     processEntry( e );
>         *   if (results.length < n)
>         *     break;
>         *   i += results.length;
>         * }
>         */
>        @Authorize( canFind, query )
>        Entry[] find( string query, int offset, int count )
>
>        /**
>         * Adds or updates the specified entry. The given parameters replace
> any
>         * existing values in an Entry whose key is serviceUri, whereas who
> and
>         * lastUpdate are set to the current user and current date / time,
>         * respectively.
>         *
>         * @param serviceUri the uri describing the service. The uri should
> be of
>         * the form "servicename/instancename/scheme[/priority]", where
> servicename
>         * is a valid fully qualified service name (e.g.,
>         * etch.services.ns.NameService"), instancename is a valid etch
> identifier
>         * (e.g, fred, alice01), scheme is a valid uri scheme (e.g., tcp,
> tls), and
>         * priority is an integer >= 1. If priority is omitted, it is
> defaulted to
>         * 1.
>         *
>         * @param qualities a map which may be used to describe additional
> features
>         * of the entry, such as purpose, licenses, capacity, location,
> owner,
>         * whatever. Query strings may test values of qualities using a
> variety of
>         * sql-like operators. Qualities may be null.
>         *
>         * @param targetUri the uri describing the contact information for
> the
>         * service.
>         *
>         * @param ttl the lifetime of the entry specified as seconds. 0 means
>         * forever, -1 means remove immediately when the connection to the
>         * NameService is dropped.
>         */
>        @Authorize( canRegister, serviceUri, qualities )
>        void register( string serviceUri, Map qualities, string targetUri,
> int ttl )
>
>        /**
>         * Registers a number of entries all in one operation. Identical to
> calling
>         * register with each entry in turn.
>         *
>         * @param entries a sequence of Entry records with serviceUri,
> targetUri,
>         * qualities, and ttl as specified in register() above. Who and
> lastUpdate
>         * fields are ignored.
>         */
>        @Authorize( canRegisterBulk, entries )
>        void registerBulk( Entry[] entries )
>
>        /**
>         * Removes the specified entry.
>         *
>         * @param serviceUri the uri describing the service.
>         */
>        @Authorize( canRegister, serviceUri, null )
>        void unregister( string serviceUri )
>
>        /**
>         * Adds a request for notification of changes to entries matching the
> query.
>         * The current value of all matching entries is delivered via
> entryChanged
>         * client message, as well as any updates or new entries.
>         *
>         * @param query a sql-like query expression using the elements of the
>         * serviceUri, targetUri, and qualities. If query is null or the
> empty
>         * string, all entries are matched.
>         */
>        @Authorize( canFind, query )
>        void subscribe( string query )
>
>        /**
>         * Removes a request for notification of changes to entries matching
> the
>         * query.
>         *
>         * @param query a query previously passed to subscribe. The string
> must
>         * match exactly.
>         */
>        void unsubscribe( string query )
>
>        /**
>         * Removes all requests for notification of changes to entries. This
>         * operation is implicitly performed when the connection to the
> NameService
>         * is dropped.
>         */
>        void unsubscribeAll()
>
>        ////////////////////
>        // RIGHTS TESTING //
>        ////////////////////
>
>        /**
>         * Tests whether the current user is authorized to lookup the source.
>         * @param source the complete specification api/instance/scheme.
>         */
>        boolean canLookup( string source )
>
>        /**
>         * Tests whether the current user is authorized to run the query.
>         *
>         * @param query a sql-like query expression using the elements of the
>         * serviceUri, targetUri, and qualities. If query is null or the
> empty
>         * string, all entries are matched.
>         *
>         * @return true if the current user is authorized to run the query.
>         */
>        boolean canFind( string query )
>
>        /**
>         * Tests whether the current user is authorized to register the
> service.
>         *
>         * @param serviceUri the uri describing the service.
>         *
>         * @param qualities a map which may be used to describe additional
> features
>         * of the entry.
>         *
>         * @return true if the current user is authorized to register the
> service.
>         */
>        boolean canRegister( string serviceUri, Map qualities )
>
>        /**
>         * Tests whether the current user is authorized to register the
> entries.
>         * This is the same as:
>         *
>         * for (Entry entry: entries)
>         *   if (!canRegister( entry.serviceUri, entry.qualities ))
>         *     return false;
>         * return true;
>         *
>         * @param entries a sequence of Entry records with serviceUri,
> targetUri,
>         * qualities, and ttl as specified in register() above. Who and
> lastUpdate
>         * fields are ignored.
>         *
>         * @return true if the current user is authorized to register the
> entries.
>         * This is an all or nothing proposition.
>         */
>        boolean canRegisterBulk( Entry[] entries )
>
>        ////////////////////
>        // CLIENT METHODS //
>        ////////////////////
>
>        /**
>         * Notifies the client of a change in an entry. The entry might have
> been
>         * created, updated, or removed.
>         *
>         * Note: while this might have normally been an event, we made it a
> call
>         * to slow down the processing of what might otherwise be a rather
> large
>         * change set.
>         *
>         * Note: when keeping track of entries, always keep the one with the
> latest
>         * lastUpdate.
>         *
>         * @param query the query which triggered the notification.
>         *
>         * @param entry the entry which has changed.
>         */
>        @Direction( client )
>        void entryChanged( string query, Entry entry )
> }
>
>

Mime
View raw message