felix-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sle...@composent.com
Subject Re: Newbie question related to device communication
Date Mon, 12 Oct 2015 16:52:15 GMT
Update:   ECF has recently completed initial documentation for 9
distribution providers [1].  We've also recently had our 3.11.0 release
[2], which includes a small new API for registering custom distribution
providers via the whiteboard pattern.



On 6/3/2015 10:03 AM, Scott Lewis wrote:
> On 6/3/2015 2:25 AM, Bruce Jackson wrote:
>> I’ve now built 2 different systems based on OSGi at scale (over 1M end
devices) where the endpoints needed to communicate together. In both
cases, it has required some ‘server’ infrastructure to help with
message passing between endpoints, but that could be a side effect of
the infrastructure we had to use. However, these server components were
again based on OSGi.
>> The http question (I.e. using the HTTP service) is actually a bit of a
red herring: there is nothing stopping each endpoint implementing an
http service and also being a client of other endpoints to pass
messages. My suggestion would be to use this approach,  as it has so
many side benefits, including:
>> * good availability of tools such as Jackson for message serialisation
>> * open protocols that will allow for different endpoint implementations
in future
>> * (by and large) avoiding problems that custom socket protocols will
run into with firewall tunnelling etc
>> …the list here is pretty long.
> I would like to discuss the Remote Service specification before
returning to discussing protocols and RS implementations.
> The specification explicitly allows the runtime selection of a
distribution provider to be used to export a remote service, via a
standardized service property called the 'config type' (property name: 
service.exported.configs).    As an example, the 'ecf.generic.server'
distribution provider is the config type used in this tutorial [1].
> I feel that one of the most useful aspects of the config type and the
underlying distribution system is that it abstracts away the
implementation specifics, i.e. the transport protocol (e.g. http, jms,
tcp, mqtt, private, etc), as well as the serialization mechanism/format
(json, binary, xml, etc).   It does this for both the remote service
creator/implementer and for the remote service consumer.
> IMHO, there are there are many advantages of doing this.   A big one is
modularity and reuse.  Why should it be necessary for every remote
service author to implement a brand new protocol and serialization, in
order to remote their micro service?   Another one is flexibility:  any
given reasonably sophisticated app has changing requirements over time,
and sometimes these changing requirements require flexibility in the
underlying libraries. Another big advantage is getting access to all the
simplicity, dynamics, versioning, and other capabilities of OSGi
services provided by the OSGi service registry and libs built on top of
it (e.g. DS, etc). There are of course many others.
> Of course, there are disadvantages in reusing distribution systems or
any set of libraries:   Perhaps the protocol or serialization mechanism
that's widely implemented and popular today doesn't meet your
application's performance needs.   Perhaps there are more appropriate or
new protocols, or improved implementations. Perhaps you have unique
security requirements.     Perhaps you have timing or reliability
requirements that are difficult to address. There are lots of
application-specific reasons for favoring or disfavoring non-functional
requirements like performance, ease of deployment and management,
security (e.g. firewalls), network reliability/failure handling, etc.
> But in my view, the Remote Services specification adopts a nice balance
here, by allowing potentially many distribution providers, down to the
level of the individual service instance set at *service instance
registration time*.   Even if one needs to create a new, entire
distribution system, there are still many advantages of using OSGI
Remote Services...e.g. the service registry's graceful handling of
dynamics inherent in a network-based service, versioning, discovery, and
the ability for service creators and perhaps more importantly service
consumers to ignore some of the subtle and difficult aspects of
distribution (e.g. partial failure, timing, etc).   I do think that
standardizing these advantages and having many implementations will over
time make it easier to build, use, understand, and manage distributed
> Returning to RS implementations:   The approach ECF has taken (since
well before the Distributed OSGi rfc and RS/RSA standardization) has
been protocol neutral.   As such we have gone to the trouble of creating
extensible APIs for various types of network communication:   e.g.
filetransfer, instant messaging, group messaging, voice over ip, network
service discovery, and remote services distribution (of course).   For
each of these types, we have multiple protocol implementations, some
created by us (in open source), some created by others (in open source
or not).   This is true, of course, for our OSGi Remote Services
implementation, and it's distribution and discovery subsystems...as I've
said we have a number of providers that are based upon tcp and we have
providers that are based upon http and REST.   But more importantly (I
think), we are trying to support the creation of hopefully many
other/new providers, created by others...possibly extending those that
already exist...that meet their needs for app-specific non-functional
requirements, new/popular application areas (like IoT) for distributed
> The way that we do this with a multi-provider/multi-protocol approach is
> 0) Fully implement the latest specs an test for compliance
> 1) Define stable, extensible, open APIs for distribution and discovery
(e.g. 'remote services api' -> distribution, 'discovery api' ->
discovery), and maintain/enhance these APIs.
> 2) Re-use existing work to create stable, extensible, open provider
implementations for these types (e.g. ecf generic, http/REST, r-osgi,
jms, mqtt, etc)
> 3) Do what we can in open cooperation with the community to document and
provide examples for both 1 and 2, as well as document how to create new
providers [2], and analyze the runtime characteristics of existing
> 4) Engage the community (commercial and open source) to drive 0-3
forword in the directions that are most needed, and get support for
these efforts via new committers, contributors, collaborators, and
supporters [3].
> 5) Provide the support for the community in doing all of the above
> Are we resource-limited?  Yes, unless you consider the community. Is
there more to do? (particularly around documenting individual
providers)?  Yes.   My comment is that in today's software world I don't
think these answers are particularly uncommon .  [4]
> Scott
> [1]
> See section:  Service Host: Register and Export the Service
> [2]
> [3] https://projects.eclipse.org/projects/rt.ecf/who
> [4] https://www.eclipse.org/ecf/NewAndNoteworthy_3.10.0.html
>> On 02/06/2015 16:08, "Scott Lewis" <slewis@composent.com> wrote:
>>> On 6/2/2015 2:16 AM, Simon Kitching wrote:
>>>> Hi Scott,
>>>> Maybe I shouldn't hijack a thread on the Felix list to ask about ECF,
>>>> but we are already talking about remote-services implementations, and
>>>> you are here so I'll do it
>>>> Link [1] has instructions on using ECF's remote services on Karaf
>>>> which lead to this karaf features file:
>>>> http://download.eclipse.org/rt/ecf/3.8.0/site.p2/karaf-features.xml.
>>>> That file appears to indicate that 49 bundles are needed (the majority
>>>> from eclipse.org). Plus bundles to support whatever discovery provider
>>>> is desired. That doesn't seem reasonable; other remote-services
>>>> implementations have 1-3 bundles. Have I understood correctly?
>>> For the latest the correct URL is:
>>> http://download.eclipse.org/rt/ecf/latest/site.p2/karaf-features.xml
>>> ECF's implementation is quite modular, and so is made up of more
>>> bundles, generally much smaller and finer-grained.
>>>> <stuff deleted>
>>>> Looking at the commit-log for the osgi code, I see that you are the
>>>> only committer since at least the start of 2014. Is this correct?
>>> I have done the majority of the commits from the start of 2014, but am
>>> not the only committer.   The source code history is not a very good
>>> indicator of overall contributions for a project in this point in it's
>>> life-cycle.   Also, much of the recent coding work on ECF (by myself and
>>> others) has taken place on our github repos[4].
>>>> <stuff deleted>
>>>> Regarding building from source : I see no Maven, Ant, Groovy or bnd
>>>> buildfiles in the sourcecode - just eclipse project files. Could you
>>>> point me to instructions on how to build artifacts from the
>>>> source-code (ideally outside of the Eclipse IDE)?
>>> We currently use Eclipse + buckminster to build, and are part of the
>>> Eclipse Simultaneous Release (which also uses buckminster). There are
>>> contributors from Red Hat that are working adding to use Tycho to
>>> build.  FWIW, we also produce a maven repos as part of our build [A].
>>>> And is there any easy way of seeing what the dependencies of the
>>>> projects under osgi/bundles are (the kind of info visible in a maven
>>>> pom-file)?
>>>  From Eclipse, certainly.  When Red Hat's contribution is in place you
>>> will be able to do with maven-pom file as well.
>>>> Is there a list of the implemented "distribution providers" (network
>>>> protocols for the actual remote method invocation) available, with
>>>> details on the features and performance of each choice?
>>> No, regrettably we don't have that.  I've opened an enhancement request
>>> to that effect [B].  There was some performance analysis work
>>> contributed some time ago, but unfortunately I think it's likely pretty
>>> obsolete by now.
>>>> I couldn't find one. In particular, I needed a high-performance
>>>> protocol - one that (a) keeps network sockets open rather than opening
>>>> a new connection for each call, and (b) uses a high-performance
>>>> serialization mechanism, ie not xml. I couldn't find any page with
>>>> that kind of information in it. The closest I found was:
>>>> *
>>>> https://wiki.eclipse.org/Discovery_and_Distribution_Provider_Configuration_Options
>>>> *
>>>> https://wiki.eclipse.org/Comparison_of_Discovery_and_Distribution_Providers
>>>> but neither of those pages are very useful. In particular, "r-osgi"
>>>> and "ecf generic" are not described in any way.
>>> For the details about everything there's always 'use the source'.
>>> I can tell you that r-osgi, ecf generic, JMS, MQTT distribution
>>> providers all
>>> 1) keep network sockets open as you describe;
>>> 2) use java object serialization as the default
>>> I can also tell you that all of those providers, and the distribution
>>> API that they implement (known to us as the 'remote services api'), are
>>> customizable/extensible for both 1 and 2...meaning that if you wish, you
>>> can replace both the serialization as well as the socket-level transport
>>> with something more suitable for your use case without having to
>>> implement a distribution provider from scratch.   As a an example, that
>>> is how websockets support was added in the 3.10 release cycle.
>>> I propose moving this to ecf-dev or direct correspondence if you want
>>> more details.
>>> Scott
>>> [A] http://build.ecf-project.org/maven/
>>> [B] https://bugs.eclipse.org/bugs/show_bug.cgi?id=469155
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
>>> For additional commands, e-mail: users-help@felix.apache.org
>> *** DISCLAIMER *** This message, including attachments, is intended
solely for the addressee indicated in this message and is strictly
confidential or otherwise privileged. If you are not the intended
recipient (or responsible for delivery of the message to such person) :
- (1) please immediately (i) notify the sender by reply email and (ii)
delete this message and attachments, - (2) any use, copy or
dissemination of this transmission is strictly prohibited. If you or
your employer does not consent to Internet email messages of this kind,
please advise Myriad Group AG by reply e-mail immediately. Opinions,
conclusions and other information expressed in this message are not
given or endorsed by Myriad Group AG unless otherwise indicated by an
authorized representative independent of this message.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
>> For additional commands, e-mail: users-help@felix.apache.org
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
> For additional commands, e-mail: users-help@felix.apache.org

To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
For additional commands, e-mail: users-help@felix.apache.org

View raw message