Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 7ED0A200BFC for ; Sat, 14 Jan 2017 14:19:49 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 7D430160B35; Sat, 14 Jan 2017 13:19:49 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 8DA5B160B28 for ; Sat, 14 Jan 2017 14:19:47 +0100 (CET) Received: (qmail 87184 invoked by uid 500); 14 Jan 2017 13:19:41 -0000 Mailing-List: contact user-help@karaf.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@karaf.apache.org Delivered-To: mailing list user@karaf.apache.org Received: (qmail 87174 invoked by uid 99); 14 Jan 2017 13:19:41 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Jan 2017 13:19:41 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id F0E501806FB for ; Sat, 14 Jan 2017 13:19:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.49 X-Spam-Level: ** X-Spam-Status: No, score=2.49 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id NGCf6aV5juya for ; Sat, 14 Jan 2017 13:19:33 +0000 (UTC) Received: from mail-ot0-f181.google.com (mail-ot0-f181.google.com [74.125.82.181]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 1279A5FCA2 for ; Sat, 14 Jan 2017 13:19:32 +0000 (UTC) Received: by mail-ot0-f181.google.com with SMTP id 65so18573266otq.2 for ; Sat, 14 Jan 2017 05:19:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:references:in-reply-to:subject:date :message-id:mime-version:thread-index:content-language; bh=7sSVc72jUrZOBLsvsT3nsweUYP38KcZm4L09hcRFmgg=; b=iF0WM8wzc/ZiAVkRTtZHNnif1AhGEXoEtXJrlfZxzJBXg7rX7DYssOi/kP9baZPUDy jmrz/rTwkooiQ0ppaGROlL/6WKc6u8spSJimc5Z2/igm8k21nvBhUxzixvDRQK+9YlBm /eFHqshxCsfWVldzea0aTZnhtraG2k5+Gj5RjScILgvblLFJRwHXy6d2SW9TztwoxmDF sUINiwn2HK8VWGNiE22uc9puUQn7qveuVpk8ip0IpWEkKKXdb6Zi3ILOgKWyoI5l5Nle Xh1IYEYNbyH3n6Guymftwq5TDltuA56STjBCqbX1SbrRModrBGOslTOeBRreRQGgA7GG 9Ggg== X-Gm-Message-State: AIkVDXKu2wdPk1nt8645F1NbblVG6h+zL7ACfRHHiDKL7PKN0Ea8Ro6kYwGqeKpU5pus0qY8 X-Received: by 10.157.18.246 with SMTP id g109mr11959695otg.10.1484399971984; Sat, 14 Jan 2017 05:19:31 -0800 (PST) Received: from Ranx (cpe-173-174-112-236.austin.res.rr.com. [173.174.112.236]) by smtp.gmail.com with ESMTPSA id s33sm7027943ota.19.2017.01.14.05.19.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 14 Jan 2017 05:19:31 -0800 (PST) From: "Brad Johnson" To: References: <08f901d26dc7$a03e1b70$e0ba5250$@redhat.com> <094001d26dd2$01bb7d70$05327850$@redhat.com> In-Reply-To: Subject: RE: Opinionated... Date: Sat, 14 Jan 2017 07:19:31 -0600 Message-ID: <0ae901d26e68$d7d6abf0$878403d0$@redhat.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0AEA_01D26E36.8D3F9750" X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQIXuAzj+f1oH9IXCwv/P7asrpL4ZQFurFL1AyhGflkB55+kkKB5XRtA Content-Language: en-us archived-at: Sat, 14 Jan 2017 13:19:49 -0000 This is a multipart message in MIME format. ------=_NextPart_000_0AEA_01D26E36.8D3F9750 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Scott, =20 It=E2=80=99s funny that you mention OSGi Remote Services as that was = sort of in the back of my mind. I think I recall Christian said he was = working on a Remote Services implementation as well. But I don=E2=80=99t = know enough about it yet to include it in the discussion. I suspect = what I=E2=80=99m going to do is set up a GitHub project with a basic = project and then a set of appliances for a variety of enterprise = integration purposes. That baseline has to be in place before lower = levels can be addressed =20 But I do think that a communications abstraction is going to be = necessary. A next generation of communications pipes should abstract = protocols away from the OSGi programmer. Or as the great and powerful Oz = put it =E2=80=9Cnever mind the little man behind the curtain=E2=80=9D. =20 When writing code in a Camel route in an OSGi bundle the programmer = should either be thinking about using a named request/response pipe or = an event pipe =E2=80=93 IPs, ports, transports, and so on should happen = via straight configuration files, parsed, configured and then registered = as services to be picked up in bundles. There isn=E2=80=99t anything = that stands in the way of doing that now other than a little elbow = grease. =20 So I=E2=80=99ll definitely give the Remote Services a deeper look. =20 =20 From: Scott Lewis [mailto:slewis@composent.com]=20 Sent: Friday, January 13, 2017 7:16 PM To: user@karaf.apache.org Subject: Re: Opinionated... =20 Hi Brad, You might be interested in the OSGi Remote Services = specification...which mentions the distributed computing fallacies in = the introduction. It's chapter 100 in the enterprise spec [1]. A big part of Remote Services is the ability to use OSGi service = dynamics to 'deal-with' distributed systems issues like partial failure = (i.e. network is reliable). For example, one way to represent the = failure of a remote service would be to make the local service proxy go = away. Note that with OSGi service dynamics and (e.g.) DS, the = consequences of such a thing on dependent services can be easily handled = without introducing special mechanism. IMO another advantage of Remote Services is that the OSGi service = contract/impl separation also decouples the service from the = distribution system. This allows the service designer to create remote = services (API and impl) that are independent of the distribution = system's serialization format (e.g json, xml, obj serialization, etc) = and comm approach/protocol (e.g. http/rest, pub/sub messaging, mqtt, = tcp, etc). As an example of this, I've created three Karaf-hosted = remote services that allow remote monitoring and management of Karaf = bundles, services, and install/uninstall of Karaf features...and these = services can be accessed from remote Eclipse via an mqtt broker, or via = server-based tcp, or via other distribution systems without changing the = service APIs or implementation. Scott [1] https://www.osgi.org/developer/specifications/ [2] https://wiki.eclipse.org/Karaf_Remote_Management_with_Eclipse On 1/13/2017 11:19 AM, Brad Johnson wrote: That is certainly the sort of library that could be used as a standard. = Get an agreement on the standard OSGi service interface and then use it = and others for that implementation. Which brings up a good question and = issue. There would have to be some set of standardized messages and = exception types. The CiruitBreaker example throws a = CircuitBreakingException (naturally enough). If there=E2=80=99s an = ErrorHandlerService it would have to know the standard set of exceptions = that could be expected or, at least, a set of parent classes. Since = CircuitBreakingException is a relatively simple class it would be = perfect for a default ErrorHandlerService to catch for that class of = exceptions. =20 =20 Obviously there will have to be some head scratching and chin rubbing = about how the pieces fit together exactly. The CircuitBreakerService = (and the others too) could also be more like container classes that = listen and pick up CircuitBreakerListenerService instances. So one = listener might just log the circuit breaker exception. But you might = instantiate an SMTPCircuitBreakerNotifcationService that implements the = CircuitBreakerListenerService and fires off an email to an admin email = address if the breaker is tripped.=20 =20 That CircuitBreakerService might also be picked by the Kontainer = instance which listens for on/off control events from the outside world. = Some thinking to do there but they are tractable problems with services = and events. =20 The main services like CircuitBreakerService and ThrottlerService might = register themselves as providers with the ErrorHandlerService which = would catch the types of exceptions they throw. It in turn could listen = for custom ExceptionHandlerListener that listen for and handle = specific exception types. Still thinking and hand waving about this but = I think a sane set of standard services, listeners and events could be = created that would permit a user to create simple handlers to register. =20 There would also be the issue of the issue of how to automate injection = of those into the Camel routes. That doesn=E2=80=99t seem like it = should be a daunting challenge but it would be important. And I think = very important that those get injected automatically even if the = services only provide basic logging initially with no client custom = code. =20 From: James Carman [mailto:james@carmanconsulting.com]=20 Sent: Friday, January 13, 2017 12:12 PM To: bradjohn@redhat.com =20 Subject: Re: Opinionated... =20 Commons Lang3 has a pretty simple CircuitBreaker implementation that I = used in Microbule: https://github.com/Microbule/microbule/blob/master/decorator/circuitbreak= er/src/main/java/org/microbule/decorator/circuitbreaker/CircuitBreakerFil= ter.java On Fri, Jan 13, 2017 at 1:05 PM Brad Johnson > wrote: Folks, =20 I wanted to make sure that my promoting CDI, Camel Java DSL, & static = profiles didn=E2=80=99t obscure the point I was trying to make. = Whatever mechanics we choose I=E2=80=99d really like us to be unified = behind a common paradigm so that our documentation, exemplars, = archetypes, blogs, libraries, and so on are all organized the same and = use the same mechanics and layouts for projects.=20 =20 We should promote an idiomatic way to develop software using Karaf Boot. = That=E2=80=99s one problem I hear from a lot of clients. There are = such cross-currents of information about how to develop OSGi-based = software that it gets confusing. Best or preferred practices are lost = in the noise. I won=E2=80=99t get into all that since I=E2=80=99m sure = most of you have dealt this problem. Not to pick on it but a good = example is that the Camel in Action book recommends using Pojos instead = of using Processors/Exchanges. It is on somewhere near the back of the = book in a few pages. I don=E2=80=99t know how many examples on the web = site actually use the Processor/Exchange but it is a lot. Then there are = examples with Spring, Blueprint, Java DSL, Scala, etc. There are = annotations that only work in one environment but not in all of them. =20 By selecting an idiomatic and =E2=80=9Copinionated=E2=80=9D way of = creating Karaf Boot microcontainers we could make sure that sort of = confusion isn=E2=80=99t continued forward. It would require a lot less = documentation to cover the same ground and make editing and updating = easier. It would make creating sample and example projects a lot = easier. It would simplify what Karaf Boot appliances have to support and = make sure there aren=E2=80=99t concerns that work in one environment and = not in another or that might work differently in a different = environment. =20 I=E2=80=99m personally interested in Karaf Appliances with standard = Maven structures, standard bundle structures, and reference = implementations that have a good chunk of the basic functionality. = I=E2=80=99d say we take a page from the =E2=80=9Cconvention over = configuration=E2=80=9D book or, at least, a =E2=80=9Cconventional = configuration=E2=80=9D and likely a bit of both. Because the appliances = are focused on microservices we should get out ahead of the Gartner hype = cycle. Right now we are at the Peak of Inflated Expectations and in a = couple of years we=E2=80=99ll be at the Trough of Disillusionment. That = disillusionment will come for a number of reasons. Flying Spaghetti = Monster topology will be one of them but, more importantly for a Karaf = Appliance, is the consistent problem of =E2=80=9Cnetwork = fallacies=E2=80=9D. Every Karaf Kontainer should have standard OSGi = service interfaces and basic implementations that address each of the = fallacies that apply to a uService. The Kontainers should insist on it = and not make it optional. If the user doesn=E2=80=99t want that = functionality they would then need to disable via configuration. But = the Kontainer will get stuck in a grace period and then fail if an = expected, standard service isn=E2=80=99t available. All of the standard = OSGi service APIs would have basic implementations to start but as more = specific Kontainers. But, because they are standard services new ones = can be developed by the community or by the end developer. =20 As developers, we=E2=80=99ve all had to implement functionality and then = come back and deal with error handling, security, etc. I say we simply = cut those services in to the Kontainer right from the get-go. The = Kontainer doesn=E2=80=99t run if it doesn=E2=80=99t find the service. = That isn=E2=80=99t to say these become a fundamental part of Karaf but a = fundamental part of the Kontainer service that runs in Karaf. =20 The standard bundles would only implement basic functionality and not do = anything sophisticated. New bundles and libraries for more = sophisticated implementations could be added later. All of the bundles = would likely have disable flags if the developer found the particular = concern irrelevant. For example, security might not be relevant. The = following aren=E2=80=99t meant to be comprehensive. Just addressing key = concerns. Other standards like LoggingService might be included by = default as well.=20 =20 The intent here isn=E2=80=99t to define the exact mechanics but the = standard OSGi service interface that would be _required_ in any = implementation of a Kontainer, even if the implemented bundle is simply = a passthrough or can be disabled, it forces the developer to explicitly = deal with the problems or choose to ignore them altogether. =20 =20 Because these service interfaces and the bundles that implement them are = standard, the set can be specified by the dependencies specified in the = Maven build, features and/or profiles. =20 1. The network is reliable. A standard =E2=80=9CError Handler=E2=80=9D OSGi service. The default = bundle would simply capture errors/exceptions and log them. Perhaps it = would specify retries. Drop in solutions might include errors going to = dead letter queues and so on. The OSGi service interface is required for = Kontainer bootstrap so use the default or use a standard one or create = one of your own. If they want to change configuration of this bundle or = put in a new one, they know exactly what it is, where it exists, how it = is specified to the build, and what configuration file is associated = with it. No rummaging around through code. When the inevitable error, = exceptions and problems arise, the developer isn=E2=80=99t left = wondering where and how they should add the functionality to handle it. =20 A standard =E2=80=9CCircuit Breaker=E2=80=9D service API and basic = implemented bundle should be provided. Perhaps the standard bundle = would simply count errors over a time frame and shut down if that limit = is hit and allow those values to be configured. Default would be a = rather unsophisticated implementation but provide the convention and = automated wiring of a circuit breaker OSGi service. Other = implementations might fire off emails to Sys Admins or be combinations. = And if it is really undesirable, set a disable flag. =20 2. Latency is zero. A standard OSGi Throttling service interface and bundle implementation = would be included. If you want different behavior, change it. If you = want to disable it, set the flag. However, there are bigger issues here = that I=E2=80=99ll address a bit more down below. =20 3. Bandwidth is infinite. Throttling OSGi service again. Ditto to comment 2. =20 4. The network is secure. Standard OSGi service to plug in in various authentication/authorization = mechanisms. By default it might be pass through but also have a = different implementation that uses a simple username/password. Obviously = LDAP, JAAS, and other bundles could be created and dropped into place.=20 =20 5. Topology doesn't change. Back to the Circuit Breaker, logging and perhaps notification mechanism. = Also the transport issue below where I=E2=80=99ll mention some = configuration. =20 6. There is one administrator. //No particular plugin for this but standardized configuration and = expected bundles help and this also relates to the transport discussion. =20 7. Transport cost is zero. //Probably not a concern here directly but will be a big issue of = uServices. =20 8. The network is homogeneous. //I think this issue can be dealt with in our context with many of the = standard libraries but can be abstracted a bit more. =20 Obviously a big issue we=E2=80=99ll see, and I=E2=80=99ve seen in the = past, is chained request/response calls. Service 1 making a REST call to = service 2 making a REST call to service 3=E2=80=A6etc. And all of a = sudden the latency is a killer. =20 ServiceMix/Karaf/Camel can already abstract away some of that via = property substitution. I=E2=80=99d suggest we take that one step further = and put _all_ transport/protocol information in configuration and create = a standardized URI. As a developer or a senior developer over a group of = developers, I don=E2=80=99t want them to be concerned with the fiddly = bits of the transport in the code and routes and I certainly = don=E2=80=99t want to recompile just to make such changes. =20 Akka, for example, uses local URIs like akka://. But a similar = Karaf/Camel URI could be used and mapped via the configuration files. = So the developer would always use karaf:// in their routes and = configuration mapping would use the URI specified. = karaf://myserviceName. In the configuration file might be mapped a = transport.configuration.cfg file. =20 I believe that is important for a lot of reasons. A mid-level or = junior-level developer shouldn=E2=80=99t be involved in configuration = like: " ftp://foo@myserver?password=3Dsecret& recursive=3Dtrue& ftpClient.dataTimeout=3D30000& ftpClientConfig.serverLanguageCode=3Dfr" =20 So the cfg file might look like this: clientService=3D"ftp://foo@myserver?password=3Dsecret = & recursive=3Dtrue& ftpClient.dataTimeout=3D30000& ftpClientConfig.serverLanguageCode=3Dfr" (At least properties get rid of the gawdaful escaped ampersands). =20 The code would then say =E2=80=9Ckaraf://clientService=E2=80=9D =20 One can do much of that via configuration right now but I think it is = critical to move it completely to configuration so that admins know = exactly what to change and where to find it when topologies change. It = also means that when the backlash from microservice calling microservice = calling microservice being slow happens, that simple mapping would = permit things like going to JMS asynchronous request/response (or other = fast, async mechanisms) that don=E2=80=99t swamp the virtual = machine=E2=80=99s or Karaf instance resources. It would also allow for = easy stubbing or mock testing of the Kontainer as it will be deployed = without using PAX exam or other mechanism. =20 Creating standard OSGi service APIs in an anticipation of these problems = would permit for an evolutionary approach to these problems in the = future and specific solutions when a standard Kontainer is developed. = Even standard error handler service implementations can be created. =20 Once such a basic, standard Kontainer exists, then uKontainers that = implement basic functionality commonly used could be created. There are = JPA examples already. But the average developer is going to be given a = task to receive some canonical data model via a REST service and poke it = into a database. That database model probably won=E2=80=99t look like = what they are receiving. So a uKontainer that has a REST front end they = can modify, a Dozer object mapping file in the middle with a transform, = and a call to the database will be used repeatedly. =20 It may be that Oracle, MySQL, BerklyDB, and so on each endup with = different error handler plugin implementations which are used with the = same REST, mapping, JPA container. Just change the Maven dependency or = profile. =20 There are a large number of examples like that. In the case of that = uKontainer there would like be a JPAErrorService for catching common = errors and another for Dozer errors and for unmarshaling errors. As a = developer looking to solve very specific problems, I just download the = uContainer and do the Dozer mapping, change some configuration and then = test it. =20 That also means, that much like Camel EIPs, open source developers can = focus on hardening these containers, fixing bugs, putting in performance = enhancements and the like. If a new error is coming from JPA that a = user finds and isn=E2=80=99t being handled in a coherent fashion, then a = new block or delegate code is added and released. Just as we=E2=80=99d = do with a Camel endpoint or component. =20 Having standard error handlers built into uKontainers would also help = make coherent messages from the large and unwieldy stack traces full of = reflection that we commonly see. The error handler OSGi plugin for a = given problem would be highly focused on identifying and reporting = problems with a specific technology or set of technologies. =20 =20 =20 https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing =20 ------=_NextPart_000_0AEA_01D26E36.8D3F9750 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable

Scott,

 

It=E2=80=99s funny that you mention OSGi Remote Services as that was = sort of in the back of my mind.=C2=A0 =C2=A0I think I recall Christian = said he was working on a Remote Services implementation as well. But I = don=E2=80=99t know enough about it yet to include it in the discussion. = =C2=A0I suspect what I=E2=80=99m going to do is set up a GitHub project = with a basic project and then a set of appliances for a variety of = enterprise integration purposes.=C2=A0 That baseline has to be in place = before lower levels can be addressed

 

But I do think that a communications abstraction is going to be = necessary.=C2=A0 A next generation of communications pipes should = abstract protocols away from the OSGi programmer. Or as the great and = powerful Oz put it =E2=80=9Cnever mind the little man behind the = curtain=E2=80=9D.

 

When writing code in a Camel route in an OSGi bundle the programmer = should either be thinking about using a named request/response pipe or = an event pipe =E2=80=93 IPs, ports, transports, and so on should happen = via straight configuration files, parsed, configured and then registered = as services to be picked up in bundles. =C2=A0There isn=E2=80=99t = anything that stands in the way of doing that now other than a little = elbow grease.

 

So I=E2=80=99ll definitely give the Remote Services a deeper = look.

 

 

From: Scott Lewis [mailto:slewis@composent.com]
Sent: Friday, = January 13, 2017 7:16 PM
To: = user@karaf.apache.org
Subject: Re: = Opinionated...

 

Hi = Brad,

You might be interested in the OSGi Remote Services = specification...which mentions the distributed computing fallacies in = the introduction.   It's chapter 100 in the enterprise spec = [1].

A big part of Remote Services is the ability to use OSGi = service dynamics to 'deal-with' distributed systems issues like partial = failure (i.e. network is reliable).   For example, one way to = represent the failure of a remote service would be to make the local = service proxy go away.  Note that with OSGi service dynamics and = (e.g.) DS, the consequences of such a thing on dependent services can be = easily handled without introducing special mechanism.

IMO another = advantage of Remote Services is that the OSGi service contract/impl = separation also decouples the service from the distribution = system.   This allows the service designer to create remote = services (API and impl) that are independent of the distribution = system's serialization format (e.g json, xml, obj serialization, etc) = and comm approach/protocol (e.g. http/rest, pub/sub messaging, mqtt, = tcp, etc).   As an example of this, I've created three = Karaf-hosted remote services that allow remote monitoring and management = of Karaf bundles, services, and install/uninstall of Karaf = features...and these services can be accessed from remote Eclipse via an = mqtt broker, or via server-based tcp, or via other distribution systems = without changing the service APIs or = implementation.

Scott

[1] https://www.osgi.= org/developer/specifications/
[2] ht= tps://wiki.eclipse.org/Karaf_Remote_Management_with_Eclipse

On= 1/13/2017 11:19 AM, Brad Johnson wrote:

That is = certainly the sort of library that could be used as a standard. Get an = agreement on the standard OSGi service interface and then use it and = others for that implementation.  Which brings up a good question = and issue.  There would have to be some set of standardized = messages and exception types.  The CiruitBreaker example throws a = CircuitBreakingException (naturally enough).  If there=E2=80=99s an = ErrorHandlerService it would have to know the standard set of exceptions = that could be expected or, at least, a set of parent classes.  = Since CircuitBreakingException is a relatively simple class it would be = perfect for a default ErrorHandlerService to catch for that class of = exceptions. 

 =

Obviously = there will have to be some head scratching and chin rubbing about how = the pieces fit together exactly.  The CircuitBreakerService (and = the others too) could also be more like container classes that listen = and pick up CircuitBreakerListenerService instances.  So one = listener might just log the circuit breaker exception.  But you = might instantiate an SMTPCircuitBreakerNotifcationService that = implements the CircuitBreakerListenerService and fires off an email to = an admin email address if the breaker is tripped. =

 =

That = CircuitBreakerService might also be picked by the Kontainer instance = which listens for on/off control events from the outside world.  = Some thinking to do there but they are tractable problems with services = and events.

 =

The main = services like CircuitBreakerService and ThrottlerService might register = themselves as providers with the ErrorHandlerService which would catch = the types of exceptions they throw.  It in turn could listen for = custom ExceptionHandlerListener<T> that listen for and handle = specific exception types. Still thinking and hand waving about this but = I think a sane set of standard services, listeners and events could be = created that would permit a user to create simple handlers to = register.

 =

There would = also be the issue of the issue of how to automate injection of those = into the Camel routes.  That doesn=E2=80=99t seem like it should be = a daunting challenge but it would be important.  And I think very = important that those get injected automatically even if the services = only provide basic logging initially with no client custom = code.

 =

From:<= /b> = James Carman [mailto:james@carmanconsulting.= com]
Sent: Friday, January 13, 2017 12:12 = PM
To: bradjohn@redhat.com
Subject= : Re: Opinionated...

 

Commons Lang3 has a pretty simple = CircuitBreaker implementation that I used in Microbule:

https://github.com/Microbule/microbule/blob/master/deco= rator/circuitbreaker/src/main/java/org/microbule/decorator/circuitbreaker= /CircuitBreakerFilter.java

On Fri, Jan 13, 2017 at 1:05 PM Brad Johnson <bradjohn@redhat.com> = wrote:

Folks,<= /o:p>

 <= /o:p>

I wanted to = make sure that my promoting CDI, Camel Java DSL, & static profiles = didn=E2=80=99t obscure the point I was trying to make.  Whatever = mechanics we choose I=E2=80=99d really like us to be unified behind a = common paradigm so that our documentation, exemplars, archetypes, blogs, = libraries, and so on are all organized the same and use the same = mechanics and layouts for projects.

 <= /o:p>

We should = promote an idiomatic way to develop software using Karaf Boot.  = That=E2=80=99s one problem I hear from a lot of clients.  There are = such cross-currents of information about how to develop OSGi-based = software that it gets confusing.  Best or preferred practices are = lost in the noise.  I won=E2=80=99t get into all that since = I=E2=80=99m sure most of you have dealt this problem. Not to pick on it = but a good example is that the Camel in Action book recommends using = Pojos instead of using Processors/Exchanges.  It is on somewhere = near the back of the book in a few pages. I don=E2=80=99t know how many = examples on the web site actually use the Processor/Exchange but it is a = lot. Then there are examples with Spring, Blueprint, Java DSL, Scala, = etc.  There are annotations that only work in one environment but = not in all of them.

 <= /o:p>

By = selecting an idiomatic and =E2=80=9Copinionated=E2=80=9D way of creating = Karaf Boot microcontainers we could make sure that sort of confusion = isn=E2=80=99t continued forward.  It would require a lot less = documentation to cover the same ground and make editing and updating = easier.  It would make creating sample and example projects a lot = easier. It would simplify what Karaf Boot appliances have to support and = make sure there aren=E2=80=99t concerns that work in one environment and = not in another or that might work differently in a different = environment.

 <= /o:p>

I=E2=80=99m = personally interested in Karaf Appliances with standard Maven = structures, standard  bundle structures, and reference = implementations that have a good chunk of the basic functionality. = I=E2=80=99d say we take a page from the =E2=80=9Cconvention over = configuration=E2=80=9D book or, at least, a =E2=80=9Cconventional = configuration=E2=80=9D and likely a bit of both. Because the appliances = are focused on microservices we should get out ahead of the Gartner hype = cycle.  Right now we are at the Peak of Inflated Expectations and = in a couple of years we=E2=80=99ll be at the Trough of = Disillusionment.  That disillusionment will come for a number of = reasons. Flying Spaghetti Monster topology will be one of them but, more = importantly for a Karaf Appliance, is the consistent problem of = =E2=80=9Cnetwork fallacies=E2=80=9D.  Every Karaf Kontainer should = have standard OSGi service interfaces and basic implementations that = address each of the fallacies that apply to a uService.  The = Kontainers should insist on it and not make it optional. If the user = doesn=E2=80=99t want that functionality they would then need to disable = via configuration.  But the Kontainer will get stuck in a grace = period and then fail if an expected, standard service isn=E2=80=99t = available. All of the standard OSGi service APIs would have basic = implementations to start but as more specific Kontainers.  But, = because they are standard services new ones can be developed by the = community or by the end developer.

 <= /o:p>

As = developers, we=E2=80=99ve all had to implement functionality and then = come back and deal with error handling, security, etc. I say we simply = cut those services in to the Kontainer right from the get-go.  The = Kontainer doesn=E2=80=99t run if it doesn=E2=80=99t find the = service.  That isn=E2=80=99t to say these become a fundamental part = of Karaf but a fundamental part of the Kontainer service that runs in = Karaf.

 <= /o:p>

The = standard bundles would only implement basic functionality and not do = anything sophisticated.  New bundles and libraries for more = sophisticated implementations could be added later. All of the bundles = would likely have disable flags if the developer found the particular = concern irrelevant.  For example, security might not be relevant. = The following aren=E2=80=99t meant to be comprehensive. Just addressing = key concerns. Other standards like LoggingService might be included by = default as well.

 <= /o:p>

The intent = here isn=E2=80=99t to define the exact mechanics but the standard OSGi = service interface that would be _required_ in any implementation = of a Kontainer, even if the implemented bundle is simply a passthrough = or can be disabled, it forces the developer to explicitly deal with the = problems or choose to ignore them altogether.  

 <= /o:p>

Because = these service interfaces and the bundles that implement them are = standard, the set can be specified by the dependencies specified in the = Maven build, features and/or profiles.

 <= /o:p>

  1. The network is = reliable.

A standard = =E2=80=9CError Handler=E2=80=9D OSGi service.  The default bundle = would simply capture errors/exceptions and log them.  Perhaps it = would specify retries. Drop in solutions might include errors going to = dead letter queues and so on. The OSGi service interface is required for = Kontainer bootstrap so use the default or use a standard one or create = one of your own.  If they want to change configuration of this = bundle or put in a new one, they know exactly what it is, where it = exists, how it is specified to the build, and what configuration file is = associated with it. No rummaging around through code.  When the = inevitable error, exceptions and problems arise, the developer = isn=E2=80=99t left wondering where and how they should add the = functionality to handle it.

 <= /o:p>

A standard = =E2=80=9CCircuit Breaker=E2=80=9D service API and basic implemented = bundle should be provided.  Perhaps the standard bundle would = simply count errors over a time frame and shut down if that limit is hit = and allow those values to be configured. Default would be a rather = unsophisticated implementation but provide the convention and automated = wiring of a circuit breaker OSGi service.  Other implementations = might fire off emails to Sys Admins or be combinations. And if it is = really undesirable, set a disable flag.

 <= /o:p>

  1. Latency is zero.

A standard = OSGi Throttling service interface and bundle implementation would be = included.  If you want different behavior, change it.  If you = want to disable it, set the flag. However, there are bigger issues here = that I=E2=80=99ll address a bit more down below.

 <= /o:p>

  1. Bandwidth is = infinite.

Throttling = OSGi service again. Ditto to comment 2.

 <= /o:p>

  1. The network is = secure.

Standard = OSGi service to plug in in various authentication/authorization = mechanisms.  By default it might be pass through but also have a = different implementation that uses a simple username/password. Obviously = LDAP, JAAS, and other bundles could be created and dropped into place. =

 <= /o:p>

  1. Topology doesn't = change.

Back to the = Circuit Breaker, logging and perhaps notification mechanism.  Also = the transport issue below where I=E2=80=99ll mention some = configuration.

 <= /o:p>

  1. There is one = administrator.

//No = particular plugin for this but standardized configuration and expected = bundles help and this also relates to the transport = discussion.

 <= /o:p>

  1. Transport cost is = zero.

//Probably = not a concern here directly but will be a big issue of = uServices.

 <= /o:p>

  1. The network is = homogeneous.

//I think = this issue can be dealt with in our context with many of the standard = libraries but can be abstracted a bit more.

 <= /o:p>

Obviously a = big issue we=E2=80=99ll see, and I=E2=80=99ve seen in the past, is = chained request/response calls. Service 1 making a REST call to service = 2 making a REST call to service 3=E2=80=A6etc.  And all of a sudden = the latency is a killer.

 <= /o:p>

ServiceMix/K= araf/Camel can already abstract away some of that via property = substitution. I=E2=80=99d suggest we take that one step further and put = _all_ transport/protocol information in configuration and create = a standardized URI. As a developer or a senior developer over a group of = developers, I don=E2=80=99t want them to be concerned with the fiddly = bits of the transport in the code and routes and I certainly = don=E2=80=99t want to recompile just to make such = changes.

 <= /o:p>

Akka, = for  example, uses local URIs like akka://.  But a similar = Karaf/Camel URI could be used and mapped via the configuration = files.  So the developer would always use karaf:// in their routes = and configuration mapping would use the URI specified.  = karaf://myserviceName.  In the configuration file might be mapped a = transport.configuration.cfg file.

 <= /o:p>

I believe = that is important for a lot of reasons.  A mid-level or = junior-level developer shouldn=E2=80=99t be involved in configuration = like:

"ftp://foo@myserver?password=3Dsecret&amp;<= /p>

        &nbs= p;  recursive=3Dtrue&amp;

        &nbs= p;  ftpClient.dataTimeout=3D30000&amp;

        &nbs= p;  ftpClientConfig.serverLanguageCode=3Dfr"=

 <= /o:p>

So the cfg = file might look like this:

clientServic= e=3D"ftp://foo@myserver?pas= sword=3Dsecret&

  =          = recursive=3Dtrue&

  =          = ftpClient.dataTimeout=3D30000&

  =          = ftpClientConfig.serverLanguageCode=3Dfr"

(At least = properties get rid of the gawdaful escaped ampersands).

 <= /o:p>

The code = would then say =E2=80=9Ckaraf://clientService=E2=80=9D

 <= /o:p>

One can do = much of that via configuration right now but I think it is critical to = move it completely to configuration so that admins know exactly what to = change and where to find it when topologies change. It also means that = when the backlash from microservice calling microservice calling = microservice being slow happens, that simple mapping would permit things = like going to JMS asynchronous request/response (or other fast, async = mechanisms) that don=E2=80=99t swamp the virtual machine=E2=80=99s or = Karaf instance resources. It would also allow for easy stubbing or mock = testing of the Kontainer as it will be deployed without using PAX exam = or other mechanism.

 <= /o:p>

Creating = standard OSGi service APIs in an anticipation of these problems would = permit for an evolutionary approach to these problems in the future and = specific solutions when a standard Kontainer is developed. Even standard = error handler service implementations can be created.

 <= /o:p>

Once such a = basic, standard Kontainer exists, then uKontainers that implement basic = functionality commonly used could be created.  There are JPA = examples already.  But the average developer is going to be given a = task to receive some canonical data model via a REST service and poke it = into a database.  That database model probably won=E2=80=99t look = like what they are receiving.  So a uKontainer that has a REST = front end they can modify, a Dozer object mapping file in the middle = with a transform, and a call to the database will be used = repeatedly.

 <= /o:p>

It may be = that Oracle, MySQL, BerklyDB, and so on each endup with different error = handler plugin implementations which are used with the same REST, = mapping, JPA container. Just change the Maven dependency or = profile.

 <= /o:p>

There are a = large number of examples like that.  In the case of that uKontainer = there would like be a JPAErrorService for catching common errors and = another for Dozer errors and for unmarshaling errors.  As a = developer looking to solve very specific problems, I just download the = uContainer and do the Dozer mapping, change some configuration and then = test it.

 <= /o:p>

That also = means, that much like Camel EIPs, open source developers can focus on = hardening these containers, fixing bugs, putting in performance = enhancements and the like.  If a new error is coming from JPA that = a user finds and isn=E2=80=99t being handled in a coherent fashion, then = a new block or delegate code is added and released.  Just as = we=E2=80=99d do with a Camel endpoint or component.

 <= /o:p>

Having = standard error handlers built into uKontainers would also help make = coherent messages from the large and unwieldy stack traces full of = reflection that we commonly see.  The error handler OSGi plugin for = a given problem would be highly focused on identifying and reporting = problems with a specific technology or set of = technologies.

 <= /o:p>

 <= /o:p>

 <= /o:p>

https://en.wikipedia.org/wiki/Fallacies_of_distributed_= computing

 

------=_NextPart_000_0AEA_01D26E36.8D3F9750--