Mailing-List: contact dev-help@river.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@river.apache.org
Message-ID: <577C8F76.8090900@zeus.net.au>
Date: Wed, 06 Jul 2016 14:56:22 +1000
From: Peter <jini@zeus.net.au>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.28) Gecko/20120306 Thunderbird/3.1.20
MIME-Version: 1.0
To: dev@river.apache.org
Subject: Re: Lotj - languages other than java
References: <8c4027b022f2b000b7a21f70f2320658@org.tizen.email> <SNT150-W9476C2A193A26E199DEFD1C2390@phx.gbl> <B55D41D7-9032-46FF-B67A-E8C626C7B265@googlemail.com>
In-Reply-To: <B55D41D7-9032-46FF-B67A-E8C626C7B265@googlemail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
archived-at: Wed, 06 Jul 2016 04:56:50 -0000

Yes, so if we have a lookup registration & search protocol, rather than 
a lookup service proxy, clients in any language can use it.

Then we need to think about the kind of search capabilities developers 
might require.

Getting that right and writing up a spec will take time, but I think 
it's worth the effort, but it's not something I'll tackle on my own, to 
do it properly requires joint experience, so I'm hoping there's 
sufficient interest to kick something off.

Obtaining java proxy's would still be possible (as would using any other 
RPC or communication protocol), but these would be obtained directly 
from services, after authentication.

Regarding "River on the Internet", my understanding is there are two camps:

   1. Those who like River how it is, build and service existing
      deployments on top of IPv4 local networks.
   2. Those who see changes to network topology coming and realise we
      need to be prepared, or have already experienced issues with IPv4
      and NAT, this camp has been simplified to "River on the
      Internet".  Some already have implemented in house solutions - see
      Sim's earlier post.

This division has caused tensions and deadlock in the development 
community (at least this is how it appears to me).

Quoting http://www.ipv6now.com.au/primers/IPv6Advantages.php

> IPv6 offers the potential to build a much more powerful Internet, with 
> vastly larger scale compared to the current situation. Addresses in 
> IPv4 have only 32 bits, allowing for only about 4 billion addresses, 
> compared to 128-bit IPv6, with some 340 trillion, trillion, trillion 
> addresses.
>
> As well as increasing the address space, the IETF took the opportunity 
> to build additional features into the IPv6 specification. IPv6 has a 
> new feature called autoconfiguration. This feature allows a device to 
> generate an IPv6 address as soon as it is given power. Using this 
> 'link local' address, there is no immediate need for any other 
> infrastructure to allow that device to begin communicating via IPv6 on 
> its local network, including communications with another local host or 
> router. If an IPv6 router is present, any IPv6-capable device can 
> generate not only a local address, but a globally routable address, 
> allowing access to the wider Internet.
>
> Provision of sufficient address space will also allow re-establishment 
> of an end-to-end architecture in the Internet. The shortage of IPv4 
> addresses has caused widespread use of private address spaces, which 
> are not directly accessible from the Internet. Devices with IPv6 
> addresses and IPv6 connectivity can be directly reachable by their 
> address. Such an approach gives rise to the potential to move beyond 
> an "Internet of desktops" to an "Internet of Things" where device to 
> device communication becomes possible. A range of other capabilities 
> were included during the IPv6 development process, for instance 
> mandatory support for security via IPsec (Internet Protocol Security).
>
> While some of the new features possible in IPv6 based networks are 
> currently possible in IPv4 based networks, the critical exception is 
> that they do not support the scale that IPv6 does, making it difficult 
> or impossible to use them to meet current and future business 
> requirements. The network applications being considered as a basis for 
> new growth in industry productivity require a vastly higher scale of 
> implementation than IPv4 can deliver; thousands or millions of devices 
> and/or addresses.
>
> Comcast (a large cable operator based in the USA) moved to IPv6 
> because it was in need of over 100 million addresses. Simple 
> projections showed Comcast that the number of IP addresses that 
> Comcast would need in order to support its future growth in terms of 
> subscriber base, as well as to be able to leverage potential new 
> services, exceeded those available. In fact, estimations were that 
> within a few years, Comcast would have some 20 million video 
> customers, an average of 2.5 set-top boxes per customer, and 2 IP 
> addresses per box. If these estimates are correct, the company will be 
> needing over 100 million IP addresses.
>
> Manual intervention is the other critical element to be considered in 
> the context of implementing large scale networks. If manual set-up is 
> required for every device with an IP address, significant costs will 
> be incurred. In IPv4 based networks, this requirement has been 
> alleviated by the use of server based configuration of devices using 
> Dynamic Host Configuration Protocol (DHCP) which is able to 
> automatically allocate IP addresses to new devices on the network with 
> the parameters set by the network administrator. However, for this 
> approach to work, each new device must interact with a DHCP server, 
> which in the case of large-scale networks is resource- and 
> time-intensive. In contrast, IPv6 address allocation is done by the 
> device itself and can occur independently of a server, or in 
> conjunction with an IPv6 enabled router, as appropriate.

IPv6 is growing exponentially and is currently at 12% global 
deployment.  Global adoption grew from 5% to 10% in a single year.

River has no support for IPv6 Multicast Discovery, only IPv4.

River currently has JERI endpoints that support the use of IPv6 and 
Unicast IPv6 Discovery, but only supports IPv4 multicast discovery.  
River also supports https using JERI Endpoints, but has no support for 
Unicast discovery using https.  So I can register services with a lookup 
service / registrar, utilising JERI https endpoints, but I cannot 
contact the lookup server from outside my local network using https, 
even when firewall rules and routing permits it.

I have registered IPv6 Multicast addresses for River with IANA.

I have implemented IPv6 Multicast discovery for River (global 
announcement and local request & announcement protocols).

I have implemented https unicast discovery for River, so my lookup 
service can be from contacted from a public network, without resorting 
to REST.

IPv6 presents new security challenges, I have been plodding away 
addressing them.

IPv4 is history, IPv4 is pain, new starters have trouble with multicast 
network configuration.  When people in camp 1 find their network is 
migrating to IPv6, what will they do?

The dynamic stuff you like, IPv6 has that in spades, if we want River to 
just work out of the box, we need IPv6.

It's inevitable this deadlock will pass.

When people are ready to discuss IPv6 adoption, I'll have a reference 
solution ready.

I think any group of developers will have disagreements, using git 
allows later merging or cross pollination of separate development paths, 
which occur when disagreement causes developers to take separate paths.  
So using git will not avoid disagreement, but it will permit progress to 
continue, when it does occur.  I think it's the ability to deal with 
disagreement that can result in better long term solutions, the 
technical issues we deal with are minor, in comparison to human 
interaction across language, culture, communication, skill and 
experience differences.  We have more in common than difference, but as 
humans, we have a tendency to focus on differences.  Unfortunately 
people with different opinions each have valuable experiences to offer, 
so it's disappointing and costly to projects when differences are 
irreconcilable.

Regards,

Peter.

On 6/07/2016 4:33 AM, Tom Hobbs wrote:
> My personal jury on River On The Internet is still out...
>
> I like the broadcast UDP model of just firing up a lookup service and finding stuff.  That works well for me and my applications are typically contained within well defined and secure networks.  I can see the benefit of something similar for IoT devices, I can’t yet figure out what the really compelling use case is though.
>
> A language agnostic lookup service interface would be really nice though, even if it was reduced to just serving up information about what was available rather than service proxies for RMI etc.  I have wanted something like that for one of my recent projects but couldn’t find anything that would Just Work.  The closest I could find was consul.io, but even that required more configuration for multiple hosts than I really wanted and didn’t allow me to express arbitrary service characteristics.
>
> A REST interface for reggie could work well, but it would still need to have language adapters so you can talk to it in ‘your’ language.  I do dislike having to use some HTTP library in my Java/Scala code to talk to some service hosted on some remote JVM because it only exposes a HTTP(S) REST interface.  If my ecosystem is heterogeneous then I shouldn’t have to do that.
>
> If there was a small reggie executable that I could wrap in an init.d service or similar and run it on all my hosts, that would be brilliant and is what I am intending to pursue when I finally get some down time.  It would be interesting to see something like that in competition to ZooKeeper or Apache Curator.  But for me, it would have to be really lightweight (like consul.io) with the default config out of the box good enough for most internal-network use cases.  I would be quite happy to see things like the service proxy, code downloading etc, all stripped out and replaced with plugin-able options.
>
> The transaction service, spaces and so on aren’t on my radar right now.
>
>
>> On 5 Jul 2016, at 04:33, Bishnu Gautam<bishnu35@hotmail.com>  wrote:
>>
>> Hi Peter
>> It is great that you pointed out lookup locator issue in firewall and its potential solution. It would be great to see the developments in River in which they really focus to have lookup discovery beyond the firewall without requiring port forward and other demanding packet filtering techniques. Once this obstacle in River is crossed, I am pretty sure that there will be new paradigm shift in IoT or ICT technology. This technology has a tremendous potential. However, I never understand why River community never try to bring this issue on the first place. River in Internet would be the most wonderful solutions for millions of users around the world. Please think, discuss and try to work on it. It would be a great news for us.
>> RegardsBishnu
>>
>>
>> Bishnu Prasad Gautam
>>
>>
>>> Date: Mon, 4 Jul 2016 18:37:25 +1000
>>> From: jini@zeus.net.au
>>> Subject: Re: Lotj - languages other than java
>>> To: dev@river.apache.org
>>> CC: simon@qcg.nl
>>>
>>>
>>>
>>>
>>> Sim,
>>>
>>> I'd like to see the project return to the days where we had a number of active committers working together on the same goals.
>>>
>>> I've got a project on github, where I've continued work that was controversial, I'd like to bring this work back to the project if possible.  It has some minor breaking changes, but if backward compatibility was essential, it could be accommodated.
>>>
>>> Changes:
>>> * New secure discovery providers, including https among others, including added https protocol support for LookupLocator.  However since firewalls may not allow ipv6 multicast packets through, DNS-SD would be useful.
>>> * IPv6 Discovery, global and local groups.
>>> * Discovery V2 support added to LookupLocator's getRegistrar method.
>>> * Updated encryption ciphers, removal of insecure ones.
>>> * Deprecation of ProxyTrust et al.
>>> * New default methods added to ServiceRegistrar to allow establising trust with a service, prior to obtaining a service proxy, or Entry's (works with both maven codebase provisioning and traditional codebase downloads).
>>> * Input validation for untrusted serial data.
>>> * META-INF/permissions.perm files list permissions required by codebase, accessible from ClassLoader using mixin interface.
>>>
>>> I recall you had a need for a SocketFactory in LookupLocator, at that time LookupLocator only used discovery v1, which was deprecated, I'd like to include a way to enable SocketFactory support in discovery V2.  Note that LookupLocator isn't serialized during discovery.
>>>
>>> While the River codebase was a little difficult to understand at first, I find it's quite easy to work with now.
>>>
>>> While I'm a critic of Rivers flaws, addressing th em is straight forward, the greatest challenge is convincing everyone that flaws exist, or that they need addressing, there's still plenty of good stuff left worth saving.
>>>
>>> Regards,
>>>
>>> Peter.
>>>
>>>
>>> Sent from my Samsung device.
>>>
>>>   Include original message
>>> ---- Original message ----
>>> From: Peter<jini@zeus.net.au>
>>> Sent: 01/07/2016 04:35:16 pm
>>> To: dev@river.apache.org<dev@river.apache.org>
>>> Subject: Re: Lotj - languages other than java
>>>
>>>
>>>
>>>
>>>
>>> Thanks Sim,
>>>
>>> These are all good questions we need to consider.
>>>
>>> I like the model of micro services where each service is responsible for implementing its own back end persistence and state.  Do you consider a microservice to be web based?
>>>
>>> I have an implementation of discovery using multicast ipv6.  However for firewalls with limited open ports such as https over ipv6, we have JERI https endpoints, but no discovery, DNS-SD is a good discovery alternative waiting to be implemented.
>>>
>>> For my own environment I will be adopting ipv6 , the global address space and autoconfiguration solve many problems that users experience with ipv4 today.
>>>
>>> I admit the locked down api caused me frustration, but I think it's clear now that we need a process for managing api evolution.
>>>
>>> Complexity - The proxy preparation api tries to determine trust after downloading untrusted code and deserialization of unverified data.  As gadget attacks demonstrate, too little too late at great complexity.  This was an attempt to bolt security onto the existing lookup service.
>>>
>>> JERI is good, method constraints are good, proxy trust is obsolete.  River's current ssl and https JERI endpoints need to be brought up to date as they're no longer secure.  I've already done this work externally, it can be donated when appropriate for the project.
>>>
>>> If we address security issues, we can provide a secure alternative to RMI  Oracle has chosen 'whack a mole' security for Serialization, rather than address some fundamental design flaws with ObjectInputStream, for this reason, authentication of the source must occur prior to accepting serial data.  Despite common belief, white listing isn't a completely secure solution and adds conplexity as it's too fine grained.
>>>
>>> For multi language support, this would limit the type system, but then, there's a lot that can be done with strings, primitive types and byte arrays.  This doesn't have to limit java service types, I think language support should be something determined during lookup, so we don't limit the type systems of more powerful languages to primitives.
>>>
>>> Looking at most Entry's used for lookup, most fields are strings and integers.  If you look at the way lookup searches are implemented, an entry is represented by a string name and each field is a tuple name value pair.
>>>
>>> I think a ground up redesign of the lookup service, would address language compatibility as well as complexity and security.
>>>
>>> In other words, recognise the need for a lookup&  registration protocol, as well as discovery, after that, the service&  client should be able to negotiate  whatever rpc protocol they have in common and to do that, we'll also need a connection negotiation protocol.  We could write specifications for these protocols.  This way we could allow any language/ platform to register and provide services.  The code for lookup would not be downloaded as Reggie is now, it would be protocol, rather than proxy based.  This would also fit well with IoT.
>>>
>>> Meanwhile we can still support existing java based services.
>>>
>>> Thoughts?
>>>
>>> Peter.
>>>
>>> Sent from my Samsung device.
>>>
>>>   Include original message
>>> ---- Original message ----
>>> From: Simon IJskes - QCG<simon@qcg.nl>
>>> Sent: 30/06/2016 06:22:30 pm
>>> To: dev@river.apache.org
>>> Subject: Re: Lotj - languages other than java
>>>
>>> If you solve the 'barrier' of the service discovery, do you also want to
>>> provide universal access to the java services in the form of microservices?
>>>
>>> It is doable to take any 'more used' service discovery solution and use
>>> this as the river discovery. To introduce a level of abstraction with
>>> the same primitives as the current river discovery mechanism offers.
>>>
>>> River would then have adapted a more common discovery mechanism.
>>>
>>> Next thing that we should decide, is how far do we go into universality.
>>> I see univeral type systems, different serialisation plugins on the horizon.
>>>
>>> The biggest showstopper for me was the API compatibility. In order to
>>> make any progress we need a more agile process for modifing the API
>>>
>>> If we leave compatibility behind us, we could ask our selfs, what
>>> benefit are we providing for the users? What can we introduce that does
>>> not duplicate what is already in the market For a java developer, i
>>> think there is no need to convince, they can see benefits in just having
>>> a java API to program against. We need to think about the environment
>>> where java receives a lot of 'non-love', how we can create a 'whow, java
>>> isn't all that bad, look at that easy solution' experience.
>>>
>>> I think that river lost the spot it could have, as a java only solution
>>> to JSON, XMLRPC, SOAP, etc libraries for java. From a helicopter view,
>>> what does it do? Whe provide secure RPC, with discovery and scaling. And
>>> we make it hard to use.
>>>
>>> G. Simon
>>>
>>>
>>> On 30-06-16 05:37, Peter wrote:
>>>> Currently with River, you need java to participate.  Other languages can provide services, but you need a jvm to participate.
>>>>
>>>> Most of discovery is language agnostic, so any language can participate in discovery.
>>>>
>>>> The major limitation for other languages is the lookup service.  Security issues and complexity also relate to the lookup service.
>>>>
>>>> My thoughts are that a lookup service that performs search and registration, but provides a language independent  and secure means of contacting service providers would be beneficial.
>>>>
>>>> Anyone interested in discussing further?
>>>>
>>>> Regards,
>>>>
>>>> Peter.
>>>>
>>>>
>>>> Sent from my Samsung device.
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>> 		 	   		
>