geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jens Deppe <jde...@pivotal.io>
Subject Re: Addition of rmi-io library
Date Thu, 25 Jan 2018 20:53:19 GMT
To address this we have two Jiras tracking the issue:

GEODE-4370 - jar deployment should not require additional ports to be opened
GEODE-4379 - gfsh deploy should push jars not have the server pull them

Essentially these will result in no new RMI ports being used/opened.

Code for 4370 is already checked in and 4379 is currently being tested.

--Jens

On Wed, Jan 24, 2018 at 11:42 AM, Anthony Baker <abaker@pivotal.io> wrote:

> Mike,
>
> ??  I think you meant 1.4.0 ??
>
> Let’s keep the release discussion on the VOTE thread.  IMO, adding another
> port has implications for security, firewalls, containers, bind addresses,
> etc.  I haven’t cast my release vote yet since I’m still considering these
> things.
>
> Anthony
>
>
> > On Jan 24, 2018, at 11:24 AM, Michael Stolz <mstolz@pivotal.io> wrote:
> >
> > There is going to be a 9.3.1 shortly after 9.3.0. Lets not hold 9.3.0 for
> > this.
> >
> > --
> > Mike Stolz
> > Principal Engineer - Gemfire Product Manager
> > Mobile: 631-835-4771
> >
> > On Jan 24, 2018 10:58 AM, "Jinmei Liao" <jiliao@pivotal.io> wrote:
> >
> > yeah, Jens just found that out too. It's opening up a new port in either
> > server/server and gfsh/jmManager cases. I think he has a solution to it
> and
> > we will get it in soon.
> >
> > On Wed, Jan 24, 2018 at 9:47 AM, Dan Smith <dsmith@pivotal.io> wrote:
> >
> >>>
> >>> the content is going over the wire on whatever port that was port
> > before.
> >>
> >>
> >> From what I see, DownloadJarFunction is calling
> >> SimpleRemoteInputStream.export() which will call
> >> UnicastRemoteObject.exportObject. That's an RMI call to start a tcp
> server
> >> socket listening for connections to interact with that object.
> >>
> >> -Dan
> >>
> >> On Tue, Jan 23, 2018 at 6:15 PM, Jinmei Liao <jiliao@pivotal.io> wrote:
> >>
> >>> As far as I can see, we are utilizing the streaming capability provided
> >> by
> >>> the rmi-io, the content is going over the wire on whatever port that
> was
> >>> port before. When streaming content from the gfsh to the jmxManager,
> > it's
> >>> using the jmx port; when getting jars between locator/servers, it's
> > using
> >>> the FunctionService, so it's whatever communication channel that
> >>> FunctionService is using.
> >>>
> >>> All the FileContent are saved in temp folder, and get cleaned up after
> >> each
> >>> deployment.
> >>>
> >>> On Tue, Jan 23, 2018 at 3:17 PM, Dan Smith <dsmith@pivotal.io> wrote:
> >>>
> >>>> I don't have an issue with the dependency. But if we are opening up
> > new
> >>>> ports for RMI connections, that seems like a potential security risk.
> >> If
> >>>> someone has enabled cluster SSL we shouldn't be opening up an insecure
> >>> port
> >>>> for RMI connections.
> >>>>
> >>>> We should also make sure this is not leaking open sockets/file
> >>> decriptors.
> >>>> How does this SimpleRemoteInputStream we are creating get shutdown and
> >>>> cleaned up?
> >>>>
> >>>> -Dan
> >>>>
> >>>> On Tue, Jan 23, 2018 at 2:36 PM, Jens Deppe <jensdeppe@apache.org>
> >>> wrote:
> >>>>
> >>>>> Apologies that this was not raised earlier in discussion but I'm
> >> happy
> >>> to
> >>>>> describe it now.
> >>>>>
> >>>>> *Background:*
> >>>>>
> >>>>> When deploying jars into Geode they are moved through the system
as
> >>>> simple
> >>>>> byte[] blobs. This obviously consumes memory. The various affected
> >>> areas
> >>>>> are:
> >>>>>
> >>>>> - gfsh reads the jars into memory
> >>>>> - the jars are pushed to the locator (via a jmx call) - again
> >> creating
> >>> a
> >>>>> byte[] blob on the locator
> >>>>> - from the locator, the jars are pushed to all servers via a
> > function
> >>>> call
> >>>>> (also sending the jars as byte[] blobs).
> >>>>>
> >>>>> Obviously if the jar is small this would not be a problem, however
> > in
> >>>>> memory constrained systems or with large jars this is obviously
> > going
> >>> to
> >>>>> put pressure on memory and possibly result in OOM situations. In
> >> fact,
> >>>> the
> >>>>> reason this came up was that some folks were unable to deploy a
40Mb
> >>> jar
> >>>> to
> >>>>> a 512Mb (heap) locator.
> >>>>>
> >>>>> *rmi-io:*
> >>>>>
> >>>>> After doing some research, it seemed that the ideal solution would
> > be
> >>>>> something that allows for serializing Input/OutputStreams. Java
> >> doesn't
> >>>>> provide anything natively.
> >>>>>
> >>>>> One library that stood out as being robust and feature complete
was
> >>>> rmi-io
> >>>>> [1]. This allows for serializing a remote Input/OutputStream object
> >>> which
> >>>>> then lets us completely avoid having to pull deploying jars into
> >> memory
> >>>>> everywhere. Under the covers it uses RMI and allows for either
> >>> 'pulling'
> >>>> or
> >>>>> 'pushing' data. The reference page [2] has nice sequence diagrams.
> >>>>>
> >>>>> If anyone sees any issues with this, please do raise them. The
> >> current
> >>>>> usage of this has not changed any user-facing interaction so
> >> ultimately
> >>>>> changing the actual implemented fix for this problem (if we needed
> >> to)
> >>>>> would not have any external effect.
> >>>>>
> >>>>> Thanks
> >>>>> --Jens
> >>>>>
> >>>>> [1] http://openhms.sourceforge.net/rmiio
> >>>>> [2] http://openhms.sourceforge.net/rmiio/class_reference.html
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Cheers
> >>>
> >>> Jinmei
> >>>
> >>
> >
> >
> >
> > --
> > Cheers
> >
> > Jinmei
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message