geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Smith <dsm...@pivotal.io>
Subject Re: Addition of rmi-io library
Date Wed, 24 Jan 2018 17:47:54 GMT
>
> the content is going over the wire on whatever port that was port before.


>From what I see, DownloadJarFunction is calling
SimpleRemoteInputStream.export() which will call
UnicastRemoteObject.exportObject. That's an RMI call to start a tcp server
socket listening for connections to interact with that object.

-Dan

On Tue, Jan 23, 2018 at 6:15 PM, Jinmei Liao <jiliao@pivotal.io> wrote:

> As far as I can see, we are utilizing the streaming capability provided by
> the rmi-io, the content is going over the wire on whatever port that was
> port before. When streaming content from the gfsh to the jmxManager, it's
> using the jmx port; when getting jars between locator/servers, it's using
> the FunctionService, so it's whatever communication channel that
> FunctionService is using.
>
> All the FileContent are saved in temp folder, and get cleaned up after each
> deployment.
>
> On Tue, Jan 23, 2018 at 3:17 PM, Dan Smith <dsmith@pivotal.io> wrote:
>
> > I don't have an issue with the dependency. But if we are opening up new
> > ports for RMI connections, that seems like a potential security risk. If
> > someone has enabled cluster SSL we shouldn't be opening up an insecure
> port
> > for RMI connections.
> >
> > We should also make sure this is not leaking open sockets/file
> decriptors.
> > How does this SimpleRemoteInputStream we are creating get shutdown and
> > cleaned up?
> >
> > -Dan
> >
> > On Tue, Jan 23, 2018 at 2:36 PM, Jens Deppe <jensdeppe@apache.org>
> wrote:
> >
> > > Apologies that this was not raised earlier in discussion but I'm happy
> to
> > > describe it now.
> > >
> > > *Background:*
> > >
> > > When deploying jars into Geode they are moved through the system as
> > simple
> > > byte[] blobs. This obviously consumes memory. The various affected
> areas
> > > are:
> > >
> > > - gfsh reads the jars into memory
> > > - the jars are pushed to the locator (via a jmx call) - again creating
> a
> > > byte[] blob on the locator
> > > - from the locator, the jars are pushed to all servers via a function
> > call
> > > (also sending the jars as byte[] blobs).
> > >
> > > Obviously if the jar is small this would not be a problem, however in
> > > memory constrained systems or with large jars this is obviously going
> to
> > > put pressure on memory and possibly result in OOM situations. In fact,
> > the
> > > reason this came up was that some folks were unable to deploy a 40Mb
> jar
> > to
> > > a 512Mb (heap) locator.
> > >
> > > *rmi-io:*
> > >
> > > After doing some research, it seemed that the ideal solution would be
> > > something that allows for serializing Input/OutputStreams. Java doesn't
> > > provide anything natively.
> > >
> > > One library that stood out as being robust and feature complete was
> > rmi-io
> > > [1]. This allows for serializing a remote Input/OutputStream object
> which
> > > then lets us completely avoid having to pull deploying jars into memory
> > > everywhere. Under the covers it uses RMI and allows for either
> 'pulling'
> > or
> > > 'pushing' data. The reference page [2] has nice sequence diagrams.
> > >
> > > If anyone sees any issues with this, please do raise them. The current
> > > usage of this has not changed any user-facing interaction so ultimately
> > > changing the actual implemented fix for this problem (if we needed to)
> > > would not have any external effect.
> > >
> > > Thanks
> > > --Jens
> > >
> > > [1] http://openhms.sourceforge.net/rmiio
> > > [2] http://openhms.sourceforge.net/rmiio/class_reference.html
> > >
> >
>
>
>
> --
> Cheers
>
> Jinmei
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message