geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jens Deppe <jde...@pivotal.io>
Subject Re: Addition of rmi-io library
Date Wed, 24 Jan 2018 19:14:09 GMT
Right - it needs to piggy-back on the existing JMX/RMI port in which case
no new ports will be required.

We're testing this now.

--Jens

On Wed, Jan 24, 2018 at 9:58 AM, Jinmei Liao <jiliao@pivotal.io> wrote:

> yeah, Jens just found that out too. It's opening up a new port in either
> server/server and gfsh/jmManager cases. I think he has a solution to it and
> we will get it in soon.
>
> On Wed, Jan 24, 2018 at 9:47 AM, Dan Smith <dsmith@pivotal.io> wrote:
>
> > >
> > > the content is going over the wire on whatever port that was port
> before.
> >
> >
> > From what I see, DownloadJarFunction is calling
> > SimpleRemoteInputStream.export() which will call
> > UnicastRemoteObject.exportObject. That's an RMI call to start a tcp
> server
> > socket listening for connections to interact with that object.
> >
> > -Dan
> >
> > On Tue, Jan 23, 2018 at 6:15 PM, Jinmei Liao <jiliao@pivotal.io> wrote:
> >
> > > As far as I can see, we are utilizing the streaming capability provided
> > by
> > > the rmi-io, the content is going over the wire on whatever port that
> was
> > > port before. When streaming content from the gfsh to the jmxManager,
> it's
> > > using the jmx port; when getting jars between locator/servers, it's
> using
> > > the FunctionService, so it's whatever communication channel that
> > > FunctionService is using.
> > >
> > > All the FileContent are saved in temp folder, and get cleaned up after
> > each
> > > deployment.
> > >
> > > On Tue, Jan 23, 2018 at 3:17 PM, Dan Smith <dsmith@pivotal.io> wrote:
> > >
> > > > I don't have an issue with the dependency. But if we are opening up
> new
> > > > ports for RMI connections, that seems like a potential security risk.
> > If
> > > > someone has enabled cluster SSL we shouldn't be opening up an
> insecure
> > > port
> > > > for RMI connections.
> > > >
> > > > We should also make sure this is not leaking open sockets/file
> > > decriptors.
> > > > How does this SimpleRemoteInputStream we are creating get shutdown
> and
> > > > cleaned up?
> > > >
> > > > -Dan
> > > >
> > > > On Tue, Jan 23, 2018 at 2:36 PM, Jens Deppe <jensdeppe@apache.org>
> > > wrote:
> > > >
> > > > > Apologies that this was not raised earlier in discussion but I'm
> > happy
> > > to
> > > > > describe it now.
> > > > >
> > > > > *Background:*
> > > > >
> > > > > When deploying jars into Geode they are moved through the system
as
> > > > simple
> > > > > byte[] blobs. This obviously consumes memory. The various affected
> > > areas
> > > > > are:
> > > > >
> > > > > - gfsh reads the jars into memory
> > > > > - the jars are pushed to the locator (via a jmx call) - again
> > creating
> > > a
> > > > > byte[] blob on the locator
> > > > > - from the locator, the jars are pushed to all servers via a
> function
> > > > call
> > > > > (also sending the jars as byte[] blobs).
> > > > >
> > > > > Obviously if the jar is small this would not be a problem, however
> in
> > > > > memory constrained systems or with large jars this is obviously
> going
> > > to
> > > > > put pressure on memory and possibly result in OOM situations. In
> > fact,
> > > > the
> > > > > reason this came up was that some folks were unable to deploy a
> 40Mb
> > > jar
> > > > to
> > > > > a 512Mb (heap) locator.
> > > > >
> > > > > *rmi-io:*
> > > > >
> > > > > After doing some research, it seemed that the ideal solution would
> be
> > > > > something that allows for serializing Input/OutputStreams. Java
> > doesn't
> > > > > provide anything natively.
> > > > >
> > > > > One library that stood out as being robust and feature complete was
> > > > rmi-io
> > > > > [1]. This allows for serializing a remote Input/OutputStream object
> > > which
> > > > > then lets us completely avoid having to pull deploying jars into
> > memory
> > > > > everywhere. Under the covers it uses RMI and allows for either
> > > 'pulling'
> > > > or
> > > > > 'pushing' data. The reference page [2] has nice sequence diagrams.
> > > > >
> > > > > If anyone sees any issues with this, please do raise them. The
> > current
> > > > > usage of this has not changed any user-facing interaction so
> > ultimately
> > > > > changing the actual implemented fix for this problem (if we needed
> > to)
> > > > > would not have any external effect.
> > > > >
> > > > > Thanks
> > > > > --Jens
> > > > >
> > > > > [1] http://openhms.sourceforge.net/rmiio
> > > > > [2] http://openhms.sourceforge.net/rmiio/class_reference.html
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Cheers
> > >
> > > Jinmei
> > >
> >
>
>
>
> --
> Cheers
>
> Jinmei
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message