From dev-return-27818-archive-asf-public=cust-asf.ponee.io@geode.apache.org Thu Jan 25 21:53:28 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id EFF56180651 for ; Thu, 25 Jan 2018 21:53:27 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id DE62B160C3D; Thu, 25 Jan 2018 20:53:27 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0794F160C17 for ; Thu, 25 Jan 2018 21:53:26 +0100 (CET) Received: (qmail 26887 invoked by uid 500); 25 Jan 2018 20:53:26 -0000 Mailing-List: contact dev-help@geode.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@geode.apache.org Delivered-To: mailing list dev@geode.apache.org Received: (qmail 26868 invoked by uid 99); 25 Jan 2018 20:53:25 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Jan 2018 20:53:25 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id DCB5E1A0D1F for ; Thu, 25 Jan 2018 20:53:24 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.98 X-Spam-Level: * X-Spam-Status: No, score=1.98 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=pivotal-io.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id uRdm0EclFrD0 for ; Thu, 25 Jan 2018 20:53:21 +0000 (UTC) Received: from mail-vk0-f49.google.com (mail-vk0-f49.google.com [209.85.213.49]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id B28B85F2C3 for ; Thu, 25 Jan 2018 20:53:20 +0000 (UTC) Received: by mail-vk0-f49.google.com with SMTP id f186so5718609vkc.1 for ; Thu, 25 Jan 2018 12:53:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pivotal-io.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=QNcxSCkwyszoQKrGfeiDLdEIqSQA65yPIWMVxumMGXM=; b=aVAWsUwP7b2zCX1pSirGyeblx5v5Gip0oXtwi3sJlcV78lLSd6gcwvpM6Gw+q+H/k6 L8VgFolnTOZeXacoh/ktu5EnOnn1Qo9MhVIh5eGRgMMd4kr/uHknlKIbYb5cGton7B// qWqfKCJN7k+gvQEa05//2YavnWbzpKvdM1K2AMSsucLi3Ql0ZWenQ+XBrGpNJEYNfq0H 7+cdeneoriSefI8zN6Gk8H1luTBxSbaaVnVsLToqTBgSQoPoSjvCehD6EgaaF9iFs/os W0tR+bHQs98QXs9MgqBuvl3mlxuHH/Bh+yXhGNZxwC1Me0xybgOCdwH59HZCSL6WukX4 Bo5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=QNcxSCkwyszoQKrGfeiDLdEIqSQA65yPIWMVxumMGXM=; b=bdRTueE83hg1ve9QpQ9zsEH1eX3WwkadgCcFv3sM1k9fgGFfitVN6kmr4EW72E2UCj 5OopYFDVznuOMO5Jyt0YOpMGY9YO8PHfSROeojldAoaKKVmBvMmbJesz0E6qFY/p41Zj r37SFQDrtXeQmkkR/1lQ3t6nx4xVyjJO37SKJPCogyzlq0xir+vQJGHnGt2bw5wLM2sf xlUAvDKARP6UTrVUQ6qBcd3JNAe1XPQvlZfnaoJCjG4O7uE5BsmxJA+C0TSCbeipZXas IBzoAUBBYquo0jII0pE+FUJ/PjR9VzlvpAVLm5y7zpjUKjdwRUvfECknEwnRB4TaOruy wI+g== X-Gm-Message-State: AKwxytediLwdce7yc12n1rqc81zPG0hHNqLOzM36WMZchYoIPuHsLR2d h9b0y4LSLs3FoQ0vK6Het5FtSxvQWaLj2s2B6WAp9QAM X-Google-Smtp-Source: AH8x224vDAG24I3z7Ki3TxKRWo3DAjPes893B8Bpe2rYLknnTjTTkfPIKvPeowenc0w/TMjUwq3Sbin3KtY9Wj4KjfE= X-Received: by 10.31.15.149 with SMTP id 143mr8256551vkp.126.1516913600035; Thu, 25 Jan 2018 12:53:20 -0800 (PST) MIME-Version: 1.0 Received: by 10.176.36.132 with HTTP; Thu, 25 Jan 2018 12:53:19 -0800 (PST) In-Reply-To: <781F8C2F-814E-4256-BE08-82B490A79818@pivotal.io> References: <781F8C2F-814E-4256-BE08-82B490A79818@pivotal.io> From: Jens Deppe Date: Thu, 25 Jan 2018 12:53:19 -0800 Message-ID: Subject: Re: Addition of rmi-io library To: dev@geode.apache.org Content-Type: multipart/alternative; boundary="001a11436f4049228305639ff8cc" --001a11436f4049228305639ff8cc Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable To address this we have two Jiras tracking the issue: GEODE-4370 - jar deployment should not require additional ports to be opene= d GEODE-4379 - gfsh deploy should push jars not have the server pull them Essentially these will result in no new RMI ports being used/opened. Code for 4370 is already checked in and 4379 is currently being tested. --Jens On Wed, Jan 24, 2018 at 11:42 AM, Anthony Baker wrote: > Mike, > > ?? I think you meant 1.4.0 ?? > > Let=E2=80=99s keep the release discussion on the VOTE thread. IMO, addin= g another > port has implications for security, firewalls, containers, bind addresses= , > etc. I haven=E2=80=99t cast my release vote yet since I=E2=80=99m still = considering these > things. > > Anthony > > > > On Jan 24, 2018, at 11:24 AM, Michael Stolz wrote: > > > > There is going to be a 9.3.1 shortly after 9.3.0. Lets not hold 9.3.0 f= or > > this. > > > > -- > > Mike Stolz > > Principal Engineer - Gemfire Product Manager > > Mobile: 631-835-4771 > > > > On Jan 24, 2018 10:58 AM, "Jinmei Liao" wrote: > > > > yeah, Jens just found that out too. It's opening up a new port in eithe= r > > server/server and gfsh/jmManager cases. I think he has a solution to it > and > > we will get it in soon. > > > > On Wed, Jan 24, 2018 at 9:47 AM, Dan Smith wrote: > > > >>> > >>> the content is going over the wire on whatever port that was port > > before. > >> > >> > >> From what I see, DownloadJarFunction is calling > >> SimpleRemoteInputStream.export() which will call > >> UnicastRemoteObject.exportObject. That's an RMI call to start a tcp > server > >> socket listening for connections to interact with that object. > >> > >> -Dan > >> > >> On Tue, Jan 23, 2018 at 6:15 PM, Jinmei Liao wrote= : > >> > >>> As far as I can see, we are utilizing the streaming capability provid= ed > >> by > >>> the rmi-io, the content is going over the wire on whatever port that > was > >>> port before. When streaming content from the gfsh to the jmxManager, > > it's > >>> using the jmx port; when getting jars between locator/servers, it's > > using > >>> the FunctionService, so it's whatever communication channel that > >>> FunctionService is using. > >>> > >>> All the FileContent are saved in temp folder, and get cleaned up afte= r > >> each > >>> deployment. > >>> > >>> On Tue, Jan 23, 2018 at 3:17 PM, Dan Smith wrote: > >>> > >>>> I don't have an issue with the dependency. But if we are opening up > > new > >>>> ports for RMI connections, that seems like a potential security risk= . > >> If > >>>> someone has enabled cluster SSL we shouldn't be opening up an insecu= re > >>> port > >>>> for RMI connections. > >>>> > >>>> We should also make sure this is not leaking open sockets/file > >>> decriptors. > >>>> How does this SimpleRemoteInputStream we are creating get shutdown a= nd > >>>> cleaned up? > >>>> > >>>> -Dan > >>>> > >>>> On Tue, Jan 23, 2018 at 2:36 PM, Jens Deppe > >>> wrote: > >>>> > >>>>> Apologies that this was not raised earlier in discussion but I'm > >> happy > >>> to > >>>>> describe it now. > >>>>> > >>>>> *Background:* > >>>>> > >>>>> When deploying jars into Geode they are moved through the system as > >>>> simple > >>>>> byte[] blobs. This obviously consumes memory. The various affected > >>> areas > >>>>> are: > >>>>> > >>>>> - gfsh reads the jars into memory > >>>>> - the jars are pushed to the locator (via a jmx call) - again > >> creating > >>> a > >>>>> byte[] blob on the locator > >>>>> - from the locator, the jars are pushed to all servers via a > > function > >>>> call > >>>>> (also sending the jars as byte[] blobs). > >>>>> > >>>>> Obviously if the jar is small this would not be a problem, however > > in > >>>>> memory constrained systems or with large jars this is obviously > > going > >>> to > >>>>> put pressure on memory and possibly result in OOM situations. In > >> fact, > >>>> the > >>>>> reason this came up was that some folks were unable to deploy a 40M= b > >>> jar > >>>> to > >>>>> a 512Mb (heap) locator. > >>>>> > >>>>> *rmi-io:* > >>>>> > >>>>> After doing some research, it seemed that the ideal solution would > > be > >>>>> something that allows for serializing Input/OutputStreams. Java > >> doesn't > >>>>> provide anything natively. > >>>>> > >>>>> One library that stood out as being robust and feature complete was > >>>> rmi-io > >>>>> [1]. This allows for serializing a remote Input/OutputStream object > >>> which > >>>>> then lets us completely avoid having to pull deploying jars into > >> memory > >>>>> everywhere. Under the covers it uses RMI and allows for either > >>> 'pulling' > >>>> or > >>>>> 'pushing' data. The reference page [2] has nice sequence diagrams. > >>>>> > >>>>> If anyone sees any issues with this, please do raise them. The > >> current > >>>>> usage of this has not changed any user-facing interaction so > >> ultimately > >>>>> changing the actual implemented fix for this problem (if we needed > >> to) > >>>>> would not have any external effect. > >>>>> > >>>>> Thanks > >>>>> --Jens > >>>>> > >>>>> [1] http://openhms.sourceforge.net/rmiio > >>>>> [2] http://openhms.sourceforge.net/rmiio/class_reference.html > >>>>> > >>>> > >>> > >>> > >>> > >>> -- > >>> Cheers > >>> > >>> Jinmei > >>> > >> > > > > > > > > -- > > Cheers > > > > Jinmei > > --001a11436f4049228305639ff8cc--