From dev-return-27795-archive-asf-public=cust-asf.ponee.io@geode.apache.org Wed Jan 24 20:43:01 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 68337180630 for ; Wed, 24 Jan 2018 20:43:01 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 5877E160C3C; Wed, 24 Jan 2018 19:43:01 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 78542160C39 for ; Wed, 24 Jan 2018 20:43:00 +0100 (CET) Received: (qmail 27468 invoked by uid 500); 24 Jan 2018 19:42:59 -0000 Mailing-List: contact dev-help@geode.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@geode.apache.org Delivered-To: mailing list dev@geode.apache.org Received: (qmail 27455 invoked by uid 99); 24 Jan 2018 19:42:58 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Jan 2018 19:42:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 6088B180337 for ; Wed, 24 Jan 2018 19:42:58 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.02 X-Spam-Level: X-Spam-Status: No, score=-0.02 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=pivotal-io.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id uj_i1MA44gwq for ; Wed, 24 Jan 2018 19:42:56 +0000 (UTC) Received: from mail-pg0-f46.google.com (mail-pg0-f46.google.com [74.125.83.46]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 49EDB5F1B3 for ; Wed, 24 Jan 2018 19:42:56 +0000 (UTC) Received: by mail-pg0-f46.google.com with SMTP id m136so3411519pga.12 for ; Wed, 24 Jan 2018 11:42:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pivotal-io.20150623.gappssmtp.com; s=20150623; h=from:content-transfer-encoding:mime-version:subject:date:references :to:in-reply-to:message-id; bh=wcjLK6Im9csMSc+Us2rz45m8PGSwY4pdbi1FSiy0SLM=; b=Tjd9hnVpdAGF4skJMDAxbqr2GvMEhYAwi0FSGPkjl2yTCT1WBnUsXZhKCtx4x36O6/ 0e1JZzqWOuz/u9lYO23aibvW0DnABdfcJ9uzE72hzIqQl5f1y/gR2HBxNjQAXVBhnDB/ 7m5+cEuy68FsWX/dk1z8WzlKxBJGeQHhiTtKakkkDkyNux4RrxT3jw8JIjn/rQOVDnGJ REuWIZlxYKXCX5M1kyv4qxh0AX+qvGOiv00IdcBwUK+7P38cZz+1q2fCNoCsPOoEhVS+ xgvTVzBNOv9Pc90M8ciEEAbO9ZWVI5PWGeuN9sKd1gAioJxYGnqQaFB74nhs52vT25nU TKJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:date:references:to:in-reply-to:message-id; bh=wcjLK6Im9csMSc+Us2rz45m8PGSwY4pdbi1FSiy0SLM=; b=f3ujGylxjLYf7lX7yBraCe+YPsLoBlvOiYBFJoZdH++eWkye/yVGfJb65EeAuhb8sy 26e6auGb5+kk1SC5HrC+4mIC5bS5ZCN3crcEImuKtbMI3UGqJsDadwZljyeRdRqwl9x3 wOjDm9qKEabMOm07t+RAmHQY7sNT8xj9XCs8Xqxx8zRSebbHrotUm/UdDgUOh6yYirii iiqpPT7UF9z4u3bM4FIizQpK5GvyUvSJI0b8/LM98Q6vEn3NsaNcizu9D6O2ZijXk52d 0Cv90+GPSAJDsUkqRcl0dIHY3PkdsPl5miFOgmXWTTraju8d6We+o0CpFMNNEc3llHgH +P+Q== X-Gm-Message-State: AKwxytc0OAakm6tOG6ehe/kq3B/Xt28x6e6o01R1n11Jmtpz4pHHFAhw iNvDAxfestqDn3k13ffcGxY7DoNzidQ= X-Google-Smtp-Source: AH8x226B9N8w6OqDVHWAlZcCxuVZIj0FgB5T0K2Y1CKd0tNxqLb4OkXF+fLHD18jauYHbze/J43XHw== X-Received: by 10.101.96.3 with SMTP id m3mr11390905pgu.409.1516822969065; Wed, 24 Jan 2018 11:42:49 -0800 (PST) Received: from [10.118.19.38] (50-203-225-134-static.hfc.comcastbusiness.net. [50.203.225.134]) by smtp.gmail.com with ESMTPSA id v24sm1312296pgc.82.2018.01.24.11.42.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Jan 2018 11:42:48 -0800 (PST) From: Anthony Baker Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: Addition of rmi-io library Date: Wed, 24 Jan 2018 11:42:46 -0800 References: To: dev@geode.apache.org In-Reply-To: Message-Id: <781F8C2F-814E-4256-BE08-82B490A79818@pivotal.io> X-Mailer: Apple Mail (2.3445.4.7) Mike, ?? I think you meant 1.4.0 ?? Let=E2=80=99s keep the release discussion on the VOTE thread. IMO, = adding another port has implications for security, firewalls, = containers, bind addresses, etc. I haven=E2=80=99t cast my release vote = yet since I=E2=80=99m still considering these things. Anthony > On Jan 24, 2018, at 11:24 AM, Michael Stolz wrote: >=20 > There is going to be a 9.3.1 shortly after 9.3.0. Lets not hold 9.3.0 = for > this. >=20 > -- > Mike Stolz > Principal Engineer - Gemfire Product Manager > Mobile: 631-835-4771 >=20 > On Jan 24, 2018 10:58 AM, "Jinmei Liao" wrote: >=20 > yeah, Jens just found that out too. It's opening up a new port in = either > server/server and gfsh/jmManager cases. I think he has a solution to = it and > we will get it in soon. >=20 > On Wed, Jan 24, 2018 at 9:47 AM, Dan Smith wrote: >=20 >>>=20 >>> the content is going over the wire on whatever port that was port > before. >>=20 >>=20 >> =46rom what I see, DownloadJarFunction is calling >> SimpleRemoteInputStream.export() which will call >> UnicastRemoteObject.exportObject. That's an RMI call to start a tcp = server >> socket listening for connections to interact with that object. >>=20 >> -Dan >>=20 >> On Tue, Jan 23, 2018 at 6:15 PM, Jinmei Liao = wrote: >>=20 >>> As far as I can see, we are utilizing the streaming capability = provided >> by >>> the rmi-io, the content is going over the wire on whatever port that = was >>> port before. When streaming content from the gfsh to the jmxManager, > it's >>> using the jmx port; when getting jars between locator/servers, it's > using >>> the FunctionService, so it's whatever communication channel that >>> FunctionService is using. >>>=20 >>> All the FileContent are saved in temp folder, and get cleaned up = after >> each >>> deployment. >>>=20 >>> On Tue, Jan 23, 2018 at 3:17 PM, Dan Smith = wrote: >>>=20 >>>> I don't have an issue with the dependency. But if we are opening up > new >>>> ports for RMI connections, that seems like a potential security = risk. >> If >>>> someone has enabled cluster SSL we shouldn't be opening up an = insecure >>> port >>>> for RMI connections. >>>>=20 >>>> We should also make sure this is not leaking open sockets/file >>> decriptors. >>>> How does this SimpleRemoteInputStream we are creating get shutdown = and >>>> cleaned up? >>>>=20 >>>> -Dan >>>>=20 >>>> On Tue, Jan 23, 2018 at 2:36 PM, Jens Deppe >>> wrote: >>>>=20 >>>>> Apologies that this was not raised earlier in discussion but I'm >> happy >>> to >>>>> describe it now. >>>>>=20 >>>>> *Background:* >>>>>=20 >>>>> When deploying jars into Geode they are moved through the system = as >>>> simple >>>>> byte[] blobs. This obviously consumes memory. The various affected >>> areas >>>>> are: >>>>>=20 >>>>> - gfsh reads the jars into memory >>>>> - the jars are pushed to the locator (via a jmx call) - again >> creating >>> a >>>>> byte[] blob on the locator >>>>> - from the locator, the jars are pushed to all servers via a > function >>>> call >>>>> (also sending the jars as byte[] blobs). >>>>>=20 >>>>> Obviously if the jar is small this would not be a problem, however > in >>>>> memory constrained systems or with large jars this is obviously > going >>> to >>>>> put pressure on memory and possibly result in OOM situations. In >> fact, >>>> the >>>>> reason this came up was that some folks were unable to deploy a = 40Mb >>> jar >>>> to >>>>> a 512Mb (heap) locator. >>>>>=20 >>>>> *rmi-io:* >>>>>=20 >>>>> After doing some research, it seemed that the ideal solution would > be >>>>> something that allows for serializing Input/OutputStreams. Java >> doesn't >>>>> provide anything natively. >>>>>=20 >>>>> One library that stood out as being robust and feature complete = was >>>> rmi-io >>>>> [1]. This allows for serializing a remote Input/OutputStream = object >>> which >>>>> then lets us completely avoid having to pull deploying jars into >> memory >>>>> everywhere. Under the covers it uses RMI and allows for either >>> 'pulling' >>>> or >>>>> 'pushing' data. The reference page [2] has nice sequence diagrams. >>>>>=20 >>>>> If anyone sees any issues with this, please do raise them. The >> current >>>>> usage of this has not changed any user-facing interaction so >> ultimately >>>>> changing the actual implemented fix for this problem (if we needed >> to) >>>>> would not have any external effect. >>>>>=20 >>>>> Thanks >>>>> --Jens >>>>>=20 >>>>> [1] http://openhms.sourceforge.net/rmiio >>>>> [2] http://openhms.sourceforge.net/rmiio/class_reference.html >>>>>=20 >>>>=20 >>>=20 >>>=20 >>>=20 >>> -- >>> Cheers >>>=20 >>> Jinmei >>>=20 >>=20 >=20 >=20 >=20 > -- > Cheers >=20 > Jinmei