From dev-return-44622-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Thu Feb 7 23:50:55 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 34850180600 for ; Fri, 8 Feb 2019 00:50:55 +0100 (CET) Received: (qmail 62897 invoked by uid 500); 7 Feb 2019 23:50:54 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 62883 invoked by uid 99); 7 Feb 2019 23:50:53 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Feb 2019 23:50:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 2EFC9C6A34 for ; Thu, 7 Feb 2019 23:50:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.114 X-Spam-Level: * X-Spam-Status: No, score=1.114 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URI_HEX=1.313] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id C2Eb-4lfy2Wh for ; Thu, 7 Feb 2019 23:50:51 +0000 (UTC) Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 73C5C60D93 for ; Thu, 7 Feb 2019 23:50:50 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id f132so763996pfa.6 for ; Thu, 07 Feb 2019 15:50:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-transfer-encoding; bh=V+rEVFS97B920apmSQM3v2z7yDvg6A7Hljq/DzyJ+vA=; b=Ied2Uq3AhAwkNr6a7Ylva4xB5aCtGqRUwhIwG2rBMn/jyPYOo4wUk+g3SyxgzsY/QR XBjuJp/u0uaeacA9Vw1KwGNIMUxVhUWDNess1q5TgLJK1pdjUcQ57yBcagmQOtVBOHIF bdwYG2sa6t1mVTQp6RuU4vZsf/GsT0gENl2k+rDLLz9UQR2hvVumlxng6t915+Uw5dp6 Y0NCpaA9xmZTWnN2I0k9GrzcSHUtOyFzuhNNyDOnKytCTpR0kgjCWtbscl7quiI+q9mf q9YbETxgCMVnVHfzTRI/vPVWsM0Fwx+0iewqCgixI3dp7MYLMORgFD+xmS86YtSRRiWD IYnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=V+rEVFS97B920apmSQM3v2z7yDvg6A7Hljq/DzyJ+vA=; b=RcoYzQHY7DeGFW4INa6VgeTETaZ/j7UzLWiX/T+QKJgBXZyK1Z1+VbeoeNmFYgubg9 wrDXizKrQTDnJS9FMfL8Rxr54wrJCKo/Pz+KNAuvyFdDrPdOrs+3dh009a4PuIlz2jbs Z1rPg4QuTg98DO/EwsYyuRq7c7kTwovw+4cjDwsx+JB6k2ExrSt8zO3HbTgKcpz5HlcN MyEswd/NsvjSEKspgfmZ0ocobIrk0BllK8gYN4pxRcUGE0ZArlWDat/A2qrjp7chkJoX PytYehqh5jIpolM0jd7nWLqP3LegG9XaoVdeXdB7/QM54FFz/Kv5IqDUTcx/kq5QDEwp xNyQ== X-Gm-Message-State: AHQUAuY4DPhSxx9Bcs/FqaIecS2mJITMjPyQyD19BKJ4NJixctbAqEkd E7TRXACGQXFD807nXyBCOMOv52OYqtAb3up3R/7h7B9k//0= X-Google-Smtp-Source: AHgI3IYAIMuVNQbRVVnggxc4wnyUQVEFXy1E4q4FLSTf2yvJZkuZNThwRz+NLzBuWZ2Yd7LPiK5ObDOjZDTOufW2IQQ= X-Received: by 2002:a62:7042:: with SMTP id l63mr19562679pfc.89.1549583448417; Thu, 07 Feb 2019 15:50:48 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Vyacheslav Daradur Date: Fri, 8 Feb 2019 02:50:38 +0300 Message-ID: Subject: Re: Services hot redeployment To: dev@ignite.apache.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Denis, I've just prepared PR [1] with a possible implementation that covers p.1 from your list. Please, have a look at it. Pay attention to the test, it demonstrates manual services hot-redeployment via DeploymentSpi. Is that what you mean? [1] https://github.com/apache/ignite/pull/6060 On Tue, Feb 5, 2019 at 2:14 PM Denis Mekhanikov wro= te: > > In general, I think, we should work on the improvements in the following > order: > > 1. Cluster availability, when services are being updated. Should be solve= d > by usage of the DeploymentSpi > 2. Service availability, when they are being updated. This one will > probably require introduction of new API methods like > redeploy(ServiceConfiguration). > 3. Service versioning and packaging. > > I'd like to focus on the first point. Service grid will become much more > usable and mature once we implement it. > > Denis > > =D0=B2=D1=82, 5 =D1=84=D0=B5=D0=B2=D1=80. 2019 =D0=B3. =D0=B2 14:06, Deni= s Mekhanikov : > > > Vyacheslav, > > > > I think, we can use IgniteConfiguration#deploymentSpi for tasks and > > services. > > Or we can add an analogous property. > > > > Nik, > > > > > 1. Is it possible to change the list of deployed resources in runtime > > via built-in DeploymentSPI implementations? > > > Can I add or remove jar to(from) class-path without node(cluster) > > restart? > > Yes, this is the reason why the DeploymentSpi exists. But currently onl= y > > compute grid can use it. > > > > > 2. Can we update service dependencies via DeploymentSPI? commons-lang= , > > log4j or any other common library? > > Ideally such libraries should be loaded via app class loader at node > > startup. Otherwise the same libraries will be loaded multiple times. It > > will lead to a lot of memory leaks. > > I think, we can support loading of dependencies, but discourage users f= rom > > doing it. The proper way should be described in the documentation, and > > warnings could be generated, if too many classes get loaded via > > DeploymentSpi. > > > > > 3. User has to execute explicit Ignite API calls(undeploy(), deploy()= ) > > to renew service implementation. Is it correct? > > > I think we should develop some watcher, that will be watch for a > > resource change and redeploy services automatically. > > Correct. I don't like the idea to redeploy services automatically. I > > think, redeployment should be triggered explicitly. Non-obvious actions > > should be avoided. > > > > 4. Such feature would for sure improve usability of the service grid. B= ut > > it requires much more time and work to implement. > > I think, it's better not to expand the scope too much. Otherwise > > development will take another 6 moths. > > This is a great idea, and we will keep it in mind though. > > > > 5. Yep, we need an extensive documentation on the service deployment > > procedure. > > This feature may not be perfectly clear to users, so we need some how-t= os. > > > > Denis > > > > =D0=B2=D1=82, 5 =D1=84=D0=B5=D0=B2=D1=80. 2019 =D0=B3. =D0=B2 08:19, Ni= kolay Izhikov : > > > >> Hello, Denis. > >> > >> Thank you for this discussion. > >> I have a few notes: > >> > >> 1. Is it possible to change the list of deployed resources in runtime = via > >> built-in DeploymentSPI implementations? > >> Can I add or remove jar to(from) class-path without node(cluster) rest= art? > >> > >> 2. Can we update service dependencies via DeploymentSPI? commons-lang, > >> log4j or any other common library? > >> > >> 3. User has to execute explicit Ignite API calls(undeploy(), deploy())= to > >> renew service implementation. Is it correct? > >> I think we should develop some watcher, that will be watch for a resou= rce > >> change and redeploy services automatically. > >> > >> 4. DeploymentSPI is *node-wide* configuration. This means we change > >> classpath for all services with this SPI. > >> I think this is a huge limitation of the SPI. > >> We should provide an ability to configure service-wide classpath to ou= r > >> users as quickly as possible. > >> It is a common feature in modern service, task executor engines. > >> > >> I think the perfect world scenario would be following: > >> > >> 1. Start a client node or connect to a cluster with thin clien= t. > >> > >> 2. Configure service classpath with some new Ignite API. > >> The only requirement for classes - they should be available > >> locally(on client node or thin client host). > >> > >> 3. User deploy the service with some Ignite API. > >> > >> 4. After depoyment completes successfully client node can be > >> stopped. > >> All required resource to run a service should be safely stored= in > >> cluster and deployed to any new node. > >> > >> 5. I think we should develop examples for a DeploymentSPI usage. > >> As far as I can see, there is no such examples in our codebase for now= . > >> Is it correct? If so, I will create a ticket to create such examples. > >> > >> =D0=92 =D0=92=D1=82, 05/02/2019 =D0=B2 01:08 +0300, Vyacheslav Daradur= =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > >> > Denis, thank you for driving of Service Grid's development! > >> > > >> > Sounds like a good plan. Does it mean that a user will have to > >> > register a classloader for service's class explicitly in case of usi= ng > >> > the feature? > >> > > >> > On Mon, Feb 4, 2019 at 4:38 PM Denis Mekhanikov > >> wrote: > >> > > > >> > > Igniters, > >> > > > >> > > I'd like to start a dedicated thread for discussion of the design = of > >> > > services hot redeployment. The previous service design discussion = can > >> be > >> > > found in the following thread: [1] > >> > > > >> > > Currently adding a new service or implementation change of an > >> existing one > >> > > requires restarting the hosting nodes. Service instances are > >> deserialized > >> > > using an application class loader, so the service class should be > >> present > >> > > on the classpath of the node. The only way to change the set of > >> available > >> > > classes is to restart the node. Potentially the whole cluster rest= art > >> can > >> > > be required. This is a major drawback in the current design. This > >> problem > >> > > should be addressed first. > >> > > > >> > > At the same time this problem can be resolved by relatively simple > >> code > >> > > changes. We need to change the way services are deserialized, and = use > >> a > >> > > mechanism, that allows dynamic class changes. Deployment SPI [2] > >> seems to > >> > > be suitable for this. We can apply the same approach, which is use= d > >> for > >> > > tasks, so services will become dynamically modifiable. > >> > > > >> > > With this approach user will still need to perform a cancel-deploy > >> routine > >> > > for the changed service. But even with that the usability improvem= ent > >> will > >> > > be huge. We'll think about service availability improvement after = the > >> first > >> > > part is finished. > >> > > > >> > > Thoughts? > >> > > > >> > > [1] > >> > > > >> http://apache-ignite-developers.2346864.n4.nabble.com/Service-versioni= ng-td20858.html > >> > > [2] https://apacheignite.readme.io/docs/deployment-spi#deployments= pi > >> > > > >> > > Denis > >> > > >> > > >> > > >> > -- > >> > Best Regards, Vyacheslav D. > >> > > --=20 Best Regards, Vyacheslav D.