From dev-return-364-archive-asf-public=cust-asf.ponee.io@marvin.apache.org Fri Apr 3 17:45:50 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id D0E6D18064F for ; Fri, 3 Apr 2020 19:45:49 +0200 (CEST) Received: (qmail 44734 invoked by uid 500); 3 Apr 2020 17:45:49 -0000 Mailing-List: contact dev-help@marvin.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@marvin.apache.org Delivered-To: mailing list dev@marvin.apache.org Received: (qmail 44721 invoked by uid 99); 3 Apr 2020 17:45:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2020 17:45:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D0581C0884 for ; Fri, 3 Apr 2020 17:45:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.2 X-Spam-Level: X-Spam-Status: No, score=0.2 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=ufscar-br.20150623.gappssmtp.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id XgEZW3j7adn7 for ; Fri, 3 Apr 2020 17:45:44 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.160.171; helo=mail-qt1-f171.google.com; envelope-from=daniel.lucredio@ufscar.br; receiver= Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id B564CBB885 for ; Fri, 3 Apr 2020 17:45:43 +0000 (UTC) Received: by mail-qt1-f171.google.com with SMTP id f20so7172747qtq.6 for ; Fri, 03 Apr 2020 10:45:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ufscar-br.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=vOMoM5p1XYPshXwHW+b3dqfD5m370qsXn+UWsYshDJI=; b=GMbqGmXIum47yVsTpKyKt1/i+vavBwNEMHkxnCPt4/MFh7OAHkp3IYy6DZY9SLFZVj cqK6FTzb/uUChuu9fXKrsCHRalfcvHc04/gAVD+gaoIc+vAsGyxp6erQbDTxHi/z24Eg p+ZzpnXGuBhRQfrUzOOzr62488pdO7+VNQik9BzW5KEMDZFZSUZMEFp6N/oM1U1rWOSG 7GWMM4ZWOwIWp28WxstnN7J3UbUbOI45+AdM6l3LelVE0KBJCsC20rZ28VO43ePfCNkH x9kt90HLoUEuBUTZgoVCVyBEi6S0JUfTbYH+nKMDRRHei/JBrf/yM5Yat6OLBQDVOYNm Oiwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=vOMoM5p1XYPshXwHW+b3dqfD5m370qsXn+UWsYshDJI=; b=XdXK3GGvAc+crLHdexM6EzWLpmyKgEt1JQyBUZULpEPxNP0rrPaElHl8NWbNCgY3Zu ljvB2X7iEWS9E+wwhsBygYMf0VMzrpUqBj8cEmw0W4qDQufjqiMPQrr1YAcQiMhWaGko KtBRa/YbPXYh/z4hby3+3URLNO34DpGT246OnMPXewSmXP/ymP07wIXQnJzuKvWG4o4S e+SLQs0P/2FIIbjCyOwPKoRBACRgdqHuWhk8zeGR2oEepAYu5aOqUJepAo6IGd+5zfBb p+RNhCjMp8Egfd5c0t++2RrMcc3EgKjJ2DWsjxZyntfyBLePK4GBJAZEtBhlmgEnLgmV d3sw== X-Gm-Message-State: AGi0PuZjQplPpekAdUdxGqDMjTiqLHokEzy8NGShtBQgNQkUNUUic5wH sf4LjmeHsTmWQ2dohP7zd+OS7sO0fz0QNWYtGUDI/rp/ X-Google-Smtp-Source: APiQypLTkzjF5nLsjVg68xML+IZzVovSUsIPRhxyP9lEuS5q9f6QkCKIMpR90JWAhSiWmav+mqZcU/KT4QZww/N4qow= X-Received: by 2002:ac8:7448:: with SMTP id h8mr9450064qtr.51.1585935942310; Fri, 03 Apr 2020 10:45:42 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Daniel Lucredio Date: Fri, 3 Apr 2020 14:45:30 -0300 Message-ID: Subject: Re: Communication between CLI and Docker container in the new architecture. To: dev@marvin.apache.org Content-Type: multipart/alternative; boundary="0000000000007a79b705a2667c76" --0000000000007a79b705a2667c76 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable If we are going for a server to maintain the daemon, instead of a simple process just to keep the container alive, why not go for a gRPC solution? In this way, both the engine's endpoints and the toolbox commands would be accessible in the same way. And it would also support other languages for the clients. What do you think? Daniel Lucr=C3=A9dio Em seg., 30 de mar. de 2020 =C3=A0s 14:38, Lucas Cardoso Silva < cardosolucas61.lcs@gmail.com> escreveu: > What do you think about using the Pyro4 library to maintain the daemon? T= he > license is compatible and it deals with security issues. We can use the > docker API to provision containers and interact at a high level while > remote method calls are made through Pyro lib. > > Pyro4 github: https://github.com/irmen/Pyro4 > > Best regards, > Lucas Cardoso > > Em qui., 19 de mar. de 2020 =C3=A0s 16:24, Lucas Cardoso Silva < > cardosolucas61.lcs@gmail.com> escreveu: > > > Great! I will do some tests with the "daemon" approach and report any > > updates. Thanks! > > > > Em ter., 17 de mar. de 2020 =C3=A0s 17:40, Daniel Takabayashi < > > daniel.takabayashi@gmail.com> escreveu: > > > >> No, I think everything is here. The "daemon" idea is the best way to > >> handle > >> this problem, basically, we need to have a Marvin agent running inside > the > >> container responsible to receive and run the commands from outside. Bu= t > I > >> didn't get the part related to Jupyter. > >> > >> > >> Em sex., 13 de mar. de 2020 =C3=A0s 15:15, Lucas Cardoso Silva < > >> cardosolucas61.lcs@gmail.com> escreveu: > >> > >> > I did some testing with the Docker API. The only problem is that a > >> > container needs a process running to stay active, if it does not exi= st > >> the > >> > instance stop before receiving the command through the API. There is > the > >> > possibility to maintain an infinite and inexpensive process like: ta= il > >> -f / > >> > dev / null (not a good idea). Another possibility would be refactor > the > >> > part of the marvin that would remain in the container to become a > >> daemon or > >> > leave an instance of Jupyter running (the easy way). Is there any > other > >> > possibility that I'm not seeing? > >> > > >> > Em sex., 13 de mar. de 2020 =C3=A0s 15:46, Daniel Takabayashi < > >> > daniel.takabayashi@gmail.com> escreveu: > >> > > >> > > Take a look at the Docker API > >> https://docs.docker.com/engine/api/v1.24/, > >> > I > >> > > believe this will simplify the solution. Cause if you add another > >> layer > >> > > (Flask application) you going to need to control/manage this layer > as > >> > well, > >> > > creating a chicken and egg problem. > >> > > > >> > > I believe the Docker API is a simple solution that solves the > problem > >> > > (communication between toolbox and containers). > >> > > > >> > > Take a look here > >> > > > >> > > > >> > > >> > https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=3D602&= h=3D461&rev=3D1423&ac=3D1&parent=3D1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF= 3deY > >> > > > >> > > > >> > > Em sex., 13 de mar. de 2020 =C3=A0s 11:19, Lucas Cardoso Silva < > >> > > cardosolucas61.lcs@gmail.com> escreveu: > >> > > > >> > > > I thought about using a flask application, which can become very > >> fast > >> > and > >> > > > scalable with a Python WSGI server for production like Gunicorn. > We > >> can > >> > > > make the communication establish by HTTPS protocol and use a > public > >> and > >> > > > private key generation system for each engine generated. Another= , > >> > > althougt > >> > > > complex, alternative would be use OAuth 2.0. > >> > > > > >> > > > Regards, > >> > > > Lucas Cardoso > >> > > > > >> > > > Em sex., 13 de mar. de 2020 =C3=A0s 13:58, Daniel Takabayashi < > >> > > > daniel.takabayashi@gmail.com> escreveu: > >> > > > > >> > > > > @Daniel The biggest challenge is to run commands in a remote > >> > container > >> > > > > without having access to the whole O.S. Basically the idea is = to > >> > create > >> > > > an > >> > > > > interface to make possible, in a secure way the communication > >> > > > > between Toolbox and remote engines. > >> > > > > > >> > > > > Creating this we could start to run our engines in clouds and > >> > services > >> > > > like > >> > > > > Google Run, Kubernetes, Lambda Functions and etc in the same w= ay > >> > Marvin > >> > > > > runs locally. > >> > > > > > >> > > > > @Lucas Lets talk about the details of these APIs (interfaces, > >> > > technology) > >> > > > > > >> > > > > Thanks, > >> > > > > Taka > >> > > > > > >> > > > > Em sex., 13 de mar. de 2020 =C3=A0s 07:21, Daniel Lucredio < > >> > > > > daniel.lucredio@ufscar.br> escreveu: > >> > > > > > >> > > > > > Hi Lucas and everyone, > >> > > > > > > >> > > > > > Couldn't the developer just run the CLI from inside the > >> container, > >> > > > > opening > >> > > > > > a shell inside it? > >> > > > > > > >> > > > > > []s > >> > > > > > > >> > > > > > Daniel > >> > > > > > > >> > > > > > Em qui., 5 de mar. de 2020 =C3=A0s 13:31, Lucas Cardoso Silv= a < > >> > > > > > cardosolucas61.lcs@gmail.com> escreveu: > >> > > > > > > >> > > > > > > I think we could define better how we will make the > >> communication > >> > > CLI > >> > > > > > with > >> > > > > > > the Docker development instance in the new architecture. T= he > >> > > > container > >> > > > > > > needs a running process to stay active. We could make a > >> > > communication > >> > > > > via > >> > > > > > > API endpoints to keep a web service running on the contain= er > >> > > > receiving > >> > > > > > > information from the CLI for the execution of tasks in > >> Marvin. I > >> > > > > created > >> > > > > > a > >> > > > > > > use-case scenario to facilitate understanding and further > >> > > discussion. > >> > > > > > > > >> > > > > > > Scenario: A developer created an engine on marvin through > the > >> > CLI, > >> > > he > >> > > > > > > configures his engine in order to describe all the > >> dependencies > >> > of > >> > > > the > >> > > > > > > operating system that will be used during development. Aft= er > >> the > >> > > > engine > >> > > > > > > configuration process, the developer uses the CLI to uploa= d > a > >> > > > > development > >> > > > > > > environment on a Docker container, that development > >> environment > >> > > will > >> > > > > > > contain the refined Marvin toolbox. The CLI then changes i= ts > >> > > > interface > >> > > > > in > >> > > > > > > order to contain the standard commands for using engines. = He > >> uses > >> > > the > >> > > > > CLI > >> > > > > > > to communicate with the toolbox endpoints and inform that = he > >> > wants > >> > > to > >> > > > > run > >> > > > > > > an instance of the notebook. After building the model, the > >> > > developer > >> > > > > can, > >> > > > > > > through the CLI, perform a dryrun and upload the http-serv= er > >> for > >> > > > > testing. > >> > > > > > > All of these procedures will be done using the communicati= on > >> > > through > >> > > > > API > >> > > > > > > endpoints. > >> > > > > > > > >> > > > > > > Does that sound like a good strategy? Do you have any > >> > suggestions, > >> > > or > >> > > > > > > something that was already foreseen in the original projec= t? > >> > > > > > > > >> > > > > > > Best regards, > >> > > > > > > Lucas Cardoso > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > --0000000000007a79b705a2667c76--