marvin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Takabayashi <daniel.takabaya...@gmail.com>
Subject Re: Communication between CLI and Docker container in the new architecture.
Date Fri, 24 Apr 2020 19:31:10 GMT
+1

Em sex., 17 de abr. de 2020 às 11:15, Bruno Sette <brunosilvasette@gmail.com>
escreveu:

> Standardizing with gRPC seems better to me. It is a good agnostic and safe
> language solution.
>
> Em sex., 17 de abr. de 2020 às 14:56, Lucas Cardoso Silva <
> cardosolucas61.lcs@gmail.com> escreveu:
>
> > Standardize with gRPC seems like a better idea. It has all the security
> > features we need and maintains the code's agnosticism. What do the rest
> of
> > you guys think?
> >
> > Lucas Cardoso
> >
> > Em sex., 3 de abr. de 2020 às 14:45, Daniel Lucredio <
> > daniel.lucredio@ufscar.br> escreveu:
> >
> > > If we are going for a server to maintain the daemon, instead of a
> simple
> > > process just to keep the container alive, why not go for a gRPC
> solution?
> > > In this way, both the engine's endpoints and the toolbox commands would
> > be
> > > accessible in the same way. And it would also support other languages
> for
> > > the clients. What do you think?
> > >
> > > Daniel Lucrédio
> > >
> > > Em seg., 30 de mar. de 2020 às 14:38, Lucas Cardoso Silva <
> > > cardosolucas61.lcs@gmail.com> escreveu:
> > >
> > > > What do you think about using the Pyro4 library to maintain the
> daemon?
> > > The
> > > > license is compatible and it deals with security issues. We can use
> the
> > > > docker API to provision containers and interact at a high level while
> > > > remote method calls are made through Pyro lib.
> > > >
> > > > Pyro4 github: https://github.com/irmen/Pyro4
> > > >
> > > > Best regards,
> > > > Lucas Cardoso
> > > >
> > > > Em qui., 19 de mar. de 2020 às 16:24, Lucas Cardoso Silva <
> > > > cardosolucas61.lcs@gmail.com> escreveu:
> > > >
> > > > > Great! I will do some tests with the "daemon" approach and report
> any
> > > > > updates. Thanks!
> > > > >
> > > > > Em ter., 17 de mar. de 2020 às 17:40, Daniel Takabayashi <
> > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > >
> > > > >> No, I think everything is here. The "daemon" idea is the best
way
> to
> > > > >> handle
> > > > >> this problem, basically, we need to have a Marvin agent running
> > inside
> > > > the
> > > > >> container responsible to receive and run the commands from
> outside.
> > > But
> > > > I
> > > > >> didn't get the part related to Jupyter.
> > > > >>
> > > > >>
> > > > >> Em sex., 13 de mar. de 2020 às 15:15, Lucas Cardoso Silva <
> > > > >> cardosolucas61.lcs@gmail.com> escreveu:
> > > > >>
> > > > >> > I did some testing with the Docker API. The only problem
is
> that a
> > > > >> > container needs a process running to stay active, if it
does not
> > > exist
> > > > >> the
> > > > >> > instance stop before receiving the command through the API.
> There
> > is
> > > > the
> > > > >> > possibility to maintain an infinite and inexpensive process
> like:
> > > tail
> > > > >> -f /
> > > > >> > dev / null (not a good idea). Another possibility would
be
> > refactor
> > > > the
> > > > >> > part of the marvin that would remain in the container to
become
> a
> > > > >> daemon or
> > > > >> > leave an instance of Jupyter running (the easy way). Is
there
> any
> > > > other
> > > > >> > possibility that I'm not seeing?
> > > > >> >
> > > > >> > Em sex., 13 de mar. de 2020 às 15:46, Daniel Takabayashi
<
> > > > >> > daniel.takabayashi@gmail.com> escreveu:
> > > > >> >
> > > > >> > > Take a look at the Docker API
> > > > >> https://docs.docker.com/engine/api/v1.24/,
> > > > >> > I
> > > > >> > > believe this will simplify the solution. Cause if you
add
> > another
> > > > >> layer
> > > > >> > > (Flask application) you going to need to control/manage
this
> > layer
> > > > as
> > > > >> > well,
> > > > >> > > creating a chicken and egg problem.
> > > > >> > >
> > > > >> > > I believe the Docker API is a simple solution that
solves the
> > > > problem
> > > > >> > > (communication between toolbox and containers).
> > > > >> > >
> > > > >> > > Take a look here
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > >> > >
> > > > >> > >
> > > > >> > > Em sex., 13 de mar. de 2020 às 11:19, Lucas Cardoso
Silva <
> > > > >> > > cardosolucas61.lcs@gmail.com> escreveu:
> > > > >> > >
> > > > >> > > > I thought about using a flask application, which
can become
> > very
> > > > >> fast
> > > > >> > and
> > > > >> > > > scalable with a Python WSGI server for production
like
> > Gunicorn.
> > > > We
> > > > >> can
> > > > >> > > > make the communication establish by HTTPS protocol
and use a
> > > > public
> > > > >> and
> > > > >> > > > private key generation system for each engine
generated.
> > > Another,
> > > > >> > > althougt
> > > > >> > > > complex, alternative would be use OAuth 2.0.
> > > > >> > > >
> > > > >> > > > Regards,
> > > > >> > > > Lucas Cardoso
> > > > >> > > >
> > > > >> > > > Em sex., 13 de mar. de 2020 às 13:58, Daniel
Takabayashi <
> > > > >> > > > daniel.takabayashi@gmail.com> escreveu:
> > > > >> > > >
> > > > >> > > > > @Daniel The biggest challenge is to run commands
in a
> remote
> > > > >> > container
> > > > >> > > > > without having access to the whole O.S. Basically
the idea
> > is
> > > to
> > > > >> > create
> > > > >> > > > an
> > > > >> > > > > interface to make possible, in a secure way
the
> > communication
> > > > >> > > > > between Toolbox and remote engines.
> > > > >> > > > >
> > > > >> > > > > Creating this we could start to run our engines
in clouds
> > and
> > > > >> > services
> > > > >> > > > like
> > > > >> > > > > Google Run, Kubernetes, Lambda Functions
and etc in the
> same
> > > way
> > > > >> > Marvin
> > > > >> > > > > runs locally.
> > > > >> > > > >
> > > > >> > > > > @Lucas Lets talk about the details of these
APIs
> > (interfaces,
> > > > >> > > technology)
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Taka
> > > > >> > > > >
> > > > >> > > > > Em sex., 13 de mar. de 2020 às 07:21, Daniel
Lucredio <
> > > > >> > > > > daniel.lucredio@ufscar.br> escreveu:
> > > > >> > > > >
> > > > >> > > > > > Hi Lucas and everyone,
> > > > >> > > > > >
> > > > >> > > > > > Couldn't the developer just run the
CLI from inside the
> > > > >> container,
> > > > >> > > > > opening
> > > > >> > > > > > a shell inside it?
> > > > >> > > > > >
> > > > >> > > > > > []s
> > > > >> > > > > >
> > > > >> > > > > > Daniel
> > > > >> > > > > >
> > > > >> > > > > > Em qui., 5 de mar. de 2020 às 13:31,
Lucas Cardoso
> Silva <
> > > > >> > > > > > cardosolucas61.lcs@gmail.com> escreveu:
> > > > >> > > > > >
> > > > >> > > > > > > I think we could define better
how we will make the
> > > > >> communication
> > > > >> > > CLI
> > > > >> > > > > > with
> > > > >> > > > > > > the Docker development instance
in the new
> architecture.
> > > The
> > > > >> > > > container
> > > > >> > > > > > > needs a running process to stay
active. We could make
> a
> > > > >> > > communication
> > > > >> > > > > via
> > > > >> > > > > > > API endpoints to keep a web service
running on the
> > > container
> > > > >> > > > receiving
> > > > >> > > > > > > information from the CLI for the
execution of tasks in
> > > > >> Marvin. I
> > > > >> > > > > created
> > > > >> > > > > > a
> > > > >> > > > > > > use-case scenario to facilitate
understanding and
> > further
> > > > >> > > discussion.
> > > > >> > > > > > >
> > > > >> > > > > > > Scenario: A developer created an
engine on marvin
> > through
> > > > the
> > > > >> > CLI,
> > > > >> > > he
> > > > >> > > > > > > configures his engine in order
to describe all the
> > > > >> dependencies
> > > > >> > of
> > > > >> > > > the
> > > > >> > > > > > > operating system that will be used
during development.
> > > After
> > > > >> the
> > > > >> > > > engine
> > > > >> > > > > > > configuration process, the developer
uses the CLI to
> > > upload
> > > > a
> > > > >> > > > > development
> > > > >> > > > > > > environment on a Docker container,
that development
> > > > >> environment
> > > > >> > > will
> > > > >> > > > > > > contain the refined Marvin toolbox.
The CLI then
> changes
> > > its
> > > > >> > > > interface
> > > > >> > > > > in
> > > > >> > > > > > > order to contain the standard commands
for using
> > engines.
> > > He
> > > > >> uses
> > > > >> > > the
> > > > >> > > > > CLI
> > > > >> > > > > > > to communicate with the toolbox
endpoints and inform
> > that
> > > he
> > > > >> > wants
> > > > >> > > to
> > > > >> > > > > run
> > > > >> > > > > > > an instance of the notebook. After
building the model,
> > the
> > > > >> > > developer
> > > > >> > > > > can,
> > > > >> > > > > > > through the CLI, perform a dryrun
and upload the
> > > http-server
> > > > >> for
> > > > >> > > > > testing.
> > > > >> > > > > > > All of these procedures will be
done using the
> > > communication
> > > > >> > > through
> > > > >> > > > > API
> > > > >> > > > > > > endpoints.
> > > > >> > > > > > >
> > > > >> > > > > > > Does that sound like a good strategy?
Do you have any
> > > > >> > suggestions,
> > > > >> > > or
> > > > >> > > > > > > something that was already foreseen
in the original
> > > project?
> > > > >> > > > > > >
> > > > >> > > > > > > Best regards,
> > > > >> > > > > > > Lucas Cardoso
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
>
> --
> Att.,
>
> *Bruno Silva Sette*
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message