marvin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Lucredio <daniel.lucre...@ufscar.br>
Subject Re: Communication between CLI and Docker container in the new architecture.
Date Fri, 03 Apr 2020 17:45:30 GMT
If we are going for a server to maintain the daemon, instead of a simple
process just to keep the container alive, why not go for a gRPC solution?
In this way, both the engine's endpoints and the toolbox commands would be
accessible in the same way. And it would also support other languages for
the clients. What do you think?

Daniel Lucrédio

Em seg., 30 de mar. de 2020 às 14:38, Lucas Cardoso Silva <
cardosolucas61.lcs@gmail.com> escreveu:

> What do you think about using the Pyro4 library to maintain the daemon? The
> license is compatible and it deals with security issues. We can use the
> docker API to provision containers and interact at a high level while
> remote method calls are made through Pyro lib.
>
> Pyro4 github: https://github.com/irmen/Pyro4
>
> Best regards,
> Lucas Cardoso
>
> Em qui., 19 de mar. de 2020 às 16:24, Lucas Cardoso Silva <
> cardosolucas61.lcs@gmail.com> escreveu:
>
> > Great! I will do some tests with the "daemon" approach and report any
> > updates. Thanks!
> >
> > Em ter., 17 de mar. de 2020 às 17:40, Daniel Takabayashi <
> > daniel.takabayashi@gmail.com> escreveu:
> >
> >> No, I think everything is here. The "daemon" idea is the best way to
> >> handle
> >> this problem, basically, we need to have a Marvin agent running inside
> the
> >> container responsible to receive and run the commands from outside. But
> I
> >> didn't get the part related to Jupyter.
> >>
> >>
> >> Em sex., 13 de mar. de 2020 às 15:15, Lucas Cardoso Silva <
> >> cardosolucas61.lcs@gmail.com> escreveu:
> >>
> >> > I did some testing with the Docker API. The only problem is that a
> >> > container needs a process running to stay active, if it does not exist
> >> the
> >> > instance stop before receiving the command through the API. There is
> the
> >> > possibility to maintain an infinite and inexpensive process like: tail
> >> -f /
> >> > dev / null (not a good idea). Another possibility would be refactor
> the
> >> > part of the marvin that would remain in the container to become a
> >> daemon or
> >> > leave an instance of Jupyter running (the easy way). Is there any
> other
> >> > possibility that I'm not seeing?
> >> >
> >> > Em sex., 13 de mar. de 2020 às 15:46, Daniel Takabayashi <
> >> > daniel.takabayashi@gmail.com> escreveu:
> >> >
> >> > > Take a look at the Docker API
> >> https://docs.docker.com/engine/api/v1.24/,
> >> > I
> >> > > believe this will simplify the solution. Cause if you add another
> >> layer
> >> > > (Flask application) you going to need to control/manage this layer
> as
> >> > well,
> >> > > creating a chicken and egg problem.
> >> > >
> >> > > I believe the Docker API is a simple solution that solves the
> problem
> >> > > (communication between toolbox and containers).
> >> > >
> >> > > Take a look here
> >> > >
> >> > >
> >> >
> >>
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> >> > >
> >> > >
> >> > > Em sex., 13 de mar. de 2020 às 11:19, Lucas Cardoso Silva <
> >> > > cardosolucas61.lcs@gmail.com> escreveu:
> >> > >
> >> > > > I thought about using a flask application, which can become very
> >> fast
> >> > and
> >> > > > scalable with a Python WSGI server for production like Gunicorn.
> We
> >> can
> >> > > > make the communication establish by HTTPS protocol and use a
> public
> >> and
> >> > > > private key generation system for each engine generated. Another,
> >> > > althougt
> >> > > > complex, alternative would be use OAuth 2.0.
> >> > > >
> >> > > > Regards,
> >> > > > Lucas Cardoso
> >> > > >
> >> > > > Em sex., 13 de mar. de 2020 às 13:58, Daniel Takabayashi <
> >> > > > daniel.takabayashi@gmail.com> escreveu:
> >> > > >
> >> > > > > @Daniel The biggest challenge is to run commands in a remote
> >> > container
> >> > > > > without having access to the whole O.S. Basically the idea
is to
> >> > create
> >> > > > an
> >> > > > > interface to make possible, in a secure way the communication
> >> > > > > between Toolbox and remote engines.
> >> > > > >
> >> > > > > Creating this we could start to run our engines in clouds
and
> >> > services
> >> > > > like
> >> > > > > Google Run, Kubernetes, Lambda Functions and etc in the
same way
> >> > Marvin
> >> > > > > runs locally.
> >> > > > >
> >> > > > > @Lucas Lets talk about the details of these APIs (interfaces,
> >> > > technology)
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Taka
> >> > > > >
> >> > > > > Em sex., 13 de mar. de 2020 às 07:21, Daniel Lucredio <
> >> > > > > daniel.lucredio@ufscar.br> escreveu:
> >> > > > >
> >> > > > > > Hi Lucas and everyone,
> >> > > > > >
> >> > > > > > Couldn't the developer just run the CLI from inside
the
> >> container,
> >> > > > > opening
> >> > > > > > a shell inside it?
> >> > > > > >
> >> > > > > > []s
> >> > > > > >
> >> > > > > > Daniel
> >> > > > > >
> >> > > > > > Em qui., 5 de mar. de 2020 às 13:31, Lucas Cardoso
Silva <
> >> > > > > > cardosolucas61.lcs@gmail.com> escreveu:
> >> > > > > >
> >> > > > > > > I think we could define better how we will make
the
> >> communication
> >> > > CLI
> >> > > > > > with
> >> > > > > > > the Docker development instance in the new architecture.
The
> >> > > > container
> >> > > > > > > needs a running process to stay active. We could
make a
> >> > > communication
> >> > > > > via
> >> > > > > > > API endpoints to keep a web service running on
the container
> >> > > > receiving
> >> > > > > > > information from the CLI for the execution of
tasks in
> >> Marvin. I
> >> > > > > created
> >> > > > > > a
> >> > > > > > > use-case scenario to facilitate understanding
and further
> >> > > discussion.
> >> > > > > > >
> >> > > > > > > Scenario: A developer created an engine on marvin
through
> the
> >> > CLI,
> >> > > he
> >> > > > > > > configures his engine in order to describe all
the
> >> dependencies
> >> > of
> >> > > > the
> >> > > > > > > operating system that will be used during development.
After
> >> the
> >> > > > engine
> >> > > > > > > configuration process, the developer uses the
CLI to upload
> a
> >> > > > > development
> >> > > > > > > environment on a Docker container, that development
> >> environment
> >> > > will
> >> > > > > > > contain the refined Marvin toolbox. The CLI then
changes its
> >> > > > interface
> >> > > > > in
> >> > > > > > > order to contain the standard commands for using
engines. He
> >> uses
> >> > > the
> >> > > > > CLI
> >> > > > > > > to communicate with the toolbox endpoints and
inform that he
> >> > wants
> >> > > to
> >> > > > > run
> >> > > > > > > an instance of the notebook. After building the
model, the
> >> > > developer
> >> > > > > can,
> >> > > > > > > through the CLI, perform a dryrun and upload the
http-server
> >> for
> >> > > > > testing.
> >> > > > > > > All of these procedures will be done using the
communication
> >> > > through
> >> > > > > API
> >> > > > > > > endpoints.
> >> > > > > > >
> >> > > > > > > Does that sound like a good strategy? Do you have
any
> >> > suggestions,
> >> > > or
> >> > > > > > > something that was already foreseen in the original
project?
> >> > > > > > >
> >> > > > > > > Best regards,
> >> > > > > > > Lucas Cardoso
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message