thrift-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thibault Piana (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (THRIFT-4802) Automatic library deployment on online CDN/Repository services
Date Wed, 13 Feb 2019 20:11:00 GMT

     [ https://issues.apache.org/jira/browse/THRIFT-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Thibault Piana updated THRIFT-4802:
-----------------------------------
    Description: 
Hello everyone,

This thread follows some other threads dealing with concerns about online libraries (THRIFT-4688,
THRIFT-4687), as well as emails being sent to the user mailing list about the Docker container
on DockerHub.

If I understand the situation correctly, at the moment, with each new version of the software,
you must prepare and update each library on each website independently.
In addition, each of those libraries is managed by different people, with whom contact is
not always easy.

Also, I would like to propose here a system that would offer the following advantages:
 - No more worrying about who is maintaining each platform/CDN/Repositories...
 - No more manual uploading on each platform at each release
 - Allow anyone to develop an extension to put a Thrift package online on a particular site
(For example, a version for Yocto as I requested in a recent email), and be sure this package
will be updated at each new release.

I will try to explain my idea in the following.
h2. Vocabulary
 * *CDN :* Content delivery network
 * *Manager :* An algorithm to upload content to a CDN/online repository
 * *Orchestrator :* The algorithm that orchestrates the execution of managers, provides credentials,
manages manager feedback, and generates the final report
 * *Package* : The library/component we want to upload on the CDN

In the following, all online resources providers/repositories (like Pypi, NPMJs, DockerHub
...) will be called "CDN"
h2. General principle

The idea would be to develop a single interface for module deployment, with an orchestrator
to automatically configure and perform these deployments. 
The organization would be like this :
 * A Keepass database will store all passwords/identifiers required for each platform (NPMJS,
Pypi, DockerHub, ...) :
 ** Each module will specify what credentials it needs from this database.
 ** This database could be committed directly on the repo, to be sure each committers have
the most recent version.
 ** The master password of this database will only be known by committers (or project chair).

I think KeePass is quite perfect for this, Keepass databases are very lightweight, highly
secure, and almost impossible to brute-force.
 * For each CDN, a manager is in charge of bundling (create the package to upload from Thrift
current release sources), uploading and checking the package on the CDN :
 ** First, the orchestrator load required IDs from the KeePass database
 ** The orchestrator creates a temporary directory where the manager will be able to bundle
everything it needs.
 *** The manager receives credentials, a path to a temporary directory and a path to Thrift
sources
 *** The manager checks online if the current version of the package already exists online.
If not, it will upload it
 *** The manager prepares the bundled package, upload it and check if everything went well.

All these steps could be executed in a dedicated Docker/Vagrant environment containing all
needed libraries to bundle the package (So that the person/commiter executing the script does
not have to download 500 dependencies).

The algorithm diagram is provided in attached (it's a draft)

The directory structure of this solution could be something like this (I chose Python arbitrarily
here):
{code:java}
root
└── deployment/
    ├── access.kdbx
    ├── deploy.py
    ├── libraries # Some managers
    │   ├── javascript
    │   │   ├── npm.py
    │   │   ├── bower.py
    │   │   └── a_random_cdn.py
    │   ├── nodejs
    │   │   └── npm.py
    │   └── python
    │   └── pypi.js
    │   └── ...
    └── miscellaneous # Some other managers
        ├── docker
        │   └── docker.py
        └── yocto
        └── yocto.py
        └── ...{code}
I am not a regular contributor on open source projects. So I don't know if I'm making my proposal
in the best way. But I really like this project and I would like to help improve it

I had this idea based on what had been put in place in a project I was recently leading. It
is not perfect but does the job properly, and it's a solution I find quite elegant because
we don't have to worry about the credentials of each site.

If you find this idea interesting, I could propose a prototype in a personal repository with
some examples of managers for Npm, Pypi, and DockerHub (for example) in several weeks.

  was:
Hello everyone,

This thread follows some other threads dealing with concerns about online libraries (THRIFT-4688,
THRIFT-4687), as well as emails being sent to the user mailing list about the Docker container
on DockerHub.

If I understand the situation correctly, at the moment, with each new version of the software,
you must prepare and update each library on each website independently.
In addition, each of those libraries is managed by different people, with whom contact is
not always easy.

Also, I would like to propose here a system that would offer the following advantages:
 - No more worrying about who is maintaining each platform
 - No more manual uploading on each platform at each release
 - Allow anyone to develop an extension to put a Thrift package online on a particular site
(For example, a version for Yocto as I requested in a recent email), and be sure this package
will be updated at each new release.

I will try to explain my idea in the following.
h2. Vocabulary
 * *CDN :* Content delivery network
 * *Manager :* An algorithm to upload content to a CDN/online repository
 * *Orchestrator :* The algorithm that orchestrates the execution of managers, provides credentials,
manages manager feedback, and generates the final report
 * *Package* : The library/component we want to upload on the CDN

In the following, all online resources providers/repositories (like Pypi, NPMJs, DockerHub
...) will be called "CDN"
h2. General principle

The idea would be to develop a single interface for module deployment, with an orchestrator
to automatically configure and perform these deployments. 
The organization would be like this :
 * A Keepass database will store all passwords/identifiers required for each platform (NPMJS,
Pypi, DockerHub, ...) :
 ** Each module will specify what credentials it needs from this database.
 ** This database could be committed directly on the repo, to be sure each committers have
the most recent version.
 ** The master password of this database will only be known by committers (or project chair).

I think KeePass is quite perfect for this, Keepass databases are very lightweight, highly
secure, and almost impossible to brute-force.
 * For each CDN, a manager is in charge of bundling (create the package to upload from Thrift
current release sources), uploading and checking the package on the CDN :
 ** First, the orchestrator load required IDs from the KeePass database
 ** The orchestrator creates a temporary directory where the manager will be able to bundle
everything it needs.
 *** The manager receives credentials, a path to a temporary directory and a path to Thrift
sources
 *** The manager checks online if the current version of the package already exists online.
If not, it will upload it
 *** The manager prepares the bundled package, upload it and check if everything went well.

All these steps could be executed in a dedicated Docker/Vagrant environment containing all
needed libraries to bundle the package (So that the person/commiter executing the script does
not have to download 500 dependencies).

The algorithm diagram is provided in attached (it's a draft)

The directory structure of this solution could be something like this (I chose Python arbitrarily
here):
{code:java}
root
└── deployment/
    ├── access.kdbx
    ├── deploy.py
    ├── libraries # Some managers
    │   ├── javascript
    │   │   ├── npm.py
    │   │   ├── bower.py
    │   │   └── a_random_cdn.py
    │   ├── nodejs
    │   │   └── npm.py
    │   └── python
    │   └── pypi.js
    │   └── ...
    └── miscellaneous # Some other managers
        ├── docker
        │   └── docker.py
        └── yocto
        └── yocto.py
        └── ...{code}
I am not a regular contributor on open source projects. So I don't know if I'm making my proposal
in the best way. But I really like this project and I would like to help improve it

I had this idea based on what had been put in place in a project I was recently leading. It
is not perfect but does the job properly, and it's a solution I find quite elegant because
we don't have to worry about the credentials of each site.

If you find this idea interesting, I could propose a prototype in a personal repository with
some examples of managers for Npm, Pypi, and DockerHub (for example) in several weeks.


> Automatic library deployment on online CDN/Repository services
> --------------------------------------------------------------
>
>                 Key: THRIFT-4802
>                 URL: https://issues.apache.org/jira/browse/THRIFT-4802
>             Project: Thrift
>          Issue Type: Improvement
>          Components: Deployment
>    Affects Versions: 0.12.0
>            Reporter: Thibault Piana
>            Priority: Critical
>              Labels: cdn, deploy,deployment, library
>         Attachments: Workflow.png
>
>
> Hello everyone,
> This thread follows some other threads dealing with concerns about online libraries (THRIFT-4688,
THRIFT-4687), as well as emails being sent to the user mailing list about the Docker container
on DockerHub.
> If I understand the situation correctly, at the moment, with each new version of the
software, you must prepare and update each library on each website independently.
> In addition, each of those libraries is managed by different people, with whom contact
is not always easy.
> Also, I would like to propose here a system that would offer the following advantages:
>  - No more worrying about who is maintaining each platform/CDN/Repositories...
>  - No more manual uploading on each platform at each release
>  - Allow anyone to develop an extension to put a Thrift package online on a particular
site (For example, a version for Yocto as I requested in a recent email), and be sure this
package will be updated at each new release.
> I will try to explain my idea in the following.
> h2. Vocabulary
>  * *CDN :* Content delivery network
>  * *Manager :* An algorithm to upload content to a CDN/online repository
>  * *Orchestrator :* The algorithm that orchestrates the execution of managers, provides
credentials, manages manager feedback, and generates the final report
>  * *Package* : The library/component we want to upload on the CDN
> In the following, all online resources providers/repositories (like Pypi, NPMJs, DockerHub
...) will be called "CDN"
> h2. General principle
> The idea would be to develop a single interface for module deployment, with an orchestrator
to automatically configure and perform these deployments. 
> The organization would be like this :
>  * A Keepass database will store all passwords/identifiers required for each platform
(NPMJS, Pypi, DockerHub, ...) :
>  ** Each module will specify what credentials it needs from this database.
>  ** This database could be committed directly on the repo, to be sure each committers
have the most recent version.
>  ** The master password of this database will only be known by committers (or project
chair).
> I think KeePass is quite perfect for this, Keepass databases are very lightweight, highly
secure, and almost impossible to brute-force.
>  * For each CDN, a manager is in charge of bundling (create the package to upload from
Thrift current release sources), uploading and checking the package on the CDN :
>  ** First, the orchestrator load required IDs from the KeePass database
>  ** The orchestrator creates a temporary directory where the manager will be able to
bundle everything it needs.
>  *** The manager receives credentials, a path to a temporary directory and a path to
Thrift sources
>  *** The manager checks online if the current version of the package already exists online.
If not, it will upload it
>  *** The manager prepares the bundled package, upload it and check if everything went
well.
> All these steps could be executed in a dedicated Docker/Vagrant environment containing
all needed libraries to bundle the package (So that the person/commiter executing the script
does not have to download 500 dependencies).
> The algorithm diagram is provided in attached (it's a draft)
> The directory structure of this solution could be something like this (I chose Python
arbitrarily here):
> {code:java}
> root
> └── deployment/
>     ├── access.kdbx
>     ├── deploy.py
>     ├── libraries # Some managers
>     │   ├── javascript
>     │   │   ├── npm.py
>     │   │   ├── bower.py
>     │   │   └── a_random_cdn.py
>     │   ├── nodejs
>     │   │   └── npm.py
>     │   └── python
>     │   └── pypi.js
>     │   └── ...
>     └── miscellaneous # Some other managers
>         ├── docker
>         │   └── docker.py
>         └── yocto
>         └── yocto.py
>         └── ...{code}
> I am not a regular contributor on open source projects. So I don't know if I'm making
my proposal in the best way. But I really like this project and I would like to help improve
it
> I had this idea based on what had been put in place in a project I was recently leading.
It is not perfect but does the job properly, and it's a solution I find quite elegant because
we don't have to worry about the credentials of each site.
> If you find this idea interesting, I could propose a prototype in a personal repository
with some examples of managers for Npm, Pypi, and DockerHub (for example) in several weeks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message