airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lahiru Ginnaliya Gamathige <glah...@gmail.com>
Subject Re: Airavata Docker Available !!
Date Tue, 07 Jul 2015 06:09:23 GMT
I am not sure about this approach. I would keep a separate script which
 compile our code and create a docker image and keep a clean docker image,
Not git, No source, No maven, only .jar files and required configuration
files. Or use a maven plugin to create the binary for us.


On Mon, Jul 6, 2015 at 11:35 AM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:

> Hi Suresh,
>
> Shall we create an docker module like distribution and put all dockers to
> that module as Lahiru suggested?
>
> Hi Lahiru/Suresh,
> This is how i am thinking this docker releases works, we will have
> snapshot dockerfiles which based on ubuntu latest and have all tools(git ,
> maven , etc) installed on it, we download from git and build airavata
> source and start airavata server. Once we come to release point, let's say
> 16 release (next release from master, here I assume 0.15 has been released
> with it's own docker files) then with release process we will
> release(publish) official docker images(based on Ubuntu 14.04 or suitable)
> and docker files which don't have git , maven , etc...installation commands
> and download released airavata distribution from one of Appache mirror
> server. We will maintain snapshot version of dockerfile with Airavata
> source code.  WDYT?
>
> Hi Pankaj,
>
> You may have confused with docker maven plugin which we are talking above,
> that is only for building purpose. you don't need to worry about this. I
> have few comments about your docker files let me reply it separately.
>
> Thanks,
> Shameera.
>
> On Mon, Jul 6, 2015 at 2:14 PM Shameera Rathnayaka <shameerainfo@gmail.com>
> wrote:
>
>> Hi Lahiru,
>>
>> I am ok to use any official docker images, but I couldn't find any official
>> java 8 image released by Oracle. There are set of official images for open
>> jdk but not for oracle jdk. In that case I would like to suggest to use
>> above steps to install java8. It is clean and no any unofficial
>> dependencies. I am not much familiar how docker world works, this is my 2
>> cents.
>>
>> Thanks,
>> Shameera.
>>
>> On Mon, Jul 6, 2015 at 1:43 PM Lahiru Ginnaliya Gamathige <
>> glahiru@gmail.com> wrote:
>>
>>> I have use sbt docker because we use sbt instead of maven to build java
>>> code. I think we can use this[1] for maven.
>>>
>>> [1]https://github.com/rhuss/docker-maven-plugin
>>>
>>> I think its worth exploring this since we already have a maven build.
>>>
>>> Lahiru
>>>
>>> On Mon, Jul 6, 2015 at 10:39 AM, Suresh Marru <smarru@apache.org> wrote:
>>>
>>>> Lahiru,
>>>>
>>>> Did you come across any maven plugins like assembly ones so we can
>>>> bundle all docker things with the main build? There seems to be few github
>>>> projects but could not weigh in which one is good.
>>>>
>>>> Suresh
>>>>
>>>>
>>>> On Mon, Jul 6, 2015 at 1:26 PM Lahiru Ginnaliya Gamathige <
>>>> glahiru@gmail.com> wrote:
>>>>
>>>>> If we use docker for deployment purpose there is no requirement for
>>>>> bundling source code in to docker image. We usually bundle minimum amount
>>>>> of data (everything required in runtime) but nothing other than that.
>>>>> During development we use a script which build the docker image for us
>>>>> using exiting source in our local file system. Once we do a release we
have
>>>>> docker images with proper naming which will not touch.
>>>>>
>>>>> Shameera,
>>>>>
>>>>> I am not sure why we need to install java or maven without using
>>>>> existing docker image provided by Oracle.
>>>>>
>>>>> Lahiru
>>>>>
>>>>> On Mon, Jul 6, 2015 at 9:52 AM, Suresh Marru <smarru@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Pankaj,
>>>>>>
>>>>>> Thinking naively, I am considering docker images same as maven
>>>>>> artifacts. Users will consume the published images. We will need
to publish
>>>>>> released docker images and also nightly builds.
>>>>>>
>>>>>> From developers perspective, I am guessing they will again work
>>>>>> similar to maven. We do not want to push to a docker hub, but create
and
>>>>>> use the image locally.
>>>>>>
>>>>>> Let me defer to Lahiru to update us how docker is used in real-world
>>>>>> by developers. I am guessing as long as we can build docker images
with the
>>>>>> maven build (through ant commands probably), it should probably help
both
>>>>>> the cases.
>>>>>>
>>>>>> Suresh
>>>>>>
>>>>>>
>>>>>> On Mon, Jul 6, 2015 at 12:25 PM Pankaj Saha <psaha4@binghamton.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Suresh
>>>>>>>
>>>>>>> Our initial plan was to make docker for the airavata developers
so
>>>>>>> that the initial hurdle of environment set up can be reduced.
>>>>>>> If we don't include the source code and only keep the executable
>>>>>>> binaries then how can we fulfill that goal?
>>>>>>> do we need two set of docker images 1. with source code (dev
>>>>>>> edition)  and 2. without source code (end user edition) ?
>>>>>>> please suggest.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Pankaj
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jul 6, 2015 at 12:10 PM, Pankaj Saha <psaha4@binghamton.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey Shameera,
>>>>>>>> what about the PATH variable? If you see my dockerfile I
have
>>>>>>>> commented the ENV for the JAVA_HOME and PATH. It did not
work for me.
>>>>>>>> Lahiru suggested to use existing jdk8+maven base image which
is
>>>>>>>> better idea , what do you think?
>>>>>>>>
>>>>>>>> On Mon, Jul 6, 2015 at 12:03 PM, Shameera Rathnayaka <
>>>>>>>> shameerainfo@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Pankaj,
>>>>>>>>>
>>>>>>>>> you can use following steps on your Dockerfile to install
java.
>>>>>>>>> After that java will be available anywhere and you don't
need to set PATH
>>>>>>>>> variable , But you need to set JAVA_HOME variable which
is used by airavata
>>>>>>>>> start script.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # Install prerequisites
>>>>>>>>> RUN apt-get update
>>>>>>>>> RUN apt-get install -y software-properties-common
>>>>>>>>>
>>>>>>>>> # Install java8
>>>>>>>>> RUN add-apt-repository -y ppa:webupd8team/java
>>>>>>>>> RUN apt-get update
>>>>>>>>> RUN echo oracle-java8-installer
>>>>>>>>> shared/accepted-oracle-license-v1-1 select true | sudo
>>>>>>>>> /usr/bin/debconf-set-selections
>>>>>>>>> RUN apt-get install -y oracle-java8-installer
>>>>>>>>>
>>>>>>>>> # set JAVA_HOME
>>>>>>>>> ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Shameera.
>>>>>>>>>
>>>>>>>>> On Mon, Jul 6, 2015 at 11:56 AM Pankaj Saha <psaha4@binghamton.edu>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Lahiru,
>>>>>>>>>>
>>>>>>>>>> 1. So you suggest to use the "java8 with maven" as
the base
>>>>>>>>>> image.
>>>>>>>>>> dockerfile/java:oracle-java8 is not based on ubuntu
image and
>>>>>>>>>> also its not official image from oracle, can we use
still use that?
>>>>>>>>>>
>>>>>>>>>> 2. Do you suggest to put only the executables for
airavata inside
>>>>>>>>>> the docker image? Does that mean docker file will
copy the local host
>>>>>>>>>> machine's airavata executable files to the docker
image through ADD
>>>>>>>>>> directive?
>>>>>>>>>> So I will build the airavata source code locally
through maven
>>>>>>>>>> and then put that inside the container image.
>>>>>>>>>>
>>>>>>>>>> 3. One thing that I could not able to do was changing
the
>>>>>>>>>> properties values for different port and server ips
when the containers
>>>>>>>>>> starts. I was trying to run a startup shell script
which can do the
>>>>>>>>>> required changes and then kick start the airavat-server.sh
file.
>>>>>>>>>> problem was  airavata-server.properties need apiserver.server.host
>>>>>>>>>> = current running containers ip address. Can you
suggest some better way to
>>>>>>>>>> achieve this through shell script.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Pankaj
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Jul 6, 2015 at 11:30 AM, Lahiru Ginnaliya
Gamathige <
>>>>>>>>>> glahiru@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Pankaj,
>>>>>>>>>>>
>>>>>>>>>>> I had a quick look. I think you can replace all
the code to
>>>>>>>>>>> install java and maven if you use this image
[1] or at least java-8 image.
>>>>>>>>>>>
>>>>>>>>>>> I think its not necessary to build airavata inside
the
>>>>>>>>>>> dockerfile, you can just assume its built in
host machine and you just have
>>>>>>>>>>> to put required executables in to the docker
image.
>>>>>>>>>>> When you create he image maven and source code
it not necessary
>>>>>>>>>>> at all. Docker image should only contain required
binaries, source code
>>>>>>>>>>> doesn't have to in the image.
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://registry.hub.docker.com/u/jamesdbloom/docker-java8-maven/dockerfile/
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jul 6, 2015 at 8:13 AM, Pankaj Saha <
>>>>>>>>>>> psaha4@binghamton.edu> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I am so sorry , I have no intention to hide
any docker code.I
>>>>>>>>>>>> am attaching the docker files here.I have
never done git check in and
>>>>>>>>>>>> trying to upload files and its not working.
I am trying to understand the
>>>>>>>>>>>> git pull request/setting up git directory
etc.
>>>>>>>>>>>>
>>>>>>>>>>>> My apology if I unnecessary delayed it.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jul 6, 2015 at 10:51 AM, Suresh Marru
<
>>>>>>>>>>>> smarru@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Pankaj,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Open source software can never be final.
Once the software is
>>>>>>>>>>>>> final, the community is dead, because
there is nothing for anyone to do. An
>>>>>>>>>>>>> important aspect for all the GSoC students
to embrace is, these are not
>>>>>>>>>>>>> academic projects where you submit the
final code. You are supposed to
>>>>>>>>>>>>> commit every few hours of code, you need
to do the development in open. We
>>>>>>>>>>>>> have been lenient on this so far, but
you have been holding off the docker
>>>>>>>>>>>>> work little too long. Please do commit
the docker files in as in condition.
>>>>>>>>>>>>> Thats the only way you can engage others,
otherwise you will loose everyone
>>>>>>>>>>>>> interest.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Suresh
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jul 6, 2015 at 10:42 AM, Lahiru
Ginnaliya Gamathige <
>>>>>>>>>>>>> glahiru@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Not sure what is the big deal here,
Why don't you create a
>>>>>>>>>>>>>> pull request and every one can review
it. It doesn't have to be finalized
>>>>>>>>>>>>>> at all. People will comment and you
can follow the comments and after
>>>>>>>>>>>>>> multiple reviews we can  move ti
to the the code base.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I propose to create a folder struction
like this and put the
>>>>>>>>>>>>>> docker files.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> airavata/deploy/thrift, airavata/deploy/rabbitmq,
>>>>>>>>>>>>>> airavata/deploy/zookeeper. Create
a directory for every image you build
>>>>>>>>>>>>>> with proper naming and add the Dockerfile
and other scripts if there's any.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Lahiru
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jul 6, 2015 at 7:03 AM, Pankaj
Saha <
>>>>>>>>>>>>>> psaha4@binghamton.edu> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This docker file is not final
yet as we have to automate the
>>>>>>>>>>>>>>> environment variable to replace
the properties file.
>>>>>>>>>>>>>>> I believe in a hangout session
we can walk through and fix
>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jul 6, 2015 at 9:58 AM,
Shameera Rathnayaka <
>>>>>>>>>>>>>>> shameerainfo@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Pankaj,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Let't do this via Apache
way, upload these files to
>>>>>>>>>>>>>>>> publicly accessible place
(as i mentioned in my previous reply) then invite
>>>>>>>>>>>>>>>> everyone to review it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Shameera.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Jul 6, 2015 at 9:44
AM Pankaj Saha <
>>>>>>>>>>>>>>>> psaha4@binghamton.edu>
wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Shameera,
>>>>>>>>>>>>>>>>> Please find the attached
Dockerfile for Airavata Server.
>>>>>>>>>>>>>>>>> Please suggest changes.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Jul 2, 2015 at
5:50 PM, Shameera Rathnayaka <
>>>>>>>>>>>>>>>>> shameerainfo@gmail.com>
wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Pankaj,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yes please share
your works with community. Do you have
>>>>>>>>>>>>>>>>>> specific location
to commit your works? If not you can use this location
>>>>>>>>>>>>>>>>>> https://github.com/apache/airavata-sandbox.
 Let's try
>>>>>>>>>>>>>>>>>> to  track the changes
with VCS.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> @Suresh do we have
specific location for this time gsoc
>>>>>>>>>>>>>>>>>> students?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Shameera.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Thu, Jul 2, 2015
at 5:24 PM Pankaj Saha <
>>>>>>>>>>>>>>>>>> psaha4@binghamton.edu>
wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Shameera
>>>>>>>>>>>>>>>>>>> Yes we can see
the containers ip they way you suggested.
>>>>>>>>>>>>>>>>>>> What I mentioned
is another way in case you are already inside the
>>>>>>>>>>>>>>>>>>> container.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regarding java
path variable yes we can do that with ENV
>>>>>>>>>>>>>>>>>>> in docker file.
It was throwing some errors so i just commented that in the
>>>>>>>>>>>>>>>>>>> docker file.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I will send you
the docker file in another email then we
>>>>>>>>>>>>>>>>>>> can go through
together  to improve it.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I was trying
mesos today and didnt have time to fix
>>>>>>>>>>>>>>>>>>> that. We can
do it later as we are already running late for the mesos
>>>>>>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Pankaj
>>>>>>>>>>>>>>>>>>>  On 02-Jul-2015
5:12 PM, "Shameera Rathnayaka" <
>>>>>>>>>>>>>>>>>>> shameerainfo@gmail.com>
wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Pankaj,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> See my comments
inline.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> docker
attach server
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> awk 'NR==1
{print $1}' /etc/hosts    ---- note the ip
>>>>>>>>>>>>>>>>>>>>>> address
of the running container
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> can't we
use "docker inspect --format '{{
>>>>>>>>>>>>>>>>>>>> .NetworkSettings.IPAddress
}}' server " to get airavata server docker
>>>>>>>>>>>>>>>>>>>> container
ip ?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> pico
airavata-server.properties
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Replaces
the below text as suggested below
>>>>>>>>>>>>>>>>>>>>> apiserver.server.host=
*<current containers ip
>>>>>>>>>>>>>>>>>>>>> address>*
>>>>>>>>>>>>>>>>>>>>> default.registry.gateway=*default*
>>>>>>>>>>>>>>>>>>>>> rabbitmq.broker.url=amqp://*<rabbit
MQ's ip address>*
>>>>>>>>>>>>>>>>>>>>> :5672
>>>>>>>>>>>>>>>>>>>>> save
changes.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> pico
/etc/bash.bashrc
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> add below
lines at the end
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> #JAVA_HOME
>>>>>>>>>>>>>>>>>>>>>> JAVA_HOME=/opt/jdk/jdk1.8.0_05
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> You can improve
you docker file to add this variables
>>>>>>>>>>>>>>>>>>>> using ENV
command.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> export
JAVA_HOME
>>>>>>>>>>>>>>>>>>>>>> PATH=$JAVA_HOME/bin:$PATH
>>>>>>>>>>>>>>>>>>>>>> export
PATH
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> save
changes.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> How you install
java 8?, There is a way to install java
>>>>>>>>>>>>>>>>>>>> on docker
which automatically available in PATH variable. You don't have to
>>>>>>>>>>>>>>>>>>>> set it here.
It is good if you can show us the Dockerfiles then we can
>>>>>>>>>>>>>>>>>>>> review those
files. is it available online ?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Shameera.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>

Mime
View raw message