airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shameera Rathnayaka <shameerai...@gmail.com>
Subject Re: Airavata Docker Available !!
Date Mon, 06 Jul 2015 18:35:08 GMT
Hi Suresh,

Shall we create an docker module like distribution and put all dockers to
that module as Lahiru suggested?

Hi Lahiru/Suresh,
This is how i am thinking this docker releases works, we will have snapshot
dockerfiles which based on ubuntu latest and have all tools(git , maven ,
etc) installed on it, we download from git and build airavata source and
start airavata server. Once we come to release point, let's say 16 release
(next release from master, here I assume 0.15 has been released with it's
own docker files) then with release process we will release(publish)
official docker images(based on Ubuntu 14.04 or suitable) and docker files
which don't have git , maven , etc...installation commands and download
released airavata distribution from one of Appache mirror server. We will
maintain snapshot version of dockerfile with Airavata source code.  WDYT?

Hi Pankaj,

You may have confused with docker maven plugin which we are talking above,
that is only for building purpose. you don't need to worry about this. I
have few comments about your docker files let me reply it separately.

Thanks,
Shameera.

On Mon, Jul 6, 2015 at 2:14 PM Shameera Rathnayaka <shameerainfo@gmail.com>
wrote:

> Hi Lahiru,
>
> I am ok to use any official docker images, but I couldn't find any official
> java 8 image released by Oracle. There are set of official images for open
> jdk but not for oracle jdk. In that case I would like to suggest to use
> above steps to install java8. It is clean and no any unofficial
> dependencies. I am not much familiar how docker world works, this is my 2
> cents.
>
> Thanks,
> Shameera.
>
> On Mon, Jul 6, 2015 at 1:43 PM Lahiru Ginnaliya Gamathige <
> glahiru@gmail.com> wrote:
>
>> I have use sbt docker because we use sbt instead of maven to build java
>> code. I think we can use this[1] for maven.
>>
>> [1]https://github.com/rhuss/docker-maven-plugin
>>
>> I think its worth exploring this since we already have a maven build.
>>
>> Lahiru
>>
>> On Mon, Jul 6, 2015 at 10:39 AM, Suresh Marru <smarru@apache.org> wrote:
>>
>>> Lahiru,
>>>
>>> Did you come across any maven plugins like assembly ones so we can
>>> bundle all docker things with the main build? There seems to be few github
>>> projects but could not weigh in which one is good.
>>>
>>> Suresh
>>>
>>>
>>> On Mon, Jul 6, 2015 at 1:26 PM Lahiru Ginnaliya Gamathige <
>>> glahiru@gmail.com> wrote:
>>>
>>>> If we use docker for deployment purpose there is no requirement for
>>>> bundling source code in to docker image. We usually bundle minimum amount
>>>> of data (everything required in runtime) but nothing other than that.
>>>> During development we use a script which build the docker image for us
>>>> using exiting source in our local file system. Once we do a release we have
>>>> docker images with proper naming which will not touch.
>>>>
>>>> Shameera,
>>>>
>>>> I am not sure why we need to install java or maven without using
>>>> existing docker image provided by Oracle.
>>>>
>>>> Lahiru
>>>>
>>>> On Mon, Jul 6, 2015 at 9:52 AM, Suresh Marru <smarru@apache.org> wrote:
>>>>
>>>>> Hi Pankaj,
>>>>>
>>>>> Thinking naively, I am considering docker images same as maven
>>>>> artifacts. Users will consume the published images. We will need to publish
>>>>> released docker images and also nightly builds.
>>>>>
>>>>> From developers perspective, I am guessing they will again work
>>>>> similar to maven. We do not want to push to a docker hub, but create
and
>>>>> use the image locally.
>>>>>
>>>>> Let me defer to Lahiru to update us how docker is used in real-world
>>>>> by developers. I am guessing as long as we can build docker images with
the
>>>>> maven build (through ant commands probably), it should probably help
both
>>>>> the cases.
>>>>>
>>>>> Suresh
>>>>>
>>>>>
>>>>> On Mon, Jul 6, 2015 at 12:25 PM Pankaj Saha <psaha4@binghamton.edu>
>>>>> wrote:
>>>>>
>>>>>> Hi Suresh
>>>>>>
>>>>>> Our initial plan was to make docker for the airavata developers so
>>>>>> that the initial hurdle of environment set up can be reduced.
>>>>>> If we don't include the source code and only keep the executable
>>>>>> binaries then how can we fulfill that goal?
>>>>>> do we need two set of docker images 1. with source code (dev edition)
>>>>>>  and 2. without source code (end user edition) ?
>>>>>> please suggest.
>>>>>>
>>>>>> Thanks
>>>>>> Pankaj
>>>>>>
>>>>>>
>>>>>> On Mon, Jul 6, 2015 at 12:10 PM, Pankaj Saha <psaha4@binghamton.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey Shameera,
>>>>>>> what about the PATH variable? If you see my dockerfile I have
>>>>>>> commented the ENV for the JAVA_HOME and PATH. It did not work
for me.
>>>>>>> Lahiru suggested to use existing jdk8+maven base image which
is
>>>>>>> better idea , what do you think?
>>>>>>>
>>>>>>> On Mon, Jul 6, 2015 at 12:03 PM, Shameera Rathnayaka <
>>>>>>> shameerainfo@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Pankaj,
>>>>>>>>
>>>>>>>> you can use following steps on your Dockerfile to install
java.
>>>>>>>> After that java will be available anywhere and you don't
need to set PATH
>>>>>>>> variable , But you need to set JAVA_HOME variable which is
used by airavata
>>>>>>>> start script.
>>>>>>>>
>>>>>>>>
>>>>>>>> # Install prerequisites
>>>>>>>> RUN apt-get update
>>>>>>>> RUN apt-get install -y software-properties-common
>>>>>>>>
>>>>>>>> # Install java8
>>>>>>>> RUN add-apt-repository -y ppa:webupd8team/java
>>>>>>>> RUN apt-get update
>>>>>>>> RUN echo oracle-java8-installer shared/accepted-oracle-license-v1-1
>>>>>>>> select true | sudo /usr/bin/debconf-set-selections
>>>>>>>> RUN apt-get install -y oracle-java8-installer
>>>>>>>>
>>>>>>>> # set JAVA_HOME
>>>>>>>> ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Shameera.
>>>>>>>>
>>>>>>>> On Mon, Jul 6, 2015 at 11:56 AM Pankaj Saha <psaha4@binghamton.edu>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Lahiru,
>>>>>>>>>
>>>>>>>>> 1. So you suggest to use the "java8 with maven" as the
base image.
>>>>>>>>> dockerfile/java:oracle-java8 is not based on ubuntu image
and also
>>>>>>>>> its not official image from oracle, can we use still
use that?
>>>>>>>>>
>>>>>>>>> 2. Do you suggest to put only the executables for airavata
inside
>>>>>>>>> the docker image? Does that mean docker file will copy
the local host
>>>>>>>>> machine's airavata executable files to the docker image
through ADD
>>>>>>>>> directive?
>>>>>>>>> So I will build the airavata source code locally through
maven and
>>>>>>>>> then put that inside the container image.
>>>>>>>>>
>>>>>>>>> 3. One thing that I could not able to do was changing
the
>>>>>>>>> properties values for different port and server ips when
the containers
>>>>>>>>> starts. I was trying to run a startup shell script which
can do the
>>>>>>>>> required changes and then kick start the airavat-server.sh
file.
>>>>>>>>> problem was  airavata-server.properties need apiserver.server.host
>>>>>>>>> = current running containers ip address. Can you suggest
some better way to
>>>>>>>>> achieve this through shell script.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Pankaj
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jul 6, 2015 at 11:30 AM, Lahiru Ginnaliya Gamathige
<
>>>>>>>>> glahiru@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Pankaj,
>>>>>>>>>>
>>>>>>>>>> I had a quick look. I think you can replace all the
code to
>>>>>>>>>> install java and maven if you use this image [1]
or at least java-8 image.
>>>>>>>>>>
>>>>>>>>>> I think its not necessary to build airavata inside
the
>>>>>>>>>> dockerfile, you can just assume its built in host
machine and you just have
>>>>>>>>>> to put required executables in to the docker image.
>>>>>>>>>> When you create he image maven and source code it
not necessary
>>>>>>>>>> at all. Docker image should only contain required
binaries, source code
>>>>>>>>>> doesn't have to in the image.
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://registry.hub.docker.com/u/jamesdbloom/docker-java8-maven/dockerfile/
>>>>>>>>>>
>>>>>>>>>> On Mon, Jul 6, 2015 at 8:13 AM, Pankaj Saha <
>>>>>>>>>> psaha4@binghamton.edu> wrote:
>>>>>>>>>>
>>>>>>>>>>> I am so sorry , I have no intention to hide any
docker code.I am
>>>>>>>>>>> attaching the docker files here.I have never
done git check in and trying
>>>>>>>>>>> to upload files and its not working. I am trying
to understand the git pull
>>>>>>>>>>> request/setting up git directory etc.
>>>>>>>>>>>
>>>>>>>>>>> My apology if I unnecessary delayed it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jul 6, 2015 at 10:51 AM, Suresh Marru
<smarru@apache.org
>>>>>>>>>>> > wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Pankaj,
>>>>>>>>>>>>
>>>>>>>>>>>> Open source software can never be final.
Once the software is
>>>>>>>>>>>> final, the community is dead, because there
is nothing for anyone to do. An
>>>>>>>>>>>> important aspect for all the GSoC students
to embrace is, these are not
>>>>>>>>>>>> academic projects where you submit the final
code. You are supposed to
>>>>>>>>>>>> commit every few hours of code, you need
to do the development in open. We
>>>>>>>>>>>> have been lenient on this so far, but you
have been holding off the docker
>>>>>>>>>>>> work little too long. Please do commit the
docker files in as in condition.
>>>>>>>>>>>> Thats the only way you can engage others,
otherwise you will loose everyone
>>>>>>>>>>>> interest.
>>>>>>>>>>>>
>>>>>>>>>>>> Suresh
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jul 6, 2015 at 10:42 AM, Lahiru Ginnaliya
Gamathige <
>>>>>>>>>>>> glahiru@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Not sure what is the big deal here, Why
don't you create a
>>>>>>>>>>>>> pull request and every one can review
it. It doesn't have to be finalized
>>>>>>>>>>>>> at all. People will comment and you can
follow the comments and after
>>>>>>>>>>>>> multiple reviews we can  move ti to the
the code base.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I propose to create a folder struction
like this and put the
>>>>>>>>>>>>> docker files.
>>>>>>>>>>>>>
>>>>>>>>>>>>> airavata/deploy/thrift, airavata/deploy/rabbitmq,
>>>>>>>>>>>>> airavata/deploy/zookeeper. Create a directory
for every image you build
>>>>>>>>>>>>> with proper naming and add the Dockerfile
and other scripts if there's any.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Lahiru
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jul 6, 2015 at 7:03 AM, Pankaj
Saha <
>>>>>>>>>>>>> psaha4@binghamton.edu> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> This docker file is not final yet
as we have to automate the
>>>>>>>>>>>>>> environment variable to replace the
properties file.
>>>>>>>>>>>>>> I believe in a hangout session we
can walk through and fix
>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jul 6, 2015 at 9:58 AM, Shameera
Rathnayaka <
>>>>>>>>>>>>>> shameerainfo@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Pankaj,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Let't do this via Apache way,
upload these files to publicly
>>>>>>>>>>>>>>> accessible place (as i mentioned
in my previous reply) then invite everyone
>>>>>>>>>>>>>>> to review it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Shameera.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jul 6, 2015 at 9:44 AM
Pankaj Saha <
>>>>>>>>>>>>>>> psaha4@binghamton.edu> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Shameera,
>>>>>>>>>>>>>>>> Please find the attached
Dockerfile for Airavata Server.
>>>>>>>>>>>>>>>> Please suggest changes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jul 2, 2015 at 5:50
PM, Shameera Rathnayaka <
>>>>>>>>>>>>>>>> shameerainfo@gmail.com>
wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Pankaj,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes please share your
works with community. Do you have
>>>>>>>>>>>>>>>>> specific location to
commit your works? If not you can use this location
>>>>>>>>>>>>>>>>> https://github.com/apache/airavata-sandbox.
 Let's try to
>>>>>>>>>>>>>>>>>  track the changes with
VCS.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> @Suresh do we have specific
location for this time gsoc
>>>>>>>>>>>>>>>>> students?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Shameera.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Jul 2, 2015 at
5:24 PM Pankaj Saha <
>>>>>>>>>>>>>>>>> psaha4@binghamton.edu>
wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Shameera
>>>>>>>>>>>>>>>>>> Yes we can see the
containers ip they way you suggested.
>>>>>>>>>>>>>>>>>> What I mentioned
is another way in case you are already inside the
>>>>>>>>>>>>>>>>>> container.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regarding java path
variable yes we can do that with ENV
>>>>>>>>>>>>>>>>>> in docker file. It
was throwing some errors so i just commented that in the
>>>>>>>>>>>>>>>>>> docker file.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I will send you the
docker file in another email then we
>>>>>>>>>>>>>>>>>> can go through together
 to improve it.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I was trying mesos
today and didnt have time to fix that.
>>>>>>>>>>>>>>>>>> We can do it later
as we are already running late for the mesos
>>>>>>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Pankaj
>>>>>>>>>>>>>>>>>>  On 02-Jul-2015 5:12
PM, "Shameera Rathnayaka" <
>>>>>>>>>>>>>>>>>> shameerainfo@gmail.com>
wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Pankaj,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> See my comments
inline.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> docker attach
server
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> awk 'NR==1
{print $1}' /etc/hosts    ---- note the ip
>>>>>>>>>>>>>>>>>>>>> address
of the running container
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> can't we use
"docker inspect --format '{{
>>>>>>>>>>>>>>>>>>> .NetworkSettings.IPAddress
}}' server " to get airavata server docker
>>>>>>>>>>>>>>>>>>> container ip
?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> pico airavata-server.properties
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Replaces
the below text as suggested below
>>>>>>>>>>>>>>>>>>>> apiserver.server.host=
*<current containers ip
>>>>>>>>>>>>>>>>>>>> address>*
>>>>>>>>>>>>>>>>>>>> default.registry.gateway=*default*
>>>>>>>>>>>>>>>>>>>> rabbitmq.broker.url=amqp://*<rabbit
MQ's ip address>*
>>>>>>>>>>>>>>>>>>>> :5672
>>>>>>>>>>>>>>>>>>>> save changes.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> pico /etc/bash.bashrc
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> add below
lines at the end
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> #JAVA_HOME
>>>>>>>>>>>>>>>>>>>>> JAVA_HOME=/opt/jdk/jdk1.8.0_05
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> You can improve
you docker file to add this variables
>>>>>>>>>>>>>>>>>>> using ENV command.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> export JAVA_HOME
>>>>>>>>>>>>>>>>>>>>> PATH=$JAVA_HOME/bin:$PATH
>>>>>>>>>>>>>>>>>>>>> export
PATH
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> save changes.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> How you install
java 8?, There is a way to install java
>>>>>>>>>>>>>>>>>>> on docker which
automatically available in PATH variable. You don't have to
>>>>>>>>>>>>>>>>>>> set it here.
It is good if you can show us the Dockerfiles then we can
>>>>>>>>>>>>>>>>>>> review those
files. is it available online ?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Shameera.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>

Mime
View raw message