incubator-bigtop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruno Mahé <bm...@apache.org>
Subject Re: Bigtop environment setup
Date Sun, 23 Sep 2012 22:41:28 GMT
Regarding the size, it depends on a lot of things (using raw vs qcow, 
how many packages installed...).
For instance the Apache Bigtop VMs take quite some space because they 
reserve some free space so people can upload their datasets. But for a 
build VM, not that much space is needed.

For reference, we could put Apache Bigtop distribution on a bootable USB 
key :)



On 09/23/2012 03:26 PM, Anatoli Fomenko wrote:
> While liking very much tools that Bruno brought up, I tend to think along with Cos' comments.
It looks like 2GB is not an unusual size of BoxGrinder based appliance, perhaps for specific
use cases. At this point, I would think of something more agile for Bigtop.
>
>
> Thanks,
> Anatoli
>
>
>
> ________________________________
>   From: Konstantin Boudnik <cos@apache.org>
> To: bigtop-dev@incubator.apache.org
> Sent: Sunday, September 23, 2012 12:06 AM
> Subject: Re: Bigtop environment setup
>
> On Fri, Sep 21, 2012 at 09:20PM, Bruno MahИ wrote:
>> On 09/19/2012 06:32 PM, Roman Shaposhnik wrote:
>>> On Wed, Sep 19, 2012 at 6:00 PM, Anatoli Fomenko <afomenko@yahoo.com> wrote:
>>>> I found that in order to avoid unnecessary build failures I need to quickly
set up additional
>>>> VMs for Bigtop supported platforms. From my experience with Precise, I would
say that it's a task that may take time.
>>>>
>>>> Any suggestions how it could be accelerated?
>>>
>>> You're raising a very good point, actually. In fact I've run into
>>> this very issue while trying to configure an extra Jenkins
>>> slave for bigtop01.
>>>
>>> Now, in the ideal world, all the build time dependencies
>>> that we have would be packaged and we'd express the
>>> fact that we depend on them via the very same packages
>>> that we're maintaining. That's what RPM's BuildRequires:
>>> and DEB's Build-Depends: fields are for -- to tell you
>>> explicitly what's required to be installed on the system
>>> before you can do the build of the package.
>>>
>>> Then you'd use the tools like:
>>>       apt-get build-dep
>>>       yum-builddep
>>> to satisfy all the dependencies and you're done.
>>>
>>> Now, this works great in the environment where
>>> you already have source packages which you
>>> can give to apt-get build-dep/yum-builddep
>>>
>>> But Bigtop has to be bootstrapped from the source.
>>> We can't assume existence of source packages.
>>>
>>> So here's the question to the bigger Bigtop
>>> community -- how do we want to proceed to
>>> manage repeatable build environments for
>>> our packages?
>>>
>>> The options I see are:
>>>      #1 maintain a parallel (very shallow) collection
>>>           of puppet code that would, essentially,
>>>           manage our "build slaves"
>>>      #2 do #1 but automate it in such a way that
>>>           the info actually gets harvested from
>>>           spec/conrol files
>>>
>>> Thoughts?
>>>
>>> Thanks,
>>> Roman.
>>>
>>
>> #1 is nice since it can deal with non-packaging issue. But it still
>> require people to install and know how to deal with puppet. From a
>> dev point of view we also need to remember to not use the latest
>> features since some OS lag significantly in term of versions of
>> puppet available.
>>
>> #2 is also nice since it can be dealt with the usual set of tools.
>> But it still requires some effort on users. Also some dependencies
>> are not and will probably never be available as packages (ex: Oracle
>> JDK).
>>
>> I also don't think there is one and only one solution.
>> My setup at home is quite different from the bigtop01 one.
>> And once you are familiar enough with Apache Bigtop and know how to
>> set it up, you may find options #1 and #2 probably not well adapted
>> to your situation.
>>
>> So this leads me to think about option #3: VMs.
>> Tools like Boxgrinder and Oz can deal with multiple OSes and can
>> create local images as well as push them to the cloud.
>> The build would be repeatable and would not require any effort from
>> the end user (apart maybe providing Oracle JDK, but that would have
>> to be the case whatever the solution). Future contributors would
>> just need to boot their VM to get started and hopefully ease
>> contribution.
>>
>> Thoughts?
>
> (writing at the end of the last post feels totally unnatural, but for the
> benefit of the future readers I will comply :)
>
> I think boxed environments (like VMs) are an overkill. One of the issues here
> (as Anatoli pointed out) is non-even support of all OSes. Say, BoxGrinder has
> some issues with Ubuntu, etc.
>
> I suppose a ol' proven toolchain type of environment that can automatically
> bootstrap upon a fresh install (or update itself: think Maven model) and pull
> whatever apps are required in whatever form they might exist. Say, JDK (or
> Groovy or else) can be downloaded as a tarball, given that a more suitable
> packaging isn't available, etc. Such approach would be more fluid than a
> somewhat rigid VMs, that would have to be updated periodically, versioned,
> etc.
>
> Another benefit of a toolchain is that BigTop packages might have to
> redistribute/wrap some of the tools for later use once a package is installed
> on a customer's system.
>
> So, in other words, #2 above (or some modification of it) looks more appealing to me.
>
> --
>    Take care,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>
> Disclaimer: Opinions expressed in this email are those of the author, and do
> not necessarily represent the views of any company the author might be
> affiliated with at the moment of writing.
>


Mime
View raw message