harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrey Chernyshev" <a.y.chernys...@gmail.com>
Subject Re: Supporting working on a single module?
Date Fri, 12 May 2006 16:55:58 GMT
On 5/12/06, Oliver Deakin <oliver.deakin@googlemail.com> wrote:
> Geir Magnusson Jr wrote:
> >
> >
> > Oliver Deakin wrote:
> >> Geir Magnusson Jr wrote:
> >>> Mark Hindess wrote:
> >>>> On 9 May 2006 at 10:32, Geir Magnusson Jr <geir@pobox.com> wrote:
> >>>>> Mark Hindess wrote:
> >>>>>> As the Harmony Classlib grows, I think that being able to work
on a
> >>>>>> single module (or some subset of the modules) will become the
> >>>>>> typical way of working for many (perhaps even most) contributors.
> >>>>> Sure, that makes sense.
> >>>>>
> >>>>>> So I think we need a plan to support this.  I also think that
> >>>>>> forcing ourselves to support this mode of working sooner rather
than
> >>>>>> later will help us keep from accidentally breaking the modularity
in
> >>>>>> the current build/development process.
> >>>>> Can you not work on a single module now?
> >>>>
> >>>> Yes.  But only by checking out everything.
> >>>
> >>> Wouldn't you do that anyway?
> >>>
> >>> I guess thats the mystery to me.  I figured that someone would do
> >>> the full checkout, and go from there.
> >>
> >> I think it's good if we can cater not only for those developers who
> >> want to completely checkout classlib trunk, but also for those who
> >> want to work only on a single module. When I say "work on a single
> >> module", I mean only checking that module's directory out of SVN, and
> >> not the code for all the other modules, thereby saving download time
> >> and disk space, not to mention the subsequent reduced build times of
> >> only recompiling a single module.
> >
> > Eh. :)  We have agreement in a root issue, but not as you describe.
> > IOW, I agree that it should be that after a full build, you should be
> > able to dive into a module and work and re-build in there in isolation.
> >
> > However, I have little sympathy for the reduced download time or
> > diskspace argument.
> >
> > I *do* sympathize with [psychos like] Mark who has 4 builds at once,
> > and see that it would be efficient to have multiple independent 'dev
> > subtrees' pointing at one or more pre-built trees.
>
> It seems we have agreement that being able to rebuild a single module in
> isolation is a good idea - we have different ideas about how that may
> then be used, but I think both are valid and possible once the proper
> modularisation of code and build scripts exists. I guess once the build
> system is in place, it's up to the developer how they wish to use it :)
>
> >
> >>
> >> Currently a developer cannot work on both the native and Java code
> >> for a module without checking out the whole of classlib trunk,
> >> because the native source directories and Ant build script associated
> >> with them are completely separate from the modules. IMHO it would be
> >> great if we could get the build scripts and code repository layout
> >> into a state where a developer can just checkout a module, grab an
> >> hdk from some location and start changing and rebuilding both Java
> >> and native code (and also running tests against the rebuilt libraries).
> >
> > This is mixing two issues.
> >
> > yes, I think we should put the native code into each module, so you
> > can just work in one module, both native and Java.
> >
> > Yes, I see a reason why you would want to have a pre-build snapshot to
> > use for multiple dev subtrees, for "known good" regression testing, etc.
> >
> > But I have trouble with linking them.
>
> ok, fair enough - I have mixed these two issues together. I think this
> stems from what I said above - we just have different ideas about how
> the modularisation will be used once it's in place. When I think of
> modularisation, I imagine a developer grabbing the source just for the
> module they are interested in and a snapshot build (hdk) and working
> away with it - thus I see the modularisation and the creation of an hdk
> as being linked. As you say, really they are two separate issues.
>
> >>
> >> To do this there are at least three steps needed, as far as I can see:
> >>
> >> 1) Refactor the native code into the modular structure we currently
> >> have for Java code and tests. This has been spoken about before at
> >> [1]. The native code would be located within each module at
> >> modules/<module name>/src/main/native. As a starting point, can I
> >> propose that the natives be broken down in the following way:
> >>
> >> modules/archive/src/main/native/
> >>                                        |-------archive/
> >>                                        |-------zip/
> >>
> >> +------zlib/                          modules/auth/src/main/native/
> >>                                        +-------auth/
> >>
> >> modules/luni/src/main/native/
> >>                                        |--------common/
> >>                                        |--------fdlibm/
> >>                                        |--------launcher/
> >>                                        |--------luni/
> >>                                        |--------pool/
> >>                                        |--------port/
> >>                                        |--------sig/
> >>                                        |--------thread/
> >>                                        |--------vmi/
> >>                                        +-------vmls/
> >>
> >> modules/prefs/src/main/native/
> >>                                       +-------prefs/
> >>
> >> modules/text/src/main/native/
> >>                                        |-------text/
> >>                                        +------unicode/ (comes from
> >> the icu4c zips stored currently in depends)
> >
> > W/o thinking too hard about it, this works for me just fine.
>
> Great - I am starting to look at how shared includes can be handled
> across modules (as Mark alluded to in his earlier post in this thread
> [1]), and at what changes will be required to split the natives into
> these locations. I will be taking this in small steps, trying to get the
> foundation and "easy" parts done first, and raising a JIRA for each step
> rather than in one monolithic change.

Great!  I think splitting the sources by modules at the top level
directory is a good idea.
A few questions before the big source tree reorganization starts:

> >> modules/archive/src/main/native/
> >>                                        |-------archive/
> >>                                        |-------zip/

(1) Do we need to keep the 'main' directory in every module? If we
need to have a distinction between tests and sources, may be just pull
tests one level up and have something like:
archive/
        src/
             native/
             java/
         tests/
             native/
             java/
I wonder if 'main' is an extra level of the directory tree we can
actually avoid of (lazy people don't like typing cd too much :))

(2) Why do we need to have 'luni' two times in the path, e.g.
modules/luni/src/main/native/luni/ ? If we need to put an additional
stuff like 'port' to the luni module, perhaps it could be just enough
to put it into a subdirectory within native, e.g:
modules/luni/src/native/port/ ?


BTW, I've noticed that this proposal is very similar to the DRLVM
source tree organization, which is like:
- vm
    - include  - top level include which contains h files exported by
various VM components;
    - interpreter
    - jitrino
    - vmcore
    ...
    <other VM components>

The module vmcore, for example, contains both native and java code:
vmcore/src/kernel_classes
       - native
       - javasrc

The building system for DRLVM has been designed in a modular way as well:
There is a "building engine part at the build/make and
build/make/targets directory which is shared by all components,
Each VM module has a building descriptor which is currently located at
build/make/components directory, but can also be easily moved to the
component source tree to support the idea of full independent checkout
of a specific module.

I think the building system provided with DRLVM can easily be used to
support the source modularization approach, the proposed 'hdk' in that
case would provide the developers, besides the "public includes", with
the shared part of the building scripts as well.


Thank you,
Andrey Chernyshev
Intel Middleware Products Division

>
> >
> >>
> >> Additionally, each module may have an include directory that contains
> >> header files required across its native components. Each native
> >> component would contain subdirectories for its shared and platform
> >> specific code.
> >
> > Ok, as far as I undertsand it.
> >
> >>
> >> 2) Alter the global build scripts to start creating an hdk under the
> >> deploy directory, which contains a jdk which contains a jre. As Mark
> >> described before, this hdk would probably look something like:
> >>
> >> deploy/hdk/
> >>               |--------jdk/
> >>               |              |-------jre/    (already being created
> >> at the moment as deploy/jre)
> >>               |              +------include/    (already being
> >> created at the moment as deploy/include)
> >>               |--------include/
> >>               +-------lib/
> >>
> >> The hdk/include directory would be much like the current
> >> native-src/<platform>/include directories, and would contain a
> >> minimal set of header files that define an interface between native
> >> modules. It would be the responsibility of the individual module to
> >> copy its required headers to this directory as a prerequisite for
> >> building.
> >> The hdk/lib directory would exist on Windows platforms, and would
> >> contain .lib files for each native component.
> >>
> >> An hdk snapshot would then be a zip of the hdk directory, and should
> >> provide everything necessary for a developer to build both native and
> >> Java code against.
> >
> > Mmmmm.  I'm struggling w/ having the /jdk part of it.
>
> What is it you don't like about /jdk?
>
> >
> >>
> >> 3) Modularise native and Java build scripts so that the individual
> >> module's Ant scripts are capable of building native and Java source
> >> against an hdk, rather than having its code compiled as part of a
> >> global build.
> >
> > 1) Re modularize build scripts, Yes!  I thought it was this way
> > already but that it wasn't used, because that's how we added security2
> > to the build - by just invoking from the master.  IMO, that's how it
> > all should have been done.
> >
> > 2) re against an HDK, fine, but never assume it's location.  Make sure
> > that I can put it anywhere on a filesystem and it works.  That way you
> > can have multiple dev trees pointing to it, or let me easily "A/B"
> > test by just changing a property somewhere.
> >
> >> The rebuilt libraries would be placed back into the hdk directory, so
> >> that the developers local hdk always contains their latest built code
> >> (much as the current deploy/jre does).
> >
> > I think this is a bad idea.
> >
> > First, I think we want to support an HDK being used by multiple dev
> > trees, so you don't want those trees dropping new untested into it.
> >
> > Second, think about it - do you really want to be modifying a
> > known-good snapshot with code you are debugging?  How do you "reset"?
> >
> > I'd keep the HDK separate from "deploy".
> >
> > Given each module knows it's artifacts, it should be able to
> > dynamically construct the CPs and library paths as it needs to for
> > in-module testing, which is the point of this exercise...?
> >
> > geir
> Does it matter if we overlay newly built libraries onto a snapshot?
> Isn't "resetting" as simple as unzipping the snapshot archive again from
> whereever you have it stored locally?
>
> The way I see it is:
>  - If you're working on an entire checkout of classlib/trunk, you don't
> really need a snapshot build, because you can rebuild the deploy
> directory from scratch. How do you "reset" in this case? If you really
> don't want the changes you've made, you can always SVN revert and rebuild ;)
>  - If you're working on a single module and don't have the whole of
> classlib checked out, you need to build against a snapshot. In this case
> I think you'd want to be able to overlay your new libraries onto the
> snapshot - otherwise what other option do you have to get a complete
> build without checking out the rest of trunk?
>
> IMHO it's not a problem to overwrite the libraries with new altered ones
> - it's what we do now with the deploy directory (whether it's a snapshot
> or built from scratch).
>
> Regards,
> Oliver
>
> [1]
> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c200605091407.k49E749L028574@d06av02.portsmouth.uk.ibm.com%3e
> >
> >>
> >>
> >> Additionally, I agree with Nathan [2] that allowing a developer
> >> working in a modular way to run the entire test suite against their
> >> work-in-progress code would be a good thing, and this could be done
> >> by bundling tests and their resources in with an hdk.
> >>
> >> Regards,
> >> Oliver
> >>
> >> [1]
> >> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200602.mbox/%3cc3755b3a0602100609x1b79da6q@mail.gmail.com%3e
> >>
> >> [2]
> >> http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/%3c000f01c673e2$92e74130$0e01a8c0@LITTLEGUY%3e
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: harmony-dev-help@incubator.apache.org
> >
> >
>
> --
> Oliver Deakin
> IBM United Kingdom Limited
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
> For additional commands, e-mail: harmony-dev-help@incubator.apache.org
>
>

---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: harmony-dev-unsubscribe@incubator.apache.org
For additional commands, e-mail: harmony-dev-help@incubator.apache.org


Mime
View raw message