Mailing-List: contact harmony-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: harmony-dev@incubator.apache.org
Received-SPF: neutral (herse.apache.org: 216.86.168.178 is neither permitted
 nor denied by domain of geir@pobox.com)
Message-ID: <45496B9E.2020109@pobox.com>
Date: Wed, 01 Nov 2006 22:53:02 -0500
From: "Geir Magnusson Jr." <geir@pobox.com>
Reply-To: geir@pobox.com
User-Agent: Thunderbird 1.5.0.7 (Macintosh/20060909)
MIME-Version: 1.0
To: harmony-dev@incubator.apache.org
Subject: Re: [classlib] Preprocessor - CHECKPOINT
References: <8E389A5F2FEABA4CB1DEC35A25CB39CE694EF8@mssmsx411>
	 <4546709B.9090704@gmail.com>
 <4546B5DE.3040609@pobox.com>	 <bc79dd600610310010w155a97f4wd99deab516187c7a@mail.gmail.com>
	 <45470FA5.2000801@gmail.com>
	 <bc79dd600610310117j5bb0903ds509f0d854845477e@mail.gmail.com>
	 <45473BA9.1070709@gmail.com>
	 <bc79dd600610310412w6b7a64dfo3c4758ddb14409c0@mail.gmail.com>
	 <45474144.5060108@pobox.com> <4547431C.5080000@pobox.com>
 <3b3f27c60610311705q1cb2aeb1wa78d0c55fb7f026b@mail.gmail.com>
 <4547F8D5.9040409@pobox.com> <454870F8.8080302@gmail.com>
 <454884DC.7070806@pobox.com> <4548A644.6060403@sablevm.org>
In-Reply-To: <4548A644.6060403@sablevm.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit


Etienne Gagnon wrote:
> Geir Magnusson Jr. wrote:
>> There's caching too, I think.  LogCache4J
>>
>> What I meant was that it didn't seem like we came to a conclusion on it
>> - that if we had a general pre-processing solution, we could use that
>> too for logging, rather than have two.
>>
>> The actual use-cases will help figure this out.
> 
> Here two typical some use cases, and some proposed solutions:
> 
> Problem
> -------
> logging, and other situations where you really don't want to put the
> "additional" source code in the main source files
> 
> Solution
> --------
> use aspects  (Plug: you might want to give a look at the optimizing abc
> compiler)
> 
> 
> Problem
> -------
> supporting a different API specifications/versions, such as j2me and
> j2se1.4, in addition to the "main" version (e.g. j2se1.5)
> 
> 
> Solution
> --------
> 

Right - these were the main hypotheticals that started this whole thread 
off - I was thinking more about a definite example, like a given class C 
has a different implementation of method M in Java5 than in Java6.

Even if we walk through this with a clear example simple manual process, 
it will be enlightening.

> This is a trickier problem.  We can divide the problem at two main levels:
> 
> 1- file/directory level
> 2- source-code level
> 
> At the file/directory level, each version (e.g. j2me, j2se1.4, ...)
> might include additional files (relative to the main version), and might
> not include some files of the main version.  In other words, j2me might
> not contain database APIs.

Yep

> 
> Managing file inclusion/exclusion can be done in various ways.
> 
>  a) ant based:  use distinct ant (sub-)files for each version.
> 
>    The problem is that IDEs (e.g. Eclipse) will most likely show some
>    of the excluded files in its class/files browser.  This wouldn't be
>    very elegant.  It also implies "always" compiling through ant files.
> 
>    Of course, one could develop Eclipse-specific configuration files to
>    mimic the inclusion/exclusion of ant files, but then, this opens the
>    door for discrepancies between ant vs eclipse inclusion/exclusion
>    lists.  I don't like it.
> 

Me neither.

>  b) custom-tool based: the custom processing tool I proposed could also
>    parse inclusion/exclusion lists, and use these lists to move files
>    around, in addition to processing the content of unmoved files.

I'm not sure anything needs to be moved.

> 
>    For example, if class X of the main version is not part of j2me,
>    "process(j2me)" would move this file to a subdirectory ".streams/".
> 
>    If a class Y is not in the "main" version (the one used for "svn
>    ci"), it resides in subdirectory ".streams" in the trunk.
>    "process(j2me)" moves this file into the normal directory.
> 
>    As for IDEs, now you can configure them to automatically exclude
>    ".stream/" directories.

This would get messy.  I'd rather just have a plug-in that reads a 
description file for version X and just doesn't show me what isn't part 
of version X.

> 
>    Inclusion/exclusion could be managed in two ways:
>    1- the processing tool could look for inclusion/exclusion list files,
>       such as "j2me.inclusion, j2me.exclusion, main.inclusion,
>       main.exclusion, etc."
> 
>       This would lead to the best performance (for process(X)), yet it
>       does require much manual update of inclusion/exclusion lists, with
>       all the discrepancies that result from human mistakes while
>       updating these files.
> 
>    2- (my preferred way), directives, at the beginning of the source
>       code would indicate whether a file is included in a version or
>       not.  Depending on target, the file would be moved to the
>       appropriate directory (normal, ".streams").

Either one - the nice thing about #1 over #2 is that you can actually 
look at it to get a summary.  But you can generate the same info w/ a 
tree walk, I suppose.

> 
> 
>     Of course, there's also the problem of non-source files, i.e.
>     resources.  IMO, resources could be managed using specific
>     directories (".main/", ".j2me", ".j2se1.4") and a ".shared/"
>     directory with symbolic links in the specific directories.
> 

This is getting messy.

> 
> As for source-level management, you would use my proposed processing
> tool to view the source files with the right spectacles [as Tim said so
> elegantly:-)].  For "development targets", it is important that:
> 
>  revert(process(X, target)) => X
> 
> By "development target" I mean a target that is meant for Harmony
> developers in prevision of reverting "modified code" to a format
> suitable for "svn ci" (i.e. revert to "main" target).
> 
> For comfortable IDE development, one could imagine that the IDE editor
> can reduce to "one-line visible" comments (or better, specially
> formatted ones) so that it gives you the impression that you are really
> wearing target-specific spectacles.  [I know Eclipse allows for such
> things already].
> 
> To release code, one would apply:
> 
>  process(X, release-target) => Y
> 
> Now, it is important to understand that Y, in this case, is NOT suitable
> for doing any modification as
> 
>  revert(Y) => Kaboom!  (The tool will simply report that it can't do it;
>                         it won't crash.)
> 
> Yet, I think that it would be important that the processing tool leaves
> "markers" in Y, so that we could also have a tool to help finding the
> canonical source line of a reported bug:
> 
>  revertLine(Y, L') => L  (where L' is a line reported by end-developer,
>                           and L the related line in "svn").
> 
> Markers would be short single lines comments, so the end-developer
> annoyance would be kept minimal.
> 
> 
> What do you think?

This is what I thought we were talking about all along - basically 
starting w/ the full source, and pre-process to the "canonical" source 
for the target version.

However, I don't understand why I can't go backwards, modulo some manual 
merging if needed.

For example, if I have X and release-version, I should be able to take a 
Y' and resolve back to X'.  There are problematic cases.  For example, 
in psuedo code and psuedo directives :

X :

   public void woogie() {

     firstLineJava;

    #ifdef(Java 6)
         middleLineJava6;
    #endif

     lastLineJava;
   }

then

process(X, Java5) outputs


Y:

   public void woogie() {

     firstLineJava;

     lastLineJava;
   }


Now, if I make a Y'

   public void woogie() {
     newLineJava5;

     firstLineJava;

     lastLineJava;
   }

I should be able to

   revert(X, java5, Y') -> X' such that

X' :

   public void woogie() {

     #ifdef(java5)
        newLineJava5;
     #endif

     firstLineJava;

    #ifdef(Java 6)
      middleLineJava6;
    #endif

     lastLineJava;
   }

No problem.  (Maybe a flag that tells revert that the change is for all 
versions....)

However if

Y' :

   public void woogie() {

     firstLineJava;

     newLineJava5;

     lastLineJava;
   }

then I'm not sure if X' is

   public void woogie() {

     firstLineJava;

     #ifdef(java5)
        newLineJava5;
     #endif

    #ifdef(Java 6)
      middleLineJava6;
    #endif

     lastLineJava;
   }

or

   public void woogie() {

     firstLineJava;

    #ifdef(Java 6)
      middleLineJava6;
    #endif

     #ifdef(java5)
        newLineJava5;
     #endif

     lastLineJava;
   }

In this case it may not matter, but either way, simply doing a 
merge-failure style >>>>>> marker should make it clear to the developer 
that some assistance is needed.

I also realize that this is a toy simplistic case :)

> 
> 
> I am really offering to develop this custom tool.  Just help me identify
> various Harmony needs I might have missed!

Cool! Go for it!

> 
> Of course, this tool is not the best solution to ALL problems, yet, so
> far, I think that it seems to best address the problem of supporting
> various API versions.

lets sketch out a few real examples, say between Java5 and Java6, and 
work them through.  That should give you a good idea on what the tool 
will be required to do.

geir