Mailing-List: contact dev-help@flex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flex.apache.org
Received-SPF: pass (athena.apache.org: domain of christofer.dutz@c-ware.de
 designates 213.199.154.80 as permitted sender)
From: Christofer Dutz <christofer.dutz@c-ware.de>
To: "dev@flex.apache.org" <dev@flex.apache.org>
Subject: AW: AW: AW: AW: AW: AW: Falcon and Antlr4
Thread-Topic: AW: AW: AW: AW: AW: Falcon and Antlr4
Thread-Index: 
 AQHPpkIeBoxAGiLXq0KN60akG6aQ2JutiOACgAATIwCAAACxAIAANrOAgAD4hVeAACPvj4ABHsUAgAA/WUA=
Date: Fri, 25 Jul 2014 06:06:25 +0000
Message-ID: 
 <7206befc26c04e3bb1b2805cf2169a1e@DBXPR05MB237.eurprd05.prod.outlook.com>
References: 
 <CFF4A737.28775%aharui@adobe.com>,<1406118993661.82952@c-ware.de>,<CAJs+wW1rX1b+Geov=BNc7r6Ja_yRN-HFDMAwLXXq80-5xV24wg@mail.gmail.com>,<1406120054728.73060@c-ware.de>,<SNT151-W48C4BE1774C77743BE5440D6FE0@phx.gbl>,<1406188644194.86575@c-ware.de>,<1406192918291.33449@c-ware.de>
 <SNT151-W18EE7A28090A59FF6FB512D6FC0@phx.gbl>
In-Reply-To: <SNT151-W18EE7A28090A59FF6FB512D6FC0@phx.gbl>
Accept-Language: de-DE, en-US
Content-Language: de-DE
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Gordon,

I won't object that Shaoting Cai certainly knew what he was doing, but Antl=
r4 is in some respects greatly different from 2 and 3. I bet if Antlr4 had =
been available at the time Falcon was initiate, he would have used that. No=
t only does it seem to allow much simpler grammars, it also generates an in=
terface for each grammar in which "enterXYZ" and "exitXYZ" callbacks are de=
fined for every rule in the grammar. I can hereby create an implementation =
of that interface and construct the FalconAST in here directly. Usually you=
 parse the input and get an AntlrAST and use a treewalker in conjunction wi=
th that Interface implementation, but you can also hook that up as parse-li=
stener and it's used by the initial parsing. This way I think I could remov=
e several tree traversals.

I also got closer on the JBurg part. I was actually wrong with the runtime =
and generator part. It's actually all just a Generator. I therefore have cr=
eate a jburg-maven-plugin that I can hook up into the build. The cool thing=
 about that ist hat every plugin has it's own classloader and therefore JBu=
rg can use whatever it likes without interfering with the rest. I am having=
 one really strange classloading error though that I still need to investig=
age and as soon as that's finished I promissed tom to donate the plugin to =
JBurg. I also had a small discussion with tom and he will be willing to upd=
ate JBurg to update JBurg to Antlr4 and Stringtemplate (The external string=
template, not the Antlr internal version). After that I agreed to make JBur=
g build with Maven ... so I guess we are having some progress on this :-)

Chris

-----Urspr=FCngliche Nachricht-----
Von: Gordon Smith [mailto:gsmithsf@hotmail.com]=20
Gesendet: Freitag, 25. Juli 2014 04:10
An: dev@flex.apache.org
Betreff: RE: AW: AW: AW: AW: AW: Falcon and Antlr4

> I am sort of confused what JBurg is actually working on ... Does it direc=
tly work on FalconAST or is there some sort of "ReducedFalconAST"?
=20
The input to the BURM (Bottom-Up Rewrite Machine) that is produced by JBurg=
 (Java Bottom-Up Rewrite Generator) is the FalconAST. For example, a BURM p=
attern like
=20
Pattern addExpr
Op_AddID(expression l, expression r);
=20
in CMCPatterns.jbg, representing a '+' expression, applies to each subtree =
headed by a BinaryOperatorPlusNode, because the getNodeId() method of this =
node class returns Op_AddId.
=20
> ABC code (I guess this is the native Flash bytecode?)
=20
Right. ABC stands for ActionScript Byte Code, which is the byte code esecut=
ed by the ActionScript Virtual Machine (AVM) in the Flash Player and AIR.
=20
> CSS... I would like to simplify this to:
> CSS --[CSSLexer]--> TokenStream --[CSSParser]--> [FalconAST]=20
> --[JBurg]--> SWF

I won't object, however the CSS implementation was done for Falcon by a for=
mer Adobe engineer (Shaoting Cai) who was well versed in Antlr, since he ha=
d been a student of the professor who developed Antlr. (The AS lexer and pa=
rser are much older, inherited from Flash Builder, and written by engineers=
 who were probably not as expert in Antlr, and were using older version of =
Antlr that didn't have tree parsers.) So I tend to think that the "Antlr wa=
y" may be to let Antlr create an AntlrAST automatically, and then transform=
 it into a more custom FalconAST with a tree parser. On the other hand, if =
this is slower than having the parser produce the FalconAST directly from t=
he token stream, which seems likely, then I vote for going for speed.
=20
> AS... Here I would like to remove the RawASTTokenizer and replace that=20
> with an Antlr4 Lexer AS --[ASLexer]--> TokenStream --[ASParser]-->=20
> FalconAST --[JBurg]--> SWF

This is what I had in mind, although again I'm not sure this would be consi=
dered Antlr best practice. (As before, if Shaoting had done the AS implemen=
tation, he probably would have produced an AntlrAST and then transformed it=
 into a FalconAST with a tree parser.)
=20
> Do we even need a dependency to JBurg itself to execute the code generate=
d by JBurg?
=20
Yes. As you discovered, there is a runtime part, which determines which seq=
uence of reductions has the lowest 'cost' and therefore gets used for code =
generation.
=20
> I will contact the JBurg project and check if it was possible to switch t=
o the new Stringtemplate project (and if it was possible to split up JBurg =
into two modules "generator", and "runtime").
=20
If Tom doesn't want to decouple JBurg from Antlr, one option would be for u=
s to fork JBurg and remove its dependency on the Antlr Stringtemplate. Is a=
nother option for JBurg to continue using whatever old Antlr it has been us=
ing and the AS and CSS lexer/parser to use the latest Antlr?
=20
- Gordon

=20
> From: christofer.dutz@c-ware.de
> To: dev@flex.apache.org
> Subject: AW: AW: AW: AW: AW: Falcon and Antlr4
> Date: Thu, 24 Jul 2014 09:08:38 +0000
>=20
> Ok ... so I created a JBurg Maven Plugin (Will contact the JBurg=20
> project lead for donating that)
>=20
> But (unfortunateley there's a but):
> - JBurg seems to consist of a Generator and a Runtime, but both are=20
> located in the same Jar (Even if this wouldn't really be a blocker)
> - JBurgs runtime part doesn't directly rely on Antlr AST classes, but=20
> unfortunateley Antlr Stringtemplate which is only bundled in Antlr3=20
> distributions because it seems to have been created a project spinoff=20
> containing only the stringtemplate part=20
> (http://www.stringtemplate.org/)
>=20
> I will contact the JBurg project and check if it was possible to switch t=
o the new Stringtemplate project (and if it was possible to split up JBurg =
into two modules "generator", and "runtime").
>=20
> Chris
>=20
>=20
> ________________________________________
> Von: Christofer Dutz <christofer.dutz@c-ware.de>
> Gesendet: Donnerstag, 24. Juli 2014 09:57
> An: dev@flex.apache.org
> Betreff: AW: AW: AW: AW: AW: Falcon and Antlr4
>=20
> So let's just define some words that might help avoid confusion.
>=20
> A typical Antlr parser geneates a parse-tree consisting of Antlr objects =
... let's call that AntlrAST.
> Falcon has it's own internal representation using its own objects ... let=
's call that FalconAST.
> Then I am sort of confused that JBurg is actually working on ... Does it =
directly work on FalconAST or is there some sort of "ReducedFalconAST"?
>=20
> Because I started with the CSS Parser, I saw that we use Antlr3 to genera=
te a parser for CSS files. That outputs an AntlrAST which is fed into a sec=
ond Antlr generated parser "CSSTree" which converts this AntlrAST into a Fa=
lconAST. After that I think (i'm not 100% sure) the FalconAST is passed in =
to the CSSReducer which sort of directly generates ABC code (I guess this i=
s the native Flash bytecode?).
>=20
> //////////
> // Css
> //////////
>=20
> CSS --[CSSLexer]--> TokenStream --[CSSParser]--> AntlrAST=20
> --[CSSTreeParser]--> FalconAST --[CSSReducer]--> [ReducedFalconAST]=20
> --[JBurg]--> SWF (Everything from the FalconAST still seems a little=20
> mystical to me though)
>=20
> I would like to simplyfy this to:
>=20
> CSS --[CSSLexer]--> TokenStream --[CSSParser]--> [FalconAST]=20
> --[JBurg]--> SWF
>=20
> //////////
> // ActionScript
> //////////
>=20
> For AS the procedure is different (As far as I got it together with your =
mail). We are using Antlr2 here:
>=20
> AS --[RawASTTokenizer]--> TokenStream --[ASParser]--> FalconAST=20
> --[JBurg]--> SWF
>=20
> Here I would like to remove the RawASTTokenizer and replace that with=20
> an Antlr4 Lexer
>=20
> AS --[ASLexer]--> TokenStream --[ASParser]--> FalconAST --[JBurg]-->=20
> SWF
>=20
> //////////
> // JBurg
> //////////
>=20
> In the meanwhile I think I had some progress. As far as I understand now =
JBurg is built by Antlr and uses Antlr to parse and generate output. The ou=
tput however no longer has a reference to Antlr. Do we even need a dependen=
cy to JBurg itself to execute the code generated by JBurg? If not I would w=
hip up a jburg-maven-plugin which wraps JBurgs execution. Then I think we s=
houldn't have any problems with Antlr versions.
>=20
> Currently a I want to do is do an Antlr4 POC for the CSS compilation. As =
soon as that's done, I would like to compare the performance to that of the=
 existing one and only if that is at least as fast as the current solution =
I would proceed to the next part (If its not slower than the current soluti=
on I would prefer the simpler process).
>=20
> Chris
>=20
>=20
> ________________________________________
> Von: Gordon Smith <gsmithsf@hotmail.com>
> Gesendet: Mittwoch, 23. Juli 2014 18:05
> An: dev@flex.apache.org
> Betreff: RE: AW: AW: AW: AW: Falcon and Antlr4
>=20
> CSSTree.g isn't involved in compiling AS files. It's involved only in com=
piling CSS files.
>=20
> For AS compilation: We use JFlex to create RawASTokenizer.java from RawAS=
Tokenizer.lex. This is a lexer which takes an AS file as input and creates =
a sequence of ASToken objects as output. We use Antlr 3 to create ASParser.=
java from ASParser.g. This is a parser which takes a sequence of ASToken ob=
jects as output and creates an AST consisting of NodeBase objects. This is =
a custom AST, not the kind of generic Antlr AST that Antlr can create autom=
atically.
>=20
> If Antlr 4 can tokenize as fast as JFlex can, then I think it would be gr=
eat to have a unified Antlr 4 lexer/parser. However, I strongly recommend k=
eeping Falcon's custom AST classes.
>=20
> I'm not an expert on JBurg and wasn't aware that it had a dependency on A=
ntlr. Can you please show me where this depencency is? The "JBurg guys" are=
 basically one person, Tom Harwood, who used to work at Adobe but no longer=
 does. I don't know whether he is still supporting JBurg or not.
>=20
> As for eliminating JBurg... This is certainly doable, but would have seri=
ous implications. The main reason that Falcon does code generation using JB=
urg is in order to perform various code optimizations such as constant prop=
agation. These code optimizations would be much harder to do without JBurg.=
 Ideally, these optimizations would be done in the ActionScript Virtual Mac=
hine rather than in the compiler (and this was the plan for AS4, which Adob=
e killed) but since they are NOT done in the current AVM, if the compiler d=
oesn't do them then the code will execute slower. However, if cross-compila=
tion to JavaScript is the main goal for the future, then perhaps slower AVM=
 execution is tolerable.
>=20
> - Gordon
>=20
>=20
> > From: christofer.dutz@c-ware.de
> > To: dev@flex.apache.org
> > Subject: AW: AW: AW: AW: Falcon and Antlr4
> > Date: Wed, 23 Jul 2014 12:54:15 +0000
> >
> > Hi Eric,
> >
> > The question is definitely not silly.
> >
> > I'm not planning on changing anything here. The package tangling=20
> > seems to be related with JBurg needing access to some objects that=20
> > are not part of the interface but part of the internal model. I=20
> > would untangle this by extending the Interface to provide anything=20
> > JBurg needs so there shouldn't be any change in the Falcon AST. But=20
> > after all that's just an optimization. Currently I'm digging in the=20
> > inner workings of JBurg ... gee I sort of forgot why I looked in the=20
> > inner workings of Falcon in the first place ;-)
> >
> > I would however like to remove as many processing phases and model=20
> > conversions as possible as each one costs time and memory and makes thi=
ngs more compilcated to understand.
> >
> > Chris
> >
> > ________________________________________
> > Von: Erik de Bruin <erik@ixsoftware.nl>
> > Gesendet: Mittwoch, 23. Juli 2014 14:47
> > An: dev@flex.apache.org
> > Betreff: Re: AW: AW: AW: Falcon and Antlr4
> >
> > I'm not at all familiar with the inner workings of Falcon, but I=20
> > know that FalconJX relies heavily on the current Falcon AST. Will=20
> > the new AST be compatible with the old (that may be a silly=20
> > question, but it's the only one I know to ask)?
> >
> > EdB
> >
> >
> >
> > On Wed, Jul 23, 2014 at 2:36 PM, Christofer Dutz=20
> > <christofer.dutz@c-ware.de>
> > wrote:
> >
> > > Ok ... so I used some idle time setting up a clean falcon-antlr4=20
> > > module starting with the CSS parsing.
> > >
> > > As far as I can see it, Falcon is currently using Antlr3 to=20
> > > generate the lexer and a parser that generates an Antlr AST from=20
> > > that Then we generate a second parser CSSTree.g to convert that=20
> > > AST into the falcon-internal object structure, which is then=20
> > > passed to JBurg to generate the output.
> > >
> > > As you mentioned the ability to manipulate the AST. Are you=20
> > > talking about the first (antlr) AST or the one the Falcon AST?
> > > I would like to elminiate the second parsing step and generate the=20
> > > internal model directly from the first parser using Antlr4s=20
> > > Listeners. Are there any objections against this? I doubt that=20
> > > anyone is manipulating the Antlr AST directly, much more would I=20
> > > expect operations to be performed on the Falcon AST.
> > >
> > > I managed to get the CSS Lexer and Parser finished. The grammar is=20
> > > way more readable now and the general structure is much more=20
> > > straight forward :-) Now I would like to Build the internal Falcon=20
> > > object model for that.
> > > Guess I would have to contact the JBurg guys for continuing from=20
> > > then as I definitely need Antlr4 support but I would bet that=20
> > > either JBurg is dead or they are working or at least thinking=20
> > > about that.
> > >
> > > I did detect some ugly cyclic dependencies in the falcon classes=20
> > > though ... for example we have interfaces for defining the=20
> > > internal models interface and have implementations of these=20
> > > interfaces in the internal packages, but unfortunately the=20
> > > interfaces reference classes in the internal model ... I think=20
> > > that's bad design and would like to clean that up.
> > >
> > > Any objections? (Keep in mind ... it's all in a dedicated branch,=20
> > > but I don't want to make it hard to merge back as soon as I'm=20
> > > finished)
> > >
> > > There is no DOM or SAX involved in Falcon ... I was using a=20
> > > metaphor to explain what I was thinking about by using an example=20
> > > from the XML processing world. Here you can load an XML document=20
> > > to memory (DOM), transform that into a second memory=20
> > > representation and so on in contrast to SAX processing where=20
> > > parser generates a stream of events which can be processed=20
> > > allowing you to process petabyte big documents with as much ram as=20
> > > a calculator ;-)
> > >
> > >
> > > Chris
> > >
> > >
> > > ________________________________________
> > > Von: Alex Harui <aharui@adobe.com>
> > > Gesendet: Mittwoch, 23. Juli 2014 08:48
> > > An: dev@flex.apache.org
> > > Betreff: Re: AW: AW: AW: Falcon and Antlr4
> > >
> > > On 7/22/14 12:07 PM, "Christofer Dutz" <christofer.dutz@c-ware.de> wr=
ote:
> > >
> > > >The way I understood JBurg is that it seems to have been designed=20
> > > >to manipulate ASTs produced by Antlr2 and Antlr3.
> > > >Therefore it has a compile and runtime dependency to Antlr (I=20
> > > >also sort of dislike them bundling the generator together with=20
> > > >the runtime stuff) but nevertheless ... I can't really use JBurg=20
> > > >with Antlr4 in my project not without splitting the module up but I =
think that's rather ugly.
> > > I'm definitely not an expert on the subject.  I'm surprised the=20
> > > AST has any Antlr-isms in it.  Maybe Gordon can confirm.  When I=20
> > > look at the AST, I don't see anything I find surprising.
> > >
> > > >
> > > >I too was thinking about making JBurg obsolete too ... the way I=20
> > > >understand Antlr4 it looked as if it provided a lot which could=20
> > > >make life a lot easier. For example currently the ASParser=20
> > > >creates the AST objects and JBurg then performs it's magic on=20
> > > >that AST tree. In Antlr4 I could create a Parser-Listener that=20
> > > >produces the AST object (Probably needed for IDE suppoer) but=20
> > > >could probably create a second listener that directly outputs=20
> > > >flash bytecode. I think this approach should be maximum=20
> > > >performant, maximum simple and use only a fragment of the memory=20
> > > >the current solutions need ... Sort of XML Dom processing (old AST p=
arsing) and SAX (Antlr4 direct Bytecode output).
> > > I have not played with SAX much nor Antlr, so I have no idea about=20
> > > parser/listener.  I will say that there are times folks want to=20
> > > create the AST and then manipulate it before generating the=20
> > > output, so unless there is a huge performance win, making sure=20
> > > there is a way to do that would be preferred.
> > >
> > > >
> > > >But I also think that this would be quite an effort. Perhaps I=20
> > > >should split up my falcon-antlr4 branch even more and sort of=20
> > > >start with the CSS parser and as soon as that's up and running get t=
he output running ...
> > > >then we could compare the performance of both and see if it's=20
> > > >worth going down that path. Don't want to waste time I could be=20
> > > >working on Flexmojos, or the new Flex-Maven-Plugin for a less=20
> > > >performant solution. But if it was faster (I would expect it to be) =
it could be quite breakthough ...
> > > >after all I have never seen several datastructure conversions be=20
> > > >faster than direct output.
> > > Up to you since you are doing the work.  I just want to make sure=20
> > > you don't feel like you have to keep all of these subsystems like=20
> > > Jburg and Jflex around.
> > >
> > > -Alex
> > >
> > >
> >
> >
> > --
> > Ix Multimedia Software
> >
> > Jan Luykenstraat 27
> > 3521 VB Utrecht
> >
> > T. 06-51952295
> > I. www.ixsoftware.nl
 		 	   		 =20