flex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christofer Dutz <christofer.d...@c-ware.de>
Subject AW: AW: AW: AW: AW: AW: Falcon and Antlr4
Date Fri, 25 Jul 2014 06:06:25 GMT
Hi Gordon,

I won't object that Shaoting Cai certainly knew what he was doing, but Antlr4 is in some respects
greatly different from 2 and 3. I bet if Antlr4 had been available at the time Falcon was
initiate, he would have used that. Not only does it seem to allow much simpler grammars, it
also generates an interface for each grammar in which "enterXYZ" and "exitXYZ" callbacks are
defined for every rule in the grammar. I can hereby create an implementation of that interface
and construct the FalconAST in here directly. Usually you parse the input and get an AntlrAST
and use a treewalker in conjunction with that Interface implementation, but you can also hook
that up as parse-listener and it's used by the initial parsing. This way I think I could remove
several tree traversals.

I also got closer on the JBurg part. I was actually wrong with the runtime and generator part.
It's actually all just a Generator. I therefore have create a jburg-maven-plugin that I can
hook up into the build. The cool thing about that ist hat every plugin has it's own classloader
and therefore JBurg can use whatever it likes without interfering with the rest. I am having
one really strange classloading error though that I still need to investigage and as soon
as that's finished I promissed tom to donate the plugin to JBurg. I also had a small discussion
with tom and he will be willing to update JBurg to update JBurg to Antlr4 and Stringtemplate
(The external stringtemplate, not the Antlr internal version). After that I agreed to make
JBurg build with Maven ... so I guess we are having some progress on this :-)


-----Urspr√ľngliche Nachricht-----
Von: Gordon Smith [mailto:gsmithsf@hotmail.com] 
Gesendet: Freitag, 25. Juli 2014 04:10
An: dev@flex.apache.org
Betreff: RE: AW: AW: AW: AW: AW: Falcon and Antlr4

> I am sort of confused what JBurg is actually working on ... Does it directly work on
FalconAST or is there some sort of "ReducedFalconAST"?
The input to the BURM (Bottom-Up Rewrite Machine) that is produced by JBurg (Java Bottom-Up
Rewrite Generator) is the FalconAST. For example, a BURM pattern like
Pattern addExpr
Op_AddID(expression l, expression r);
in CMCPatterns.jbg, representing a '+' expression, applies to each subtree headed by a BinaryOperatorPlusNode,
because the getNodeId() method of this node class returns Op_AddId.
> ABC code (I guess this is the native Flash bytecode?)
Right. ABC stands for ActionScript Byte Code, which is the byte code esecuted by the ActionScript
Virtual Machine (AVM) in the Flash Player and AIR.
> CSS... I would like to simplify this to:
> CSS --[CSSLexer]--> TokenStream --[CSSParser]--> [FalconAST] 
> --[JBurg]--> SWF

I won't object, however the CSS implementation was done for Falcon by a former Adobe engineer
(Shaoting Cai) who was well versed in Antlr, since he had been a student of the professor
who developed Antlr. (The AS lexer and parser are much older, inherited from Flash Builder,
and written by engineers who were probably not as expert in Antlr, and were using older version
of Antlr that didn't have tree parsers.) So I tend to think that the "Antlr way" may be to
let Antlr create an AntlrAST automatically, and then transform it into a more custom FalconAST
with a tree parser. On the other hand, if this is slower than having the parser produce the
FalconAST directly from the token stream, which seems likely, then I vote for going for speed.
> AS... Here I would like to remove the RawASTTokenizer and replace that 
> with an Antlr4 Lexer AS --[ASLexer]--> TokenStream --[ASParser]--> 
> FalconAST --[JBurg]--> SWF

This is what I had in mind, although again I'm not sure this would be considered Antlr best
practice. (As before, if Shaoting had done the AS implementation, he probably would have produced
an AntlrAST and then transformed it into a FalconAST with a tree parser.)
> Do we even need a dependency to JBurg itself to execute the code generated by JBurg?
Yes. As you discovered, there is a runtime part, which determines which sequence of reductions
has the lowest 'cost' and therefore gets used for code generation.
> I will contact the JBurg project and check if it was possible to switch to the new Stringtemplate
project (and if it was possible to split up JBurg into two modules "generator", and "runtime").
If Tom doesn't want to decouple JBurg from Antlr, one option would be for us to fork JBurg
and remove its dependency on the Antlr Stringtemplate. Is another option for JBurg to continue
using whatever old Antlr it has been using and the AS and CSS lexer/parser to use the latest
- Gordon

> From: christofer.dutz@c-ware.de
> To: dev@flex.apache.org
> Subject: AW: AW: AW: AW: AW: Falcon and Antlr4
> Date: Thu, 24 Jul 2014 09:08:38 +0000
> Ok ... so I created a JBurg Maven Plugin (Will contact the JBurg 
> project lead for donating that)
> But (unfortunateley there's a but):
> - JBurg seems to consist of a Generator and a Runtime, but both are 
> located in the same Jar (Even if this wouldn't really be a blocker)
> - JBurgs runtime part doesn't directly rely on Antlr AST classes, but 
> unfortunateley Antlr Stringtemplate which is only bundled in Antlr3 
> distributions because it seems to have been created a project spinoff 
> containing only the stringtemplate part 
> (http://www.stringtemplate.org/)
> I will contact the JBurg project and check if it was possible to switch to the new Stringtemplate
project (and if it was possible to split up JBurg into two modules "generator", and "runtime").
> Chris
> ________________________________________
> Von: Christofer Dutz <christofer.dutz@c-ware.de>
> Gesendet: Donnerstag, 24. Juli 2014 09:57
> An: dev@flex.apache.org
> Betreff: AW: AW: AW: AW: AW: Falcon and Antlr4
> So let's just define some words that might help avoid confusion.
> A typical Antlr parser geneates a parse-tree consisting of Antlr objects ... let's call
that AntlrAST.
> Falcon has it's own internal representation using its own objects ... let's call that
> Then I am sort of confused that JBurg is actually working on ... Does it directly work
on FalconAST or is there some sort of "ReducedFalconAST"?
> Because I started with the CSS Parser, I saw that we use Antlr3 to generate a parser
for CSS files. That outputs an AntlrAST which is fed into a second Antlr generated parser
"CSSTree" which converts this AntlrAST into a FalconAST. After that I think (i'm not 100%
sure) the FalconAST is passed in to the CSSReducer which sort of directly generates ABC code
(I guess this is the native Flash bytecode?).
> //////////
> // Css
> //////////
> CSS --[CSSLexer]--> TokenStream --[CSSParser]--> AntlrAST 
> --[CSSTreeParser]--> FalconAST --[CSSReducer]--> [ReducedFalconAST] 
> --[JBurg]--> SWF (Everything from the FalconAST still seems a little 
> mystical to me though)
> I would like to simplyfy this to:
> CSS --[CSSLexer]--> TokenStream --[CSSParser]--> [FalconAST] 
> --[JBurg]--> SWF
> //////////
> // ActionScript
> //////////
> For AS the procedure is different (As far as I got it together with your mail). We are
using Antlr2 here:
> AS --[RawASTTokenizer]--> TokenStream --[ASParser]--> FalconAST 
> --[JBurg]--> SWF
> Here I would like to remove the RawASTTokenizer and replace that with 
> an Antlr4 Lexer
> AS --[ASLexer]--> TokenStream --[ASParser]--> FalconAST --[JBurg]--> 
> //////////
> // JBurg
> //////////
> In the meanwhile I think I had some progress. As far as I understand now JBurg is built
by Antlr and uses Antlr to parse and generate output. The output however no longer has a reference
to Antlr. Do we even need a dependency to JBurg itself to execute the code generated by JBurg?
If not I would whip up a jburg-maven-plugin which wraps JBurgs execution. Then I think we
shouldn't have any problems with Antlr versions.
> Currently a I want to do is do an Antlr4 POC for the CSS compilation. As soon as that's
done, I would like to compare the performance to that of the existing one and only if that
is at least as fast as the current solution I would proceed to the next part (If its not slower
than the current solution I would prefer the simpler process).
> Chris
> ________________________________________
> Von: Gordon Smith <gsmithsf@hotmail.com>
> Gesendet: Mittwoch, 23. Juli 2014 18:05
> An: dev@flex.apache.org
> Betreff: RE: AW: AW: AW: AW: Falcon and Antlr4
> CSSTree.g isn't involved in compiling AS files. It's involved only in compiling CSS files.
> For AS compilation: We use JFlex to create RawASTokenizer.java from RawASTokenizer.lex.
This is a lexer which takes an AS file as input and creates a sequence of ASToken objects
as output. We use Antlr 3 to create ASParser.java from ASParser.g. This is a parser which
takes a sequence of ASToken objects as output and creates an AST consisting of NodeBase objects.
This is a custom AST, not the kind of generic Antlr AST that Antlr can create automatically.
> If Antlr 4 can tokenize as fast as JFlex can, then I think it would be great to have
a unified Antlr 4 lexer/parser. However, I strongly recommend keeping Falcon's custom AST
> I'm not an expert on JBurg and wasn't aware that it had a dependency on Antlr. Can you
please show me where this depencency is? The "JBurg guys" are basically one person, Tom Harwood,
who used to work at Adobe but no longer does. I don't know whether he is still supporting
JBurg or not.
> As for eliminating JBurg... This is certainly doable, but would have serious implications.
The main reason that Falcon does code generation using JBurg is in order to perform various
code optimizations such as constant propagation. These code optimizations would be much harder
to do without JBurg. Ideally, these optimizations would be done in the ActionScript Virtual
Machine rather than in the compiler (and this was the plan for AS4, which Adobe killed) but
since they are NOT done in the current AVM, if the compiler doesn't do them then the code
will execute slower. However, if cross-compilation to JavaScript is the main goal for the
future, then perhaps slower AVM execution is tolerable.
> - Gordon
> > From: christofer.dutz@c-ware.de
> > To: dev@flex.apache.org
> > Subject: AW: AW: AW: AW: Falcon and Antlr4
> > Date: Wed, 23 Jul 2014 12:54:15 +0000
> >
> > Hi Eric,
> >
> > The question is definitely not silly.
> >
> > I'm not planning on changing anything here. The package tangling 
> > seems to be related with JBurg needing access to some objects that 
> > are not part of the interface but part of the internal model. I 
> > would untangle this by extending the Interface to provide anything 
> > JBurg needs so there shouldn't be any change in the Falcon AST. But 
> > after all that's just an optimization. Currently I'm digging in the 
> > inner workings of JBurg ... gee I sort of forgot why I looked in the 
> > inner workings of Falcon in the first place ;-)
> >
> > I would however like to remove as many processing phases and model 
> > conversions as possible as each one costs time and memory and makes things more
compilcated to understand.
> >
> > Chris
> >
> > ________________________________________
> > Von: Erik de Bruin <erik@ixsoftware.nl>
> > Gesendet: Mittwoch, 23. Juli 2014 14:47
> > An: dev@flex.apache.org
> > Betreff: Re: AW: AW: AW: Falcon and Antlr4
> >
> > I'm not at all familiar with the inner workings of Falcon, but I 
> > know that FalconJX relies heavily on the current Falcon AST. Will 
> > the new AST be compatible with the old (that may be a silly 
> > question, but it's the only one I know to ask)?
> >
> > EdB
> >
> >
> >
> > On Wed, Jul 23, 2014 at 2:36 PM, Christofer Dutz 
> > <christofer.dutz@c-ware.de>
> > wrote:
> >
> > > Ok ... so I used some idle time setting up a clean falcon-antlr4 
> > > module starting with the CSS parsing.
> > >
> > > As far as I can see it, Falcon is currently using Antlr3 to 
> > > generate the lexer and a parser that generates an Antlr AST from 
> > > that Then we generate a second parser CSSTree.g to convert that 
> > > AST into the falcon-internal object structure, which is then 
> > > passed to JBurg to generate the output.
> > >
> > > As you mentioned the ability to manipulate the AST. Are you 
> > > talking about the first (antlr) AST or the one the Falcon AST?
> > > I would like to elminiate the second parsing step and generate the 
> > > internal model directly from the first parser using Antlr4s 
> > > Listeners. Are there any objections against this? I doubt that 
> > > anyone is manipulating the Antlr AST directly, much more would I 
> > > expect operations to be performed on the Falcon AST.
> > >
> > > I managed to get the CSS Lexer and Parser finished. The grammar is 
> > > way more readable now and the general structure is much more 
> > > straight forward :-) Now I would like to Build the internal Falcon 
> > > object model for that.
> > > Guess I would have to contact the JBurg guys for continuing from 
> > > then as I definitely need Antlr4 support but I would bet that 
> > > either JBurg is dead or they are working or at least thinking 
> > > about that.
> > >
> > > I did detect some ugly cyclic dependencies in the falcon classes 
> > > though ... for example we have interfaces for defining the 
> > > internal models interface and have implementations of these 
> > > interfaces in the internal packages, but unfortunately the 
> > > interfaces reference classes in the internal model ... I think 
> > > that's bad design and would like to clean that up.
> > >
> > > Any objections? (Keep in mind ... it's all in a dedicated branch, 
> > > but I don't want to make it hard to merge back as soon as I'm 
> > > finished)
> > >
> > > There is no DOM or SAX involved in Falcon ... I was using a 
> > > metaphor to explain what I was thinking about by using an example 
> > > from the XML processing world. Here you can load an XML document 
> > > to memory (DOM), transform that into a second memory 
> > > representation and so on in contrast to SAX processing where 
> > > parser generates a stream of events which can be processed 
> > > allowing you to process petabyte big documents with as much ram as 
> > > a calculator ;-)
> > >
> > >
> > > Chris
> > >
> > >
> > > ________________________________________
> > > Von: Alex Harui <aharui@adobe.com>
> > > Gesendet: Mittwoch, 23. Juli 2014 08:48
> > > An: dev@flex.apache.org
> > > Betreff: Re: AW: AW: AW: Falcon and Antlr4
> > >
> > > On 7/22/14 12:07 PM, "Christofer Dutz" <christofer.dutz@c-ware.de> wrote:
> > >
> > > >The way I understood JBurg is that it seems to have been designed 
> > > >to manipulate ASTs produced by Antlr2 and Antlr3.
> > > >Therefore it has a compile and runtime dependency to Antlr (I 
> > > >also sort of dislike them bundling the generator together with 
> > > >the runtime stuff) but nevertheless ... I can't really use JBurg 
> > > >with Antlr4 in my project not without splitting the module up but I think
that's rather ugly.
> > > I'm definitely not an expert on the subject.  I'm surprised the 
> > > AST has any Antlr-isms in it.  Maybe Gordon can confirm.  When I 
> > > look at the AST, I don't see anything I find surprising.
> > >
> > > >
> > > >I too was thinking about making JBurg obsolete too ... the way I 
> > > >understand Antlr4 it looked as if it provided a lot which could 
> > > >make life a lot easier. For example currently the ASParser 
> > > >creates the AST objects and JBurg then performs it's magic on 
> > > >that AST tree. In Antlr4 I could create a Parser-Listener that 
> > > >produces the AST object (Probably needed for IDE suppoer) but 
> > > >could probably create a second listener that directly outputs 
> > > >flash bytecode. I think this approach should be maximum 
> > > >performant, maximum simple and use only a fragment of the memory 
> > > >the current solutions need ... Sort of XML Dom processing (old AST parsing)
and SAX (Antlr4 direct Bytecode output).
> > > I have not played with SAX much nor Antlr, so I have no idea about 
> > > parser/listener.  I will say that there are times folks want to 
> > > create the AST and then manipulate it before generating the 
> > > output, so unless there is a huge performance win, making sure 
> > > there is a way to do that would be preferred.
> > >
> > > >
> > > >But I also think that this would be quite an effort. Perhaps I 
> > > >should split up my falcon-antlr4 branch even more and sort of 
> > > >start with the CSS parser and as soon as that's up and running get the
output running ...
> > > >then we could compare the performance of both and see if it's 
> > > >worth going down that path. Don't want to waste time I could be 
> > > >working on Flexmojos, or the new Flex-Maven-Plugin for a less 
> > > >performant solution. But if it was faster (I would expect it to be) it
could be quite breakthough ...
> > > >after all I have never seen several datastructure conversions be 
> > > >faster than direct output.
> > > Up to you since you are doing the work.  I just want to make sure 
> > > you don't feel like you have to keep all of these subsystems like 
> > > Jburg and Jflex around.
> > >
> > > -Alex
> > >
> > >
> >
> >
> > --
> > Ix Multimedia Software
> >
> > Jan Luykenstraat 27
> > 3521 VB Utrecht
> >
> > T. 06-51952295
> > I. www.ixsoftware.nl

View raw message