drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jinfeng Ni <...@apache.org>
Subject Re: [DISCUSS] Reduce use of freemarker templates for functions
Date Tue, 28 Apr 2015 15:46:30 GMT
Agree that we need add comments in the Freemarker template. At minimum, we
need put the "template file name" in the comment, so that by looking at the
generated source code, we know which template produces the source code, and
could easily go to the corresponding template if we have to make some
change.



On Tue, Apr 28, 2015 at 8:18 AM, Jacques Nadeau <jacques@apache.org> wrote:

> I agree with a number of thoughts here.  The reality is that we're not
> applying the same level of engineering to our templates that we apply to
> our other code.  A number of our templates are very poorly constructed
> right now.  I think correction is a combination of
>
> 1) don't use templates when they gain us nothing or extremely little.
> 2) Use more of Freemarker constructs to make code more manageable.  This
> includes using macro blocks, includes, comments (very important!) and
> pre-generating variables rather than doing all sorts of if/else's in the
> generated code.
>
> So (1) has a place but in most cases, we just need to leverage the
> technology better.  Many of our current templates look like a first time
> programmer might write a program.  (e.g. no abstraction, stream of
> consciousness if statements, no reuse and no thought to recurring
> patterns.)
>
>
>
> On Tue, Apr 28, 2015 at 4:50 AM, Steven Phillips <sphillips@maprtech.com>
> wrote:
>
> > -1
> >
> > Even considering how inconvenient working with the templates can be, I
> > think having to deal with the functions manually will be much worse.
> >
> > On Mon, Apr 27, 2015 at 4:50 PM, Jason Altekruse <
> altekrusejason@gmail.com
> > >
> > wrote:
> >
> > > I agree that there are parts of the current design that are
> frustrating,
> > > even with the relatively small amount of time I have spent with the
> > > freemarker templates.
> > >
> > > I am thinking that Julian is worried we will be taking a more
> traditional
> > > java architecture and losing performance, but I'm getting the
> impression
> > > that you want to explode out the templates into source code that will
> be
> > > similarly structured to the current output of the templates and check
> > that
> > > into version control.
> > >
> > > Obviously, if my impression is correct, we will be running the risk of
> > > needing to fix problems in several places and introducing
> > inconsistencies.
> > > This needs to always be balanced against debug efficiency and execution
> > > speed.
> > >
> > > I believe there is actually a separate architecture/build problem that
> > > might more appropriately address the issue with the functions. When I
> > have
> > > done work on the value vectors it has always worked that I can modify
> the
> > > freemarker results and rebuild the project in my IDE and get immediate
> > > feedback on the changes. On the other hand, I have always had to run
> the
> > > build from the command line when changing the functions, because we
> pick
> > up
> > > the source code of the functions for java code generation from the Jar
> > > files created in the build. As far as I know, there is no way to
> compile
> > > all of the current functions during the build without re-running the
> > > freemarker generation. This pretty much requires making all changes to
> > the
> > > functions in the templates themselves. It provides no opportunity to
> > debug
> > > quickly in the IDE to get the changes working with one of the types and
> > > then making the corresponding changes to the templates for final
> testing
> > > with the other types after a run of the full build.
> > >
> > > I have two suggestions to possibly address this problem. The first is
> to
> > > take a look at the interpreted expression evaluator that was added for
> > the
> > > constant folding rule. To avoid the overhead of compiling java during
> > > planning an alternative expression interpreter is used. The interpreter
> > as
> > > it works now (to solve some build issues it was refactored a little
> from
> > > Jinfeng's original design) just creates instances of the java classes
> for
> > > the functions and ties together the results of each with reflection
> when
> > > walking the expression tree. This allows for stepping into the function
> > > bodies directly in the IDE, with full debugging available for all of
> the
> > > members of the function class. This also does not rely on fetching the
> > > source code from the full build process, so you can make changes to
> > > functions and test them immediately.
> > >
> > > Secondly, it may be worthwhile to allow the build to be run while
> > ignoring
> > > the freemarker generation. This would allow for testing changes to the
> > > functions after modifying the freemarker result java files for a small
> > set
> > > of types, rather than requiring a change to the template that must work
> > for
> > > everything. This would still require running most of the build between
> > > tests, but I think this is the only way to test the functions in the
> > > context of the code-generation based evaluation.
> > >
> > > - Jason
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Apr 27, 2015 at 3:09 PM, Julian Hyde <julianhyde@gmail.com>
> > wrote:
> > >
> > > > If you can write all of this as Java code and the JIT can handle it,
> > that
> > > > would be great. But I’d be surprised.
> > > >
> > > > If freemarker template for a particular function is complex, that is
> > > > probably because it is doing a lot of customization for particular
> > cases.
> > > > In other words, it is adding a lot of value.
> > > >
> > > > If the templates make simple things look simple and complex things
> > > > possible, they are a textbook example of a good architecture. Don’t
> > throw
> > > > them out.
> > > >
> > > > You know the code a lot better than I do. If you have any
> > > counter-examples
> > > > to what I’ve said above, please prove me wrong. :)
> > > >
> > > > Julian
> > > >
> > > >
> > > >
> > > > On Apr 27, 2015, at 2:57 PM, Mehant Baid <baid.mehant@gmail.com>
> > wrote:
> > > >
> > > > > Hey All,
> > > > >
> > > > > We use freemarker templates extensively for generating source code
> > for
> > > > value vectors and functions. However over time the template logic for
> > > > functions has become complicated with the need to modify the function
> > > > templates based on input data type, nullability, function type etc.
> > This
> > > > makes the template code extremely hard to read and debug. Proposal
> > going
> > > > forward is to de-templatize the functions and check-in the source for
> > > > various functions (like how we have been doing in recent times for
> hash
> > > > functions etc). Although its not ideal as it involves some code
> > > duplication
> > > > but the advantage is that the source is then 'IDE friendly' and much
> > more
> > > > readable. As part of the work for DRILL-2870 I am planning to
> > > de-templatize
> > > > the aggregate functions.
> > > > >
> > > > > Thanks
> > > > > Mehant
> > > >
> > > >
> > >
> >
> >
> >
> > --
> >  Steven Phillips
> >  Software Engineer
> >
> >  mapr.com
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message