Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Mon, 19 Dec 2016 22:33:58 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.13021973.1479607723000.557692.1482186838750@Atlassian.JIRA>
In-Reply-To: <JIRA.13021973.1479607723000@Atlassian.JIRA>
References: <JIRA.13021973.1479607723000@Atlassian.JIRA> <JIRA.13021973.1479607723588@arcas>
Subject: [jira] [Commented] (DRILL-5052) Option to debug generated Java code
 using an IDE
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 19 Dec 2016 22:34:00 -0000


    [ https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762510#comment-15762510 ] 

ASF GitHub Bot commented on DRILL-5052:
---------------------------------------

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/660#discussion_r93133935
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassBuilder.java ---
    @@ -0,0 +1,189 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.drill.exec.compile;
    +
    +import java.io.File;
    +import java.io.FileWriter;
    +import java.io.IOException;
    +import java.util.Map;
    +
    +import org.apache.drill.common.config.DrillConfig;
    +import org.apache.drill.exec.compile.ClassTransformer.ClassNames;
    +import org.apache.drill.exec.exception.ClassTransformationException;
    +import org.apache.drill.exec.expr.CodeGenerator;
    +import org.apache.drill.exec.server.options.OptionManager;
    +import org.codehaus.commons.compiler.CompileException;
    +
    +/**
    + * Implements the "plain-old Java" method of code generation and
    + * compilation. Given a {@link CodeGenerator}, obtains the generated
    + * source code, compiles it with the selected compiler, loads the
    + * byte-codes into a class loader and provides the resulting
    + * class. Compared with the {@link ClassTransformer} mechanism,
    + * this one requires the code generator to have generated a complete
    + * Java class that is capable of direct compilation and loading.
    + * This means the generated class must be a subclass of the template
    + * so that the JVM can use normal Java inheritance to associate the
    + * template and generated methods.
    + * <p>
    + * Here is how to use the plain-old Java technique to debug
    + * generated code:
    + * <ul>
    + * <li>Set the config option <var>drill.exec.compile.save_source</var>
    + * to <var>true</var>.</li>
    + * <li>Set the config option <var>drill.exec.compile.code_dir</var>
    + * to the location where you want to save the generated source
    + * code.</li>
    + * <li>Where you generate code (using a {@link CodeGenerator}),
    + * set the "plain-old Java" options:<pre>
    + * CodeGenerator&lt;Foo> cg = ...
    + * cg.plainOldJavaCapable(true); // Class supports plain-old Java
    + * cg.preferPlainOldJava(true); // Actually generate plain-old Java
    + * ...</pre></li>
    + * <li>In your favorite IDE, add to the code lookup path the
    + * code directory saved earlier. In Eclipse, for example, you do
    + * this in the debug configuration you will use to debug Drill.</li>
    + * <li>Set a breakpoint in template used for the generated code.</li>
    + * <li>Run Drill. The IDE will stop at your breakpoint.</li>
    + * <li>Step into the generated code. Examine class field and
    + * local variables. Have fun!</li>
    + * </ul>
    + * <p>
    + * Note: not all generated code is ready to be compiled as plain-old
    + * Java. Some classes omit from the template the proper <code>throws</code>
    + * declarations. Other minor problems may also crop up. All are easy
    + * to fix. Once you've done so, add the following to mark that you've
    + * done the clean-up:<pre>
    + * cg.plainOldJavaCapable(true); // Class supports plain-old Java</pre>
    + * <p>
    + * The setting to prefer plain-old Java is ignored for generated
    + * classes not marked as plain-old Java capable.
    + */
    +
    +public class ClassBuilder {
    +
    +  public static final String SAVE_CODE_OPTION = CodeCompiler.COMPILE_BASE + ".save_source";
    +  public static final String CODE_DIR_OPTION = CodeCompiler.COMPILE_BASE + ".code_dir";
    +
    +  private final DrillConfig config;
    +  private final OptionManager options;
    +  private final boolean saveCode;
    +  private final File codeDir;
    +
    +  public ClassBuilder(DrillConfig config, OptionManager optionManager) {
    +    this.config = config;
    +    options = optionManager;
    +
    +    // The option to save code is a boot-time option because
    +    // it is used selectively during debugging, but can cause
    +    // excessive I/O in a running server if used to save all code.
    +
    +    saveCode = config.getBoolean(SAVE_CODE_OPTION);
    +    codeDir = new File(config.getString(CODE_DIR_OPTION));
    +  }
    +
    +  /**
    +   * Given a code generator which has already generated plain-old Java
    +   * code, compile the code, create a class loader, and return the
    +   * resulting Java class.
    +   *
    +   * @param cg a plain-old Java capable code generator that has generated
    +   * plain-old Java code
    +   * @return the class that the code generator defines
    +   * @throws ClassTransformationException
    +   */
    +
    +  public Class<?> getImplementationClass(CodeGenerator<?> cg) throws ClassTransformationException {
    +    try {
    +      return compileClass(cg);
    +    } catch (CompileException | ClassNotFoundException|IOException e) {
    +      throw new ClassTransformationException(e);
    +    }
    +  }
    +
    +  /**
    +   * Performs the actual work of compiling the code and loading the class.
    +   *
    +   * @param cg the code generator that has built the class(es) to be generated.
    +   * @return the class, after code generation and (if needed) compilation.
    +   * @throws IOException if an error occurs when optionally writing code to disk.
    +   * @throws CompileException if the generated code has compile issues.
    +   * @throws ClassNotFoundException if the generated code references unknown classes.
    +   * @throws ClassTransformationException generic "something is wrong" error from
    +   * Drill class compilation code.
    +   */
    +  private Class<?> compileClass(CodeGenerator<?> cg) throws IOException, CompileException, ClassNotFoundException, ClassTransformationException {
    +
    +    // Get the plain-old Java code.
    +
    +    String code = cg.getGeneratedCode();
    +
    +    // Get the class names (dotted, file path, etc.)
    +
    +    String className = cg.getMaterializedClassName();
    +    ClassTransformer.ClassNames name = new ClassTransformer.ClassNames(className);
    +
    +    // A key advantage of this method is that the code can be
    +    // saved and debugged, if needed.
    +
    +    saveCode(code, name);
    +
    +    // Compile the code and load it into a class loader.
    +
    +    CachedClassLoader classLoader = new CachedClassLoader();
    --- End diff --
    
    Please see note above. Here is what we have:
    
    ```
    Global class cache --> class --> class loader
    ```
    The above pattern is the same in the byte-code fixup path and in the "plain old Java" path. The Class Loader is Java's way to get at the class.
    
    Why we have one per class? That is the original design and it was replicated here. An advantage is that we don't actually need to add a numeric class suffix (though we do anyway) as each class loader creates its own name space. 


> Option to debug generated Java code using an IDE
> ------------------------------------------------
>
>                 Key: DRILL-5052
>                 URL: https://issues.apache.org/jira/browse/DRILL-5052
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Execution - Codegen
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>            Priority: Minor
>              Labels: ready-to-commit
>
> Drill makes extensive use of Java code generation to implement its operators. Drill uses sophisticated techniques to blend generated code with pre-compiled template code. An unfortunate side-effect of this behavior is that it is very difficult to visualize and debug the generated code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself version of subclassing. The Drill "template" is the parent class, the generated code is the subclass. But, rather than using plain-old subclassing, Drill combines the code from the two classes into a single "artificial" packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" compilation: the generated code is a subclass of the template. Compile the generated code as a plain-old Java class with no byte-code fix-up. Write the code to a known location that the IDE can search when looking for source files.
> With this change, developers can turn on the above feature, set a breakpoint in a template, then step directly into the generated Java code called from the template.
> This feature should be an option, enabled by developers when needed. The existing byte-code technique should be used for production code generation.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)