drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5052) Option to debug generated Java code using an IDE
Date Mon, 19 Dec 2016 21:57:58 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762418#comment-15762418
] 

ASF GitHub Bot commented on DRILL-5052:
---------------------------------------

Github user jinfengni commented on a diff in the pull request:

    https://github.com/apache/drill/pull/660#discussion_r93123021
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/compile/ClassBuilder.java
---
    @@ -0,0 +1,189 @@
    +/**
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.drill.exec.compile;
    +
    +import java.io.File;
    +import java.io.FileWriter;
    +import java.io.IOException;
    +import java.util.Map;
    +
    +import org.apache.drill.common.config.DrillConfig;
    +import org.apache.drill.exec.compile.ClassTransformer.ClassNames;
    +import org.apache.drill.exec.exception.ClassTransformationException;
    +import org.apache.drill.exec.expr.CodeGenerator;
    +import org.apache.drill.exec.server.options.OptionManager;
    +import org.codehaus.commons.compiler.CompileException;
    +
    +/**
    + * Implements the "plain-old Java" method of code generation and
    + * compilation. Given a {@link CodeGenerator}, obtains the generated
    + * source code, compiles it with the selected compiler, loads the
    + * byte-codes into a class loader and provides the resulting
    + * class. Compared with the {@link ClassTransformer} mechanism,
    + * this one requires the code generator to have generated a complete
    + * Java class that is capable of direct compilation and loading.
    + * This means the generated class must be a subclass of the template
    + * so that the JVM can use normal Java inheritance to associate the
    + * template and generated methods.
    + * <p>
    + * Here is how to use the plain-old Java technique to debug
    + * generated code:
    + * <ul>
    + * <li>Set the config option <var>drill.exec.compile.save_source</var>
    + * to <var>true</var>.</li>
    + * <li>Set the config option <var>drill.exec.compile.code_dir</var>
    + * to the location where you want to save the generated source
    + * code.</li>
    + * <li>Where you generate code (using a {@link CodeGenerator}),
    + * set the "plain-old Java" options:<pre>
    + * CodeGenerator&lt;Foo> cg = ...
    + * cg.plainOldJavaCapable(true); // Class supports plain-old Java
    + * cg.preferPlainOldJava(true); // Actually generate plain-old Java
    + * ...</pre></li>
    + * <li>In your favorite IDE, add to the code lookup path the
    + * code directory saved earlier. In Eclipse, for example, you do
    + * this in the debug configuration you will use to debug Drill.</li>
    + * <li>Set a breakpoint in template used for the generated code.</li>
    + * <li>Run Drill. The IDE will stop at your breakpoint.</li>
    + * <li>Step into the generated code. Examine class field and
    + * local variables. Have fun!</li>
    + * </ul>
    + * <p>
    + * Note: not all generated code is ready to be compiled as plain-old
    + * Java. Some classes omit from the template the proper <code>throws</code>
    + * declarations. Other minor problems may also crop up. All are easy
    + * to fix. Once you've done so, add the following to mark that you've
    + * done the clean-up:<pre>
    + * cg.plainOldJavaCapable(true); // Class supports plain-old Java</pre>
    + * <p>
    + * The setting to prefer plain-old Java is ignored for generated
    + * classes not marked as plain-old Java capable.
    + */
    +
    +public class ClassBuilder {
    +
    +  public static final String SAVE_CODE_OPTION = CodeCompiler.COMPILE_BASE + ".save_source";
    +  public static final String CODE_DIR_OPTION = CodeCompiler.COMPILE_BASE + ".code_dir";
    +
    +  private final DrillConfig config;
    +  private final OptionManager options;
    +  private final boolean saveCode;
    +  private final File codeDir;
    +
    +  public ClassBuilder(DrillConfig config, OptionManager optionManager) {
    +    this.config = config;
    +    options = optionManager;
    +
    +    // The option to save code is a boot-time option because
    +    // it is used selectively during debugging, but can cause
    +    // excessive I/O in a running server if used to save all code.
    +
    +    saveCode = config.getBoolean(SAVE_CODE_OPTION);
    +    codeDir = new File(config.getString(CODE_DIR_OPTION));
    +  }
    +
    +  /**
    +   * Given a code generator which has already generated plain-old Java
    +   * code, compile the code, create a class loader, and return the
    +   * resulting Java class.
    +   *
    +   * @param cg a plain-old Java capable code generator that has generated
    +   * plain-old Java code
    +   * @return the class that the code generator defines
    +   * @throws ClassTransformationException
    +   */
    +
    +  public Class<?> getImplementationClass(CodeGenerator<?> cg) throws ClassTransformationException
{
    +    try {
    +      return compileClass(cg);
    +    } catch (CompileException | ClassNotFoundException|IOException e) {
    +      throw new ClassTransformationException(e);
    +    }
    +  }
    +
    +  /**
    +   * Performs the actual work of compiling the code and loading the class.
    +   *
    +   * @param cg the code generator that has built the class(es) to be generated.
    +   * @return the class, after code generation and (if needed) compilation.
    +   * @throws IOException if an error occurs when optionally writing code to disk.
    +   * @throws CompileException if the generated code has compile issues.
    +   * @throws ClassNotFoundException if the generated code references unknown classes.
    +   * @throws ClassTransformationException generic "something is wrong" error from
    +   * Drill class compilation code.
    +   */
    +  private Class<?> compileClass(CodeGenerator<?> cg) throws IOException,
CompileException, ClassNotFoundException, ClassTransformationException {
    +
    +    // Get the plain-old Java code.
    +
    +    String code = cg.getGeneratedCode();
    +
    +    // Get the class names (dotted, file path, etc.)
    +
    +    String className = cg.getMaterializedClassName();
    +    ClassTransformer.ClassNames name = new ClassTransformer.ClassNames(className);
    +
    +    // A key advantage of this method is that the code can be
    +    // saved and debugged, if needed.
    +
    +    saveCode(code, name);
    +
    +    // Compile the code and load it into a class loader.
    +
    +    CachedClassLoader classLoader = new CachedClassLoader();
    +    ClassCompilerSelector compilerSelector = new ClassCompilerSelector(classLoader, config,
options);
    +    Map<String,byte[]> results = compilerSelector.compile(name, code);
    +    classLoader.addClasses(results);
    +
    +    // Get the class from the class loader.
    +
    +    try {
    +      return classLoader.findClass(className);
    --- End diff --
    
    Per the comment above, this findClass() call will always end up with  cache-miss?



> Option to debug generated Java code using an IDE
> ------------------------------------------------
>
>                 Key: DRILL-5052
>                 URL: https://issues.apache.org/jira/browse/DRILL-5052
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Execution - Codegen
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>            Priority: Minor
>              Labels: ready-to-commit
>
> Drill makes extensive use of Java code generation to implement its operators. Drill uses
sophisticated techniques to blend generated code with pre-compiled template code. An unfortunate
side-effect of this behavior is that it is very difficult to visualize and debug the generated
code.
> As it turns out, Drill's code-merge facility is, in essence, a do-it-yourself version
of subclassing. The Drill "template" is the parent class, the generated code is the subclass.
But, rather than using plain-old subclassing, Drill combines the code from the two classes
into a single "artificial" packet of byte codes for which no source exists.
> Modify the code generation path to optionally allow "plain-old Java" compilation: the
generated code is a subclass of the template. Compile the generated code as a plain-old Java
class with no byte-code fix-up. Write the code to a known location that the IDE can search
when looking for source files.
> With this change, developers can turn on the above feature, set a breakpoint in a template,
then step directly into the generated Java code called from the template.
> This feature should be an option, enabled by developers when needed. The existing byte-code
technique should be used for production code generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message