drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-4726) Dynamic UDFs support
Date Thu, 08 Sep 2016 15:34:20 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15474180#comment-15474180
] 

ASF GitHub Bot commented on DRILL-4726:
---------------------------------------

Github user arina-ielchiieva commented on a diff in the pull request:

    https://github.com/apache/drill/pull/574#discussion_r78030875
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFunctionRegistry.java
---
    @@ -64,62 +76,134 @@
           .put("CONVERT_FROM", Pair.of(2, 2))
           .put("FLATTEN", Pair.of(1, 1)).build();
     
    +  /** Registers all functions present in Drill classpath on start-up. All functions will
be marked as built-in.*/
       public DrillFunctionRegistry(ScanResult classpathScan) {
    +    validate(BUILT_IN, classpathScan);
    +    register(BUILT_IN, classpathScan, this.getClass().getClassLoader());
    +    if (logger.isTraceEnabled()) {
    +      StringBuilder allFunctions = new StringBuilder();
    +      for (DrillFuncHolder method: registryHolder.getAllFunctionsWithHolders().values())
{
    +        allFunctions.append(method.toString()).append("\n");
    +      }
    +      logger.trace("Registered functions: [\n{}]", allFunctions);
    +    }
    +  }
    +
    +  /**
    +   * Validates all functions, present in jars.
    +   * Will throw {@link FunctionValidationException} if:
    +   * 1. Jar with the same name has been already registered.
    +   * 2. Conflicting function with the similar signature is found.
    +   * 3. Aggregating function is not deterministic.
    +   *
    +   * @return list of validated functions
    +   */
    +  public List<Func> validate(String jarName, ScanResult classpathScan) {
    +    List<Func> functions = Lists.newArrayList();
         FunctionConverter converter = new FunctionConverter();
         List<AnnotatedClassDescriptor> providerClasses = classpathScan.getAnnotatedClasses();
     
    -    // Hash map to prevent registering functions with exactly matching signatures
    -    // key: Function Name + Input's Major Type
    -    // value: Class name where function is implemented
    -    //
    -    final Map<String, String> functionSignatureMap = new HashMap<>();
    +    if (registryHolder.containsJar(jarName)) {
    +      throw new FunctionValidationException(String.format("Jar %s is already registered",
jarName));
    +    }
    +
    +    final ListMultimap<String, String> allFuncWithSignatures = registryHolder.getAllFunctionsWithSignatures();
    +
         for (AnnotatedClassDescriptor func : providerClasses) {
    -      DrillFuncHolder holder = converter.getHolder(func);
    +      DrillFuncHolder holder = converter.getHolder(func, ClassLoader.getSystemClassLoader());
           if (holder != null) {
    -        // register handle for each name the function can be referred to
    -        String[] names = holder.getRegisteredNames();
     
    -        // Create the string for input types
             String functionInput = "";
    +        List<MajorType> types = Lists.newArrayList();
             for (DrillFuncHolder.ValueReference ref : holder.parameters) {
               functionInput += ref.getType().toString();
    +          types.add(ref.getType());
             }
    +
    +        String[] names = holder.getRegisteredNames();
             for (String name : names) {
               String functionName = name.toLowerCase();
    -          registeredFunctions.put(functionName, holder);
               String functionSignature = functionName + functionInput;
    -          String existingImplementation;
    -          if ((existingImplementation = functionSignatureMap.get(functionSignature))
!= null) {
    -            throw new AssertionError(
    -                String.format(
    -                    "Conflicting functions with similar signature found. Func Name: %s,
Class name: %s " +
    -                " Class name: %s", functionName, func.getClassName(), existingImplementation));
    -          } else if (holder.isAggregating() && !holder.isDeterministic() ) {
    -            logger.warn("Aggregate functions must be deterministic, did not register
function {}", func.getClassName());
    +
    +          if (allFuncWithSignatures.get(functionName).contains(functionSignature)) {
    --- End diff --
    
    Explained in the comment above.


> Dynamic UDFs support
> --------------------
>
>                 Key: DRILL-4726
>                 URL: https://issues.apache.org/jira/browse/DRILL-4726
>             Project: Apache Drill
>          Issue Type: New Feature
>    Affects Versions: 1.6.0
>            Reporter: Arina Ielchiieva
>            Assignee: Arina Ielchiieva
>             Fix For: Future
>
>
> Allow register UDFs without  restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message