drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5290) Provide an option to build operator table once for built-in static functions and reuse it across queries.
Date Thu, 23 Feb 2017 00:56:44 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15879627#comment-15879627

ASF GitHub Bot commented on DRILL-5290:

Github user ppadma commented on a diff in the pull request:

    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
    @@ -413,4 +413,8 @@
       String DYNAMIC_UDF_SUPPORT_ENABLED = "exec.udf.enable_dynamic_support";
true, true);
    +  String USE_DYNAMIC_UDFS = "exec.udf.use_dynamic";
    --- End diff --
    Currently, we have one FunctionRegistryHolder (LocalFunctionRegistry) for both static
and dynamic functions, from which we register for each query.  It gets updated when we download
new jars/functions from zookeeper and will not be same as what it is during startup if dynamic
UDF support is enabled and disabled.  That means if I use table I built during startup, it
will miss the dynamic UDFs that got added in between and to include them, you have to rebuild
the table.  For that reason, I choose to  add a new option. 

> Provide an option to build operator table once for built-in static functions and reuse
it across queries.
> ---------------------------------------------------------------------------------------------------------
>                 Key: DRILL-5290
>                 URL: https://issues.apache.org/jira/browse/DRILL-5290
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.9.0
>            Reporter: Padma Penumarthy
>            Assignee: Padma Penumarthy
>             Fix For: 1.10
> Currently, DrillOperatorTable which contains standard SQL operators and functions and
Drill User Defined Functions (UDFs) (built-in and dynamic) gets built for each query as part
of creating QueryContext. This is an expensive operation ( ~30 msec to build) and allocates
 ~2M on heap for each query. For high throughput, low latency operational queries, we quickly
run out of heap memory, causing JVM hangs. Build operator table once during startup for static
built-in functions and save in DrillbitContext, so we can reuse it across queries.
> Provide a system/session option to not use dynamic UDFs so we can use the operator table
saved in DrillbitContext and avoid building each time.

This message was sent by Atlassian JIRA

View raw message