drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-4963) Issues when overloading Drill native functions with dynamic UDFs
Date Sat, 04 Feb 2017 00:01:51 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852368#comment-15852368

ASF GitHub Bot commented on DRILL-4963:

Github user paul-rogers commented on the issue:

    As I understand it, the issues are these:
    * When parsing/planning a query, function references are ambiguous.
    * If a function x() simply does not exist at all, we get a clear signal and can check
the function registry for updates.
    * When names are overridden, there may be many x(arg1) functions: some defined locally,
some recently added to the registry.
    * In this case, we get no clear signal that we should check the registry since we might
find a good-enough match locally and not know to check for a better match in the registry.
    The solution is to check the registry version on each new query. This must be done for
every query (with functions) since we can never be certain whether an override exists in the
    The problem is that a check of ZK, even to get a version, adds latency. 99.9% of the time,
nothing will have changed. How can we fix that?
    Let's take a step back. The original problems we tried to solve were:
    If a user executes a CREATE FUNCTION x on Drillbit A and the command returns successfully,
    * If the same user immediately executes a query using x on the same Drillbit, that query
    * If some other user executes a query using x (on any Drillbit, say Drillbit B), then
the query either fails (x is not found) or succeeds (x is found on Drillbit B and on all Drillbits
that run fragments.)
    In general, the above requires lots of synchronization: we'd want every Drillbit to synchronize
with the registry before every query parse and fragment execution. We know that is expensive.
So, we looked for special case opportunities to exploit. The "found/not found" semantics above
appeared to provide that special case. What that trick off the table, we are back to the massive
synchronization solution, which is costly.
    We want to keep the semantics listed above, but without the cost of synchronization.
    Just tossing out ideas (I'm not (yet) proposing changes), perhaps:
    * Each Drillbit maintains a cache of the most recent ZK registry version it has seen.
    * When a Foreman registers a function, it updates the ZK registry and its locally-cached
version number.
    * When a Foreman runs a query, it includes the registry version number in the physical
plan sent to each Drillbit for execution.
    * The Drillbit checks the plan's version number against its cached version. If the plan
version is newer, the Drillbit downloads the latest registry data and updates its cached version
    The above ensures that, if a function is registered on Drillbit A, then any query that
is submitted to A will execute anywhere on the cluster. So far so good.
    But, what about queries submitted to Drillbits B, C and D? How do they learn about the
new functions? Here, perhaps we can use an eventually consistent pattern. The other Drillbits
listen for ZK change notifications and refresh their local registries from ZK, and update
the locally cached version number.
    Now, we get the semantics that if a function is defined on Drillbit A, a brief time later
it will be available on Drillbits B, C and D. Once available, the above rules apply: the registry
version is written into the plan and all other fragment executors will force update their
local registries if they haven't yet gotten the ZK update notices.
    The only drawback is a slight delay in a function becoming available to the cluster. But,
that delay is fine in an eventually consistent design. Said another way, we make no transaction
guarantees that updating something on Drillbit A will be immediately reflected on B, etc.
    This is a rough draft. It may be what the code does. I'll continue to review and revise
the note as I learn more.

> Issues when overloading Drill native functions with dynamic UDFs
> ----------------------------------------------------------------
>                 Key: DRILL-4963
>                 URL: https://issues.apache.org/jira/browse/DRILL-4963
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Functions - Drill
>    Affects Versions: 1.9.0
>            Reporter: Roman
>            Assignee: Arina Ielchiieva
>             Fix For: Future
>         Attachments: subquery_udf-1.0.jar, subquery_udf-1.0-sources.jar, test_overloading-1.0.jar,
> I created jar file which overloads 3 DRILL native functions (LOG(VARCHAR-REQUIRED), CURRENT_DATE(VARCHAR-REQUIRED)
and ABS(VARCHAR-REQUIRED,VARCHAR-REQUIRED)) and registered it as dynamic UDF.
> If I try to use my functions I will get errors:
> {code:xml}
> {code}
> Error: FUNCTION ERROR: CURRENT_DATE does not support operand types (CHAR)
> SQL Query null
> {code:xml}
> SELECT ABS('test','test') FROM (VALUES(1));
> {code}
> Error: FUNCTION ERROR: ABS does not support operand types (CHAR,CHAR)
> SQL Query null
> {code:xml}
> SELECT LOG('test') FROM (VALUES(1));
> {code}
> Error: SYSTEM ERROR: DrillRuntimeException: Failure while materializing expression in
constant expression evaluator LOG('test').  Errors: 
> Error in expression at index -1.  Error: Missing function implementation: castTINYINT(VARCHAR-REQUIRED).
 Full expression: UNKNOWN EXPRESSION.
> But if I rerun all this queries after "DrillRuntimeException", they will run correctly.
It seems that Drill have not updated the function signature before that error. Also if I add
jar as usual UDF (copy jar to /drill_home/jars/3rdparty and restart drillbits), all queries
will run correctly without errors.

This message was sent by Atlassian JIRA

View raw message