drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From arina-ielchiieva <...@git.apache.org>
Subject [GitHub] drill pull request #574: DRILL-4726: Dynamic UDFs support
Date Mon, 05 Sep 2016 10:05:22 GMT
Github user arina-ielchiieva commented on a diff in the pull request:

    --- Diff: protocol/src/main/protobuf/UserBitShared.proto ---
    @@ -298,3 +298,17 @@ enum CoreOperatorType {
       NESTED_LOOP_JOIN = 35;
       AVRO_SUB_SCAN = 36;
    +message Func {
    +  optional string name = 1;
    +  repeated common.MajorType major_type = 2;
    +message Jar {
    +  optional string name = 1;
    +  repeated Func function = 2;
    +message Registry {
    --- End diff --
    Registry is stored at Zookeeper. Unfortunately, we can't consider different structure,
since each time we try to register UDF we need all registry to validate against it and since
we don't impose any locks on registry during validation, we use ZK versioning feature. Basically,
we take registry from ZK with its version, validate against it, update it and try to store
updated version in ZK. If registry version has changed by that time, we re-validate again
(see 5.1.3 Registration). Versioning only works for object stored under znode but not on child
znodes. I have tried to add in remote registry only required information to reduce it's size
(jar name, function name, list of in parameters). Eventually, the size of registry might grow
too big, in this case we expect user periodically move verified dynamic UDFs to static UDFs,
remove unused UDFs.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message