hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Dere" <jd...@hortonworks.com>
Subject Re: Review Request 26854: HIVE-2573 Create per-session function registry
Date Fri, 31 Oct 2014 07:44:18 GMT


> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 465
> > <https://reviews.apache.org/r/26854/diff/1-3/?file=723909#file723909line465>
> >
> >     There is no longer a way to query the metastore for UDFs apart from the static
initialization. So if one CLI user creates a permanent UDF, another user on CLI, or HS2, will
not be able to use that new UDF if the 2nd CLI or HS2 was initialized before this UDF was
created.
> 
> Navis Ryu wrote:
>     Permanent functions (persistent function seemed better name, imho) are registered
to system registry, which is shared to all clients. So if one user creates new permanent function,
it's shared to all clients. The time a user accesses the function, the class is loaded with
required resources and registered to session registry as a temporary function.
> 
> Jason Dere wrote:
>     So this would work if all clients are using hiveserver2, because all clients in this
scenario would share the same system registry.
>     But if one or more clients are using the Hive CLI, any persistent UDFs created/dropped
by this CLI client would not be reflected in the other clients (or HS2), since it's a different
process/system registry.
> 
> Navis Ryu wrote:
>     I've missed this message. Yes, it will act like you've commented. But is it a common
case to expect things done in separate VM refleceted to others? Should dropping presistent
function by a jdbc client be reflected to other CLI clients?

Would be good to have if it's possible. Certainly if one client creates a table, we would
want the other client to be able to see it. Being able to have the same behavior with functions
seems consistent. I suppose it is a bit harder for this to work in the case of drop function,
unless we are looking up the UDF in the metastore each time it is used in a query.


- Jason


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57952
-----------------------------------------------------------


On Oct. 30, 2014, 11:41 p.m., Jason Dere wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
> 
> (Updated Oct. 30, 2014, 11:41 p.m.)
> 
> 
> Review request for hive, Navis Ryu and Thejas Nair.
> 
> 
> Bugs: HIVE-2573
>     https://issues.apache.org/jira/browse/HIVE-2573
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed
from registry
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7d8e5bc 
>   contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3 
>   itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
6647ce5 
>   metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
88b0791 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e43a328 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bdeb92 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4e3df75 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java e2768ff 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java
793f117 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1796b7b 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052

>   ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb 
>   ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed

>   ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a 
>   ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5 
>   ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787 
>   service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907

> 
> Diff: https://reviews.apache.org/r/26854/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Jason Dere
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message