hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "cheng xu" <cheng.a...@intel.com>
Subject Re: Review Request 24986: HIVE-7553: decouple the auxiliary jar loading from hive server2 starting phase
Date Tue, 26 Aug 2014 08:55:29 GMT


> On Aug. 25, 2014, 8:22 p.m., Brock Noland wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java, line 1916
> > <https://reviews.apache.org/r/24986/diff/3/?file=668171#file668171line1916>
> >
> >     Let's put some trace logging in here as which classloader we are returning.
> >     
> >     I don't see the classloader on Conf actually getting set anywhere. Is it set
by someone for us?

The classloader was set in the method refreshAuxJars in the class session state or the method
removeFromClassPath in the class Utilities. Besides that, it was initialized in the static
block from the HiveConf.


> On Aug. 25, 2014, 8:22 p.m., Brock Noland wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java, line 749
> > <https://reviews.apache.org/r/24986/diff/3/?file=668188#file668188line749>
> >
> >     I think that HIVEREFRESHJARS  should be a list of directories like HIVEAUXJARS

Pretty good for figuring it out. Maybe we can follow what hive_aux_jar_path was handled. You
can see that there are two main cases to be handled: 1. hive_aux_jar_path was a directory
2. hive_aux_jar_path consists of several jar files. Attached are the lines of code.

# add the auxillary jars such as serdes
if [ -d "${HIVE_AUX_JARS_PATH}" ]; then
  for f in ${HIVE_AUX_JARS_PATH}/*.jar; do
    if [[ ! -f $f ]]; then
        continue;
    fi
    if $cygwin; then
	f=`cygpath -w "$f"`
    fi
    AUX_CLASSPATH=${AUX_CLASSPATH}:$f
    if [ "${AUX_PARAM}" == "" ]; then
        AUX_PARAM=file://$f
    else
        AUX_PARAM=${AUX_PARAM},file://$f;
    fi
  done
elif [ "${HIVE_AUX_JARS_PATH}" != "" ]; then 
  if $cygwin; then
      HIVE_AUX_JARS_PATH=`echo $HIVE_AUX_JARS_PATH | sed 's/,/:/g'`
      HIVE_AUX_JARS_PATH=`cygpath -p -w "$HIVE_AUX_JARS_PATH"`
      HIVE_AUX_JARS_PATH=`echo $HIVE_AUX_JARS_PATH | sed 's/;/,/g'`
  fi
  AUX_CLASSPATH=${HIVE_AUX_JARS_PATH}
  AUX_PARAM=file://${HIVE_AUX_JARS_PATH}
  AUX_PARAM=`echo $AUX_PARAM | sed 's/,/,file:\/\//g'`
fi

# adding jars from auxlib directory
for f in ${HIVE_HOME}/auxlib/*.jar; do
  if [[ ! -f $f ]]; then
      continue;
  fi
  if $cygwin; then
      f=`cygpath -w "$f"`
  fi
  AUX_CLASSPATH=${AUX_CLASSPATH}:$f
  if [ "${AUX_PARAM}" == "" ]; then
    AUX_PARAM=file://$f
  else
    AUX_PARAM=${AUX_PARAM},file://$f;
  fi
done
if $cygwin; then
    CLASSPATH=`cygpath -p -w "$CLASSPATH"`
    CLASSPATH=${CLASSPATH};${AUX_CLASSPATH}
else
    CLASSPATH=${CLASSPATH}:${AUX_CLASSPATH}
fi


- cheng


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24986/#review51436
-----------------------------------------------------------


On Aug. 25, 2014, 6:45 a.m., cheng xu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24986/
> -----------------------------------------------------------
> 
> (Updated Aug. 25, 2014, 6:45 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-7553
>     https://issues.apache.org/jira/browse/HIVE-7553
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-7553: decouple the auxiliary jar loading from hive server2 starting phase
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f4afd9d64aff18329e7850342855aade42e21f5

>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 93a03adeab7ba3c3c91344955d303e4252005239

>   hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatClient.java
f25039dcf55b3b24bbf8dcba05855665a1c7f3b0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DefaultFetchFormatter.java 5924bcf1f55dc4c2dd06f312f929047b7df9de55

>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 0c6a3d44ef1f796778768421dc02f8bf3ede6a8c

>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java bd45df1a401d1adb009e953d08205c7d5c2d5de2

>   ql/src/java/org/apache/hadoop/hive/ql/exec/ListSinkOperator.java dcc19f70644c561e17df8c8660ca62805465f1d6

>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 76fee612a583cdc2c632d27932623521b735e768

>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java eb2851b2c5fa52e0f555b3d8d1beea5d1ac3b225

>   ql/src/java/org/apache/hadoop/hive/ql/hooks/HookUtils.java 3f474f846c7af5f1f65f1c14f3ce51308f1279d4

>   ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java 0962cadce0d515e046371d0a816f4efd70b8eef7

>   ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java 9051ba6d80e619ddbb6c27bb161e1e7a5cdb08a5

>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java edec1b734fb2f015902fd5e1c8afd5acdf4cb3bf

>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 2f13ac2e30195a25844a25e9ec8a7c42ed99b75c

>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java b15aedc15d8cd0979aced6ff4c9e87606576f0a3

>   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java d86df453cd7686627940ade62c0fd72f1636dd0b

>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java 0a1c660b4bbd46d8410e646270b23c99a4de8b7e

>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b05d3b48ec014e4dc8026bb5f6615f62da0e2210

>   ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java 17eeae1a3435fceb4b57325675c58b599e0973ea

>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 930acbc98e81f8d421cee1170659d8b7a427fe7d

>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 39f1793aaa5bed8a494883cac516ad314be951f4

>   ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 0d237f01a248a65b4092eb7202fe30eebf27be82

>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java f5bc427a5834860441f21bfc72e175c6a1cf877f

>   ql/src/java/org/apache/hadoop/hive/ql/processors/RefreshProcessor.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 9798cf3f537a27d1f828f8139790c62c5945c366

>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java e247184b7d95c85fd3e12432e7eb75eb1e2a0b68

>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 959007a54b335bb0bdef0256f60e6cbc65798dc7

>   ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java ef0052f5763922d50986f127c416af5eaa6ae30d

>   ql/src/test/resources/SessionStateTest-V1.jar PRE-CREATION 
>   ql/src/test/resources/SessionStateTest-V2.jar PRE-CREATION 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java bc0a02c1df7f9fdb848d5f078e94a663a579e571

> 
> Diff: https://reviews.apache.org/r/24986/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> cheng xu
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message