hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-10319) Hive CLI startup takes a long time with a large number of databases
Date Thu, 16 Apr 2015 01:17:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497423#comment-14497423
] 

Hive QA commented on HIVE-10319:
--------------------------------



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12725625/HIVE-10319.patch

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3452/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3452/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3452/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-3452/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'metastore/scripts/upgrade/derby/hive-schema-1.2.0.derby.sql'
Reverted 'metastore/scripts/upgrade/derby/upgrade-1.1.0-to-1.2.0.derby.sql'
Reverted 'metastore/scripts/upgrade/oracle/hive-schema-1.2.0.oracle.sql'
Reverted 'metastore/scripts/upgrade/oracle/upgrade-1.1.0-to-1.2.0.oracle.sql'
Reverted 'metastore/scripts/upgrade/postgres/upgrade-1.1.0-to-1.2.0.postgres.sql'
Reverted 'metastore/scripts/upgrade/postgres/hive-schema-1.2.0.postgres.sql'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20S/target shims/0.23/target
shims/aggregator/target shims/common/target shims/scheduler/target packaging/target hbase-handler/target
testutils/target testutils/metastore/dbs/derby testutils/metastore/dbs/oracle testutils/metastore/dbs/postgres
jdbc/target metastore/target metastore/scripts/upgrade/derby/022-HIVE-10239.derby.sql metastore/scripts/upgrade/oracle/022-HIVE-10239.oracle.sql
metastore/scripts/upgrade/postgres/022-HIVE-10239.postgres.sql itests/target itests/thirdparty
itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target
itests/hive-minikdc/target itests/hive-jmh/target itests/hive-unit/target itests/custom-serde/target
itests/util/target itests/qtest-spark/target hcatalog/target hcatalog/core/target hcatalog/streaming/target
hcatalog/server-extensions/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target
hcatalog/hcatalog-pig-adapter/target accumulo-handler/target hwi/target common/target common/src/gen
spark-client/target service/target contrib/target serde/target beeline/target odbc/target
cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update
U    ql/src/test/results/clientnegative/udf_next_day_error_1.q.out
U    ql/src/test/results/clientnegative/udf_add_months_error_1.q.out
U    ql/src/test/results/clientnegative/udf_next_day_error_2.q.out
U    ql/src/test/results/clientnegative/udf_last_day_error_1.q.out
U    ql/src/test/results/clientpositive/spark/vector_elt.q.out
U    ql/src/test/results/clientpositive/spark/load_dyn_part14.q.out
U    ql/src/test/results/clientpositive/spark/join8.q.out
U    ql/src/test/results/clientpositive/spark/optimize_nullscan.q.out
U    ql/src/test/results/clientpositive/spark/auto_join8.q.out
U    ql/src/test/results/clientpositive/annotate_stats_select.q.out
U    ql/src/test/results/clientpositive/udf4.q.out
U    ql/src/test/results/clientpositive/udf_isnull_isnotnull.q.out
U    ql/src/test/results/clientpositive/decimal_udf.q.out
U    ql/src/test/results/clientpositive/udf_hour.q.out
U    ql/src/test/results/clientpositive/udf_if.q.out
U    ql/src/test/results/clientpositive/input8.q.out
U    ql/src/test/results/clientpositive/udf6.q.out
U    ql/src/test/results/clientpositive/udf_when.q.out
U    ql/src/test/results/clientpositive/udf_size.q.out
U    ql/src/test/results/clientpositive/udf_trunc.q.out
U    ql/src/test/results/clientpositive/tez/vector_coalesce.q.out
U    ql/src/test/results/clientpositive/tez/vector_decimal_udf.q.out
U    ql/src/test/results/clientpositive/tez/optimize_nullscan.q.out
U    ql/src/test/results/clientpositive/tez/vector_elt.q.out
U    ql/src/test/results/clientpositive/udf_nvl.q.out
U    ql/src/test/results/clientpositive/udf_second.q.out
U    ql/src/test/results/clientpositive/udf_greatest.q.out
U    ql/src/test/results/clientpositive/udf_parse_url.q.out
U    ql/src/test/results/clientpositive/vector_decimal_udf.q.out
U    ql/src/test/results/clientpositive/vector_elt.q.out
U    ql/src/test/results/clientpositive/udf_coalesce.q.out
U    ql/src/test/results/clientpositive/udf_minute.q.out
U    ql/src/test/results/clientpositive/load_dyn_part14.q.out
U    ql/src/test/results/clientpositive/num_op_type_conv.q.out
U    ql/src/test/results/clientpositive/vector_coalesce.q.out
U    ql/src/test/results/clientpositive/udf_case.q.out
U    ql/src/test/results/clientpositive/optimize_nullscan.q.out
U    ql/src/test/results/clientpositive/input9.q.out
U    ql/src/test/results/clientpositive/udf7.q.out
U    ql/src/test/results/clientpositive/udf_locate.q.out
U    ql/src/test/results/clientpositive/udf_instr.q.out
U    ql/src/test/results/clientpositive/udf_least.q.out
U    ql/src/test/results/clientpositive/ppd_constant_expr.q.out
U    ql/src/test/results/clientpositive/udf_elt.q.out
U    ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFNextDay.java
U    ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFLevenshtein.java
U    ql/src/test/queries/clientpositive/optimize_nullscan.q
U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcFactory.java
U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
U    ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/ConstantVectorExpression.java
U    ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java
U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDecode.java
U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSoundex.java
U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLastDay.java
U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCbrt.java
U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLevenshtein.java
U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFactorial.java
U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNextDay.java
U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAddMonths.java
U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMap.java
U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArray.java
U    ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrunc.java

Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1673986.

Updated to revision 1673986.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12725625 - PreCommit-HIVE-TRUNK-Build

> Hive CLI startup takes a long time with a large number of databases
> -------------------------------------------------------------------
>
>                 Key: HIVE-10319
>                 URL: https://issues.apache.org/jira/browse/HIVE-10319
>             Project: Hive
>          Issue Type: Improvement
>          Components: CLI
>    Affects Versions: 1.0.0
>            Reporter: Nezih Yigitbasi
>            Assignee: Nezih Yigitbasi
>         Attachments: HIVE-10319.patch
>
>
> The Hive CLI takes a long time to start when there is a large number of databases in
the DW. I think the root cause is the way permanent UDFs are loaded from the metastore. When
I looked at the logs and the source code I see that at startup Hive first gets all the databases
from the metastore and then for each database it makes a metastore call to get the permanent
functions for that database [see Hive.java | https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L162-185].
So the number of metastore calls made is in the order of the number of databases. In production
we have several hundreds of databases so Hive makes several hundreds of RPC calls during startup,
taking 30+ seconds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message