hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions
Date Wed, 24 Feb 2016 19:10:18 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163542#comment-15163542
] 

Hive QA commented on HIVE-13132:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12789357/HIVE-13132.1.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7081/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7081/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7081/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-7081/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
   74c2f06..f8f2014  branch-2.0 -> origin/branch-2.0
   402fabe..2f73233  master     -> origin/master
+ git reset --hard HEAD
HEAD is now at 402fabe HIVE-13064 : Fix passing serde properties from insert overwrite directory
(Rajat Khandelwal, reviewed by Amareshwari Sriramadasu)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 3 commits, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at 2f73233 HIVE-13128. NullScan fails on a secure setup. (Siddharth Seth, reviewed
by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12789357 - PreCommit-HIVE-TRUNK-Build

> Hive should lazily load and cache metastore (permanent) functions
> -----------------------------------------------------------------
>
>                 Key: HIVE-13132
>                 URL: https://issues.apache.org/jira/browse/HIVE-13132
>             Project: Hive
>          Issue Type: Improvement
>    Affects Versions: 0.13.1
>            Reporter: Anthony Hsu
>            Assignee: Anthony Hsu
>         Attachments: HIVE-13132.1.patch
>
>
> In Hive 0.13.1, we have noticed that as the number of databases increases, the start-up
time of the Hive interactive shell increases. This is because during start-up, all databases
are iterated over to fetch the permanent functions to display in the {{SHOW FUNCTIONS}} output.
> {noformat:title=FunctionRegistry.java}
>   private static Set<String> getFunctionNames(boolean searchMetastore) {
>     Set<String> functionNames = mFunctions.keySet();
>     if (searchMetastore) {
>       functionNames = new HashSet<String>(functionNames);
>       try {
>         Hive db = getHive();
>         List<String> dbNames = db.getAllDatabases();
>         for (String dbName : dbNames) {
>           List<String> funcNames = db.getFunctions(dbName, "*");
>           for (String funcName : funcNames) {
>             functionNames.add(FunctionUtils.qualifyFunctionName(funcName, dbName));
>           }
>         }
>       } catch (Exception e) {
>         LOG.error(e);
>         // Continue on, we can still return the functions we've gotten to this point.
>       }
>     }
>     return functionNames;
>   }
> {noformat}
> Instead of eagerly loading all metastore functions, we should only load them the first
time {{SHOW FUNCTIONS}} is invoked. We should also cache the results.
> Note that this issue may have been fixed by HIVE-2573, though I haven't verified this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message