hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8456) Support Hive Counter to collect spark job metric[Spark Branch]
Date Tue, 14 Oct 2014 09:54:33 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170733#comment-14170733
] 

Hive QA commented on HIVE-8456:
-------------------------------



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12674734/HIVE-8456.1-spark.patch

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/215/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/215/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-215/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-SPARK-Build-215/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-spark-source ]]
+ [[ ! -d apache-svn-spark-source/.svn ]]
+ [[ ! -d apache-svn-spark-source ]]
+ cd apache-svn-spark-source
+ svn revert -R .
Reverted 'itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java'
Reverted 'ql/src/test/results/clientpositive/spark/parallel.q.out'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageInfo.java'
++ svn status --no-ignore
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
+ rm -rf lr target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target
shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target
hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/lr
itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target
itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target
itests/qtest-spark/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target
accumulo-handler/target hwi/target common/target common/src/gen contrib/target service/target
serde/target beeline/target cli/target odbc/target ql/dependency-reduced-pom.xml ql/target
+ svn update
U    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java
U    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java
A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status
A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobStatus.java
A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl
A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/JobStateListener.java
A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/SimpleSparkJobStatus.java
A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java
A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkProgress.java
A    ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobState.java

Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1631702.

Updated to revision 1631702.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
patch: **** malformed patch at line 297:  

patch: **** malformed patch at line 297:  

patch: **** malformed patch at line 297:  

The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12674734

> Support Hive Counter to collect spark job metric[Spark Branch]
> --------------------------------------------------------------
>
>                 Key: HIVE-8456
>                 URL: https://issues.apache.org/jira/browse/HIVE-8456
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Chengxiang Li
>            Assignee: Chengxiang Li
>              Labels: Spark-M3
>         Attachments: HIVE-8456.1-spark.patch
>
>
> Several Hive query metric in Hive operators is collected by Hive Counter, such as CREATEDFILES
and DESERIALIZE_ERRORS, Besides, Hive use Counter as an option to collect table stats info.
 Spark support Accumulator which is pretty similiar with Hive Counter, we could try to enable
Hive Counter based on it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message