hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-6579) HiveLockObjectData constructor makes too many queryStr instance causing oom
Date Sat, 08 Mar 2014 18:48:49 GMT

    [ https://issues.apache.org/jira/browse/HIVE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924972#comment-13924972
] 

Hive QA commented on HIVE-6579:
-------------------------------



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12633317/HIVE-6579.1.patch.txt

Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1663/testReport
Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1663/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1663/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java'
Reverted 'hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java'
Reverted 'hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java'
Reverted 'hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java'
Reverted 'hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java'
Reverted 'hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java'
Reverted 'hbase-handler/pom.xml'
Reverted 'itests/util/pom.xml'
Reverted 'serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObjectBase.java'
Reverted 'serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObject.java'
Reverted 'serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java'
Reverted 'serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java'
Reverted 'serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarStructBase.java'
Reverted 'serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryObject.java'
Reverted 'serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStoragePredicateHandler.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/index/IndexSearchCondition.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target
shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target
hbase-handler/target hbase-handler/src/test/results/positive/hbase_custom_key.q.out hbase-handler/src/test/results/positive/hbase_custom_key2.q.out
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory.java hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory2.java
hbase-handler/src/test/queries/positive/hbase_custom_key.q hbase-handler/src/test/queries/positive/hbase_custom_key2.q
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseLazyObjectFactory.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseScanRange.java
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseWritableKeyFactory.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseKeyFactory.java
testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target
itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target
hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target
hwi/target common/target common/src/gen service/target contrib/target serde/target serde/src/java/org/apache/hadoop/hive/serde2/StructObjectBaseInspector.java
serde/src/java/org/apache/hadoop/hive/serde2/StructObject.java beeline/target odbc/target
cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update
U    ql/src/test/queries/clientpositive/subquery_multiinsert.q
U    ql/src/test/results/clientpositive/correlationoptimizer6.q.out
U    ql/src/test/results/clientpositive/auto_join12.q.out
U    ql/src/test/results/clientpositive/subquery_multiinsert.q.out
U    ql/src/test/results/clientpositive/auto_join_without_localtask.q.out
U    ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out
U    ql/src/test/results/clientpositive/multiMapJoin2.q.out
U    ql/src/test/results/clientpositive/mapjoin_hook.q.out
U    ql/src/test/results/clientpositive/auto_join3.q.out
U    ql/src/test/results/clientpositive/auto_join16.q.out
U    ql/src/test/results/clientpositive/auto_join25.q.out
U    ql/src/test/results/clientpositive/auto_join0.q.out
U    ql/src/test/results/clientpositive/mapjoin_test_outer.q.out
U    ql/src/test/results/clientpositive/correlationoptimizer4.q.out
U    ql/src/test/results/clientpositive/auto_join29.q.out
U    ql/src/test/results/clientpositive/auto_join10.q.out
U    ql/src/test/results/clientpositive/join35.q.out
U    ql/src/test/results/clientpositive/index_auto_update.q.out
U    ql/src/test/results/clientpositive/infer_bucket_sort_convert_join.q.out
U    ql/src/test/org/apache/hadoop/hive/ql/plan/TestConditionalResolverCommonJoin.java
U    ql/src/java/org/apache/hadoop/hive/ql/Driver.java
U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java
U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java
U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MapJoinResolver.java
U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java
U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java
U    ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationOptimizer.java
U    ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverCommonJoin.java
U    ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java
U    ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
U    ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java
U    ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java
U    ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
U    ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java

Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1575587.

Updated to revision 1575587.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12633317

> HiveLockObjectData constructor makes too many queryStr instance causing oom
> ---------------------------------------------------------------------------
>
>                 Key: HIVE-6579
>                 URL: https://issues.apache.org/jira/browse/HIVE-6579
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: xieyuchen
>         Attachments: HIVE-6579.1.patch.txt
>
>
> We have a huge sql which full outer joins 10+ partitoned tables, each table has at least
1k partitions. The sql has 300kb in length(it constructed  automatically of cause).
> So when we running this sql, there are over 10k HiveLockObjectData instances. Because
of the constructor of HiveLockObjectData trim the queryStr, there will be 10k individual String
instances, each has 300kb in length! Then the Hive client will get an oom exception.
> Trying to trim the queryStr in Driver.compile function instead of doing it in HiveLockObjectData
constructor to reduce memory wasting.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message