Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Sat, 20 Jul 2013 16:31:09 +0000 (UTC)
From: Apache Jenkins Server <jenkins@builds.apache.org>
To: dev@hive.apache.org, brock@apache.org, ashutosh.chauhan@gmail.com,
	gates@hortonworks.com
Message-ID: <1909530361.996.1374337872036.JavaMail.hudson@aegis>
In-Reply-To: <189698274.880.1374295626776.JavaMail.hudson@aegis>
References: <189698274.880.1374295626776.JavaMail.hudson@aegis>
Subject: Hive-trunk-h0.21 - Build # 2209 - Still Failing
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----=_Part_995_528266177.1374337869460"

------=_Part_995_528266177.1374337869460
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Changes for Build #2169

Changes for Build #2170
[hashutosh] HIVE-4740 : HIVE2379 is missing hbase.jar itself (Navis via Ash=
utosh Chauhan)

[hashutosh] HIVE-4756 : Upgrade Hadoop 0.23 profile to 2.0.5-alpha (Brock N=
oland via Ashutosh Chauhan)

[hashutosh] HIVE-4791 : improve test coverage of package org.apache.hadoop.=
hive.ql.udf.xml (Ivan Veselovsky via Ashutosh Chauhan)


Changes for Build #2171
[hashutosh] HIVE-4618 : show create table creating unusable DDL when field =
delimiter is \001 (Navis via Ashutosh Chauhan)

[hashutosh] HIVE-4559 : hcatalog/webhcat scripts in tar.gz don't have execu=
te permissions set (Eugene Koifman via Ashutosh Chauhan)

[hashutosh] HIVE-4798 : NPE when we call isSame from an instance of ExprNod=
eConstantDesc with null value (Yin Huai via Ashutosh Chauhan)

[hashutosh] HIVE-4781 : LEFT SEMI JOIN generates wrong results when the num=
ber of rows belonging to a single key of the right table exceed hive.join.e=
mit.interval (Yin Huai via Ashutosh Chauhan)

[hashutosh] HIVE-4647 : RetryingHMSHandler logs too many error messages (Na=
vis via Ashutosh Chauhan)

[hashutosh] HIVE-4692 : Constant agg parameters will be replaced by ExprNod=
eColumnDesc with single-sourced multi-gby cases (Navis via Ashutosh Chauhan=
)


Changes for Build #2172
[hashutosh] HIVE-4781 : Adding new data files for tests. Missed in original=
 commit.


Changes for Build #2173

Changes for Build #2174
[navis] HIVE-2517 : Support group by on struct type (Ashutosh Chauhan via N=
avis)

[hashutosh] HIVE-4406 : Missing / or /<dbname> in hs2 jdbc uri switches mod=
e to embedded mode(Anandha Ranganathan via Ashutosh Chauhan)

[hashutosh] HIVE-4430 : Semantic analysis fails in presence of certain lite=
rals in on clause (Kevin Wilfong via Ashutosh Chauhan)

[hashutosh] HIVE-4757 : LazyTimestamp goes into irretrievable NULL mode onc=
e inited with NULL once (Gopal V via Ashutosh Chauhan)

[hashutosh] HIVE-4785 : Implement isCaseSensitive for Hive JDBC driver (Rob=
ert Roland via Ashutosh Chauhan)


Changes for Build #2175
[navis] HIVE-4436 : hive.exec.parallel=3Dtrue doesn't work on hadoop-2
 (Gopal V via Navis)


Changes for Build #2176

Changes for Build #2177
[hashutosh] HIVE-4689 : For outerjoins, joinEmitInterval might make wrong r=
esult (Navis via Ashutosh Chauhan)

[hashutosh] HIVE-3253 : ArrayIndexOutOfBounds exception for deeply nested s=
tructs (Thejas Nair via Ashutosh Chauhan)


Changes for Build #2178

Changes for Build #2179

Changes for Build #2180

Changes for Build #2181
[hashutosh] HIVE-4089 : javax.jdo : jdo2-api dependency not in Maven Centra=
l (Navis via Ashutosh Chauhan)

[ecapriolo] HIVE-4804 parallel order by fails for small datasets (Navis via=
 egc)

Submitted by:=09Navis
Reviewed by:=09Edward Capriolo


Changes for Build #2182

Changes for Build #2183
[hashutosh] HIVE-4814 : Adjust WebHCat e2e tests until HIVE4703 is addresse=
d (Eugene Koifman via Ashutosh Chauhan)


Changes for Build #2184
[hashutosh] HIVE-4811 : (Slightly) break up the SemanticAnalyzer monstrosit=
y (Gunther Hagleitner via Ashutosh Chauhan)


Changes for Build #2185
[hashutosh] HIVE-4251 : Indices can't be built on tables whose schema info =
comes from SerDe (Mark Wagner via Ashutosh Chauhan)

[hashutosh] HIVE-4805 : Enhance coverage of package org.apache.hadoop.hive.=
ql.exec.errors (Ivan Veselovsky via Ashutosh Chauhan)


Changes for Build #2186
[hashutosh] HIVE-4733 : HiveLockObjectData is not compared properly (Navis =
via Ashutosh Chauhan)

[ecapriolo] HIVE-3475 INLINE UDTF does not convert types properly (Igor Kab=
iljo and Navis Ryu via egc)

Submitted by:=09Navis Ryu and Igor Kabiljo
Reviewed by:=09Edward Capriolo


Changes for Build #2187
[hashutosh] HIVE-4802 : Fix url check for missing / or /<db> after hostname=
 in jdb uri (Thejas Nair via Ashutosh Chauhan)


Changes for Build #2188
[hashutosh] HIVE-4813 : Improve test coverage of package org.apache.hadoop.=
hive.ql.optimizer.pcr (Ivan Veselovsky via Ashutosh Chauhan)

[hashutosh] HIVE-4580 : Change DDLTask to report errors using canonical err=
or messages rather than http status codes (Eugene Koifman via Ashutosh Chau=
han)

[hashutosh] HIVE-4796 : Increase coverage of package org.apache.hadoop.hive=
.common.metrics (Ivan Veselovsky via Ashutosh Chauhan)

[navis] HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis)

[hashutosh] HIVE-3810 : HiveHistory.log need to replace \r with space befor=
e writing Entry.value to historyfile (Mark Grover via Ashutosh Chauhan)


Changes for Build #2189
[hashutosh] HIVE-4810 [jira] Refactor exec package
(Gunther Hagleitner via Ashutosh Chauhan)

Summary:
HIVE-4810

The exec package contains both operators and classes used to execute the jo=
b. Moving the latter into a sub package makes the package slightly more man=
ageable and will make it easier to provide a tez-based implementation.

Test Plan: Refactoring

Reviewers: ashutoshc

Reviewed By: ashutoshc

Differential Revision: https://reviews.facebook.net/D11625

[hashutosh] HIVE-4829 : TestWebHCatE2e checkstyle violation causes all test=
s to fail (Eugene Koifman via Ashutosh Chauhan)

[hashutosh] HIVE-4819 : Comments in CommonJoinOperator for aliasTag is not =
valid (Navis via Ashutosh Chauhan)


Changes for Build #2190
[hashutosh] HIVE-4807 : Hive metastore hangs (Sarvesh Sakalanaga via Ashuto=
sh Chauhan)

[hashutosh] HIVE-4833 : Fix eclipse template classpath to include the corre=
ct jdo lib (Yin Huai via Ashutosh Chauhan)

[hashutosh] HIVE-4830 : Test clientnegative/nested_complex_neg.q got broken=
 due to 4580 (Vikram Dixit via Ashutosh Chauhan)


Changes for Build #2191
[hashutosh] HIVE-3691 : TestDynamicSerDe failed with IBM JDK (Bing Li & Ren=
ata Ghisloti via Ashutosh Chauhan)


Changes for Build #2192

Changes for Build #2193

Changes for Build #2194

Changes for Build #2195
[hashutosh] HIVE-4840 : Fix eclipse template classpath to include the BoneC=
P lib (Yin Huai via Ashutosh Chauhan)


Changes for Build #2196
[navis] HIVE-4290 : Build profiles: Partial builds for quicker dev (Gunther=
 Hagleitner via Navis)

[navis] HIVE-4658 : Make KW_OUTER optional in outer joins (Edward Capriolo =
via Navis)


Changes for Build #2197

Changes for Build #2198

Changes for Build #2199
[hashutosh] HIVE-4852 : -Dbuild.profile=3Dcore fails (Gunther Hagleitner vi=
a Ashutosh Chauhan)

[hashutosh] HIVE-4854 : testCliDriver_load_hdfs_file_with_space_in_the_name=
 fails on hadoop 2 (Gunther Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-4853 : junit timeout needs to be updated (Gunther Hagleitn=
er via Ashutosh Chauhan)

[hashutosh] HIVE-4721 : Fix TestCliDriver.ptf_npath.q on 0.23 (Gunther Hagl=
eitner via Ashutosh Chauhan)


Changes for Build #2200
[ecapriolo] HIVE-3603 Enable client-side caching for scans on HBase (Navis =
Ryu via EGC)

Submitted by:=09Navis Ryu
Reviewed by:=09Edward Capriolo


Changes for Build #2201

Changes for Build #2203
[daijy] HIVE-4820 : webhcat_config.sh should set default values for HIVE_HO=
ME and HCAT_PREFIX that work with default build tree structure (Eugene Koif=
man via Jianyong Dai)

[hashutosh] HIVE-4845 : Correctness issue with MapJoins using the null safe=
 operator (Brock Noland via Ashutosh Chauhan)


Changes for Build #2204
[brock] HIVE-4865 - HiveLockObjects: Unlocking retries/times out when query=
 contains ":" (Gunther Hagleitner via Brock Noland)


Changes for Build #2205
[hashutosh] HIVE-2206 [jira] add a new optimizer for query correlation disc=
overy and optimization
(Yin Huai via Ashutosh Chauhan)

Summary:
update test results

This issue proposes a new logical optimizer called Correlation Optimizer, w=
hich is used to merge correlated MapReduce jobs (MR jobs) into a single MR =
job. The idea is based on YSmart (http://ysmart.cse.ohio-state.edu/). The p=
aper and slides of YSmart are linked at the bottom.

Since Hive translates queries in a sentence by sentence fashion, for every =
operation which may need to shuffle the data (e.g. join and aggregation ope=
rations), Hive will generate a MapReduce job for that operation. However, f=
or those operations which may need to shuffle the data, they may involve co=
rrelations explained below and thus can be executed in a single MR job.

=09Input Correlation: Multiple MR jobs have input correlation (IC) if their=
 input relation sets are not disjoint;
=09Transit Correlation: Multiple MR jobs have transit correlation (TC) if t=
hey have not only input correlation, but also the same partition key;
=09Job Flow Correlation: An MR has job =EF=AC=82ow correlation (JFC) with o=
ne of its child nodes if it has the same partition key as that child node.

The current implementation of correlation optimizer only detect correlation=
s among MR jobs for reduce-side join operators and reduce-side aggregation =
operators (not map only aggregation). A query will be optimized if it satis=
fies following conditions.

=09There exists a MR job for reduce-side join operator or reduce side aggre=
gation operator which have JFC with all of its parents MR jobs (TCs will be=
 also exploited if JFC exists);
=09All input tables of those correlated MR job are original input tables (n=
ot intermediate tables generated by sub-queries); and
=09No self join is involved in those correlated MR jobs.

Correlation optimizer is implemented as a logical optimizer. The main reaso=
ns are that it only needs to manipulate the query plan tree and it can leve=
rage the existing component on generating MR jobs.

Current implementation can serve as a framework for correlation related opt=
imizations. I think that it is better than adding individual optimizers.

There are several work that can be done in future to improve this optimizer=
. Here are three examples.

=09Support queries only involve TC;
=09Support queries in which input tables of correlated MR jobs involves int=
ermediate tables; and
=09Optimize queries involving self join.

References:
Paper and presentation of YSmart.
Paper: http://www.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-1=
1-7.pdf
Slides: http://sdrv.ms/UpwJJc

Test Plan: EMPTY

Reviewers: JIRA, ashutoshc

Reviewed By: ashutoshc

CC: brock

Differential Revision: https://reviews.facebook.net/D11097

[ecapriolo] HIVE-4873 Sort candidate functions in case of UDFArgumentExcept=
ion (Xuefu Zhang via egc)

Submitted by:=09Xuefu Zhang
Reviewed by:=09Edward Capriolo


Changes for Build #2206

Changes for Build #2207
[ecapriolo] HIVE-4675 Create new parallel unit test environment (Brock Nola=
nd via egc)

Submitted by: Brock Noland=09
Reviewed by: Edward Capriolo


Changes for Build #2208

Changes for Build #2209
[gates] Enable parallel execution of various E2E tests (deepeshk via gates)

[hashutosh] HIVE-4730 : Join on more than 2^31 records on single reducer fa=
iled (wrong results) (Navis via Ashutosh Chauhan)

[brock] HIVE-4818: SequenceId in operator is not thread safe (Edward Caprio=
lo via Brock Noland)

[brock] HIVE-4874 Identical methods PTFDeserializer.addOIPropertiestoSerDeP=
ropsMap(), PTFTranslator.addOIPropertiestoSerDePropsMap() (Edward Capriolo =
via Brock Noland


All tests passed

The Apache Jenkins build system has built Hive-trunk-h0.21 (build #2209)

Status: Still Failing

Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/2209=
/ to view the results.
------=_Part_995_528266177.1374337869460--