hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugene Koifman <ekoif...@hortonworks.com>
Subject Re: Review Request 65413: HIVE-18575 ACID properties usage in jobconf is ambiguous for MM tables
Date Fri, 09 Feb 2018 21:16:56 GMT


> On Feb. 7, 2018, 11:24 a.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java
> > Line 429 (original), 429 (patched)
> > <https://reviews.apache.org/r/65413/diff/3/?file=1953196#file1953196line429>
> >
> >     Below, you left the same compBuilder.setIsAcid(isFullAcid) but here it using
isTransactional - why the logical change?
> 
> Sergey Shelukhin wrote:
>     fixed; this affects whether locks are taken so my understanding is it's also needed
for MM
> 
> Eugene Koifman wrote:
>     it doesn't affect whether locsk are taken.  locks are taken for all tables even transactional=false.
 it affects whether certain entries are made for compactor which is not needed for MM and
just pollutes those tables
> 
> Sergey Shelukhin wrote:
>     I'm not sure what this has to do with compactor. This sets the flag on LockComponent.
>     The only place where this flag is used is 
>     or (LockComponent lc : rqst.getComponent()) {
>                 if(lc.isSetIsAcid() && !lc.isIsAcid()) {
>                   //we don't prevent using non-acid resources in a txn but we do lock
them
>                   continue;
>                 }
>                 
>     Is it valid to treat MM table as non-acid resource?

isAcid here means full acid, so yes.  the locks will still be acquired


- Eugene


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65413/#review196904
-----------------------------------------------------------


On Feb. 8, 2018, 5:52 p.m., Sergey Shelukhin wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65413/
> -----------------------------------------------------------
> 
> (Updated Feb. 8, 2018, 5:52 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> .
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 67e22f6649 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
5ee8aadfa7 
>   hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/HiveEndPoint.java 3388a34446

>   hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/client/lock/Lock.java
c2728376b2 
>   hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java 4e928121c7

>   hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/StreamingAssert.java
c98d22be2e 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
a5e6293a3e 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
d252279be9 
>   llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
68bb168bd2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 20c2c3294a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 090a18852a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 270b576199 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java abd42ec651 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 430e0fc551 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 856b026c91 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ff2cc0455c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java 61565ef030 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java da200049bc

>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 7b157e6486

>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 3968b0e899 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c8d1589f44 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
0fdff7d853 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 69447d9d34 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 190771ea6b

>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java b766791ebc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 8e587f1cf6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java e1cea22005 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 661446df0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/Partish.java 78f48b169a 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 0e456df19c 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 8945fdf1e7 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 92f005d1dc

>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java c6a866a164

>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
65508f4ddd 
>   standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/LockComponentBuilder.java
de6c718ba9 
> 
> 
> Diff: https://reviews.apache.org/r/65413/diff/5/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message