hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anishek <anis...@gmail.com>
Subject Re: Review Request 62360: HIVE-16898: Validation of source file after distcp in repl load
Date Tue, 19 Sep 2017 05:46:00 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62360/#review185643
-----------------------------------------------------------




ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 51 (patched)
<https://reviews.apache.org/r/62360/#comment261959>

    This can be private



ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 86 (patched)
<https://reviews.apache.org/r/62360/#comment261960>

    we can start with
    
    if(!srcFile.isUseSourcePath() && sourceFs.exists(srcFile.getEffectivePath())){continue;}

    
    as the first statement inside the for loop for srcFiles, that way it wouldnt have to try
and get checksum etc for files in CM. Check to see existence of CM is to make sure that while
doing the copy the cleaner thread did not clean the CM.
    
    This will also make a few subsequent conditional checking irrelevant.



ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 120 (patched)
<https://reviews.apache.org/r/62360/#comment261962>

    May be i am missing something but even when we are using the CM path in copyAndVerify
towards the last, the file from CM itself might be deleted via the CM cleaner thread, in that
case the doCopy will quietly fail and move on rather than failing replication.



ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
Lines 130 (patched)
<https://reviews.apache.org/r/62360/#comment261961>

    copyAsUser and hiveConf are member variables, they dont need to be passed to functions.


- anishek


On Sept. 19, 2017, 12:14 a.m., Daniel Dai wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62360/
> -----------------------------------------------------------
> 
> (Updated Sept. 19, 2017, 12:14 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> See HIVE-16898
> 
> 
> Diffs
> -----
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ReplChangeManager.java 88d6a7a

>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReplCopyTask.java 54746d3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java 28e7bcb 
> 
> 
> Diff: https://reviews.apache.org/r/62360/diff/3/
> 
> 
> Testing
> -------
> 
> Manually test it with debugger: setup a breakpoint right before copy, and drop table
in another session.
> 
> 
> Thanks,
> 
> Daniel Dai
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message