hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates" <ga...@hortonworks.com>
Subject Re: Review Request 25341: HIVE-7078 Need file sink operators that work with ACID
Date Fri, 05 Sep 2014 16:28:50 GMT


> On Sept. 5, 2014, 8:21 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java, line 211
> > <https://reviews.apache.org/r/25341/diff/1/?file=676838#file676838line211>
> >
> >     I don't see needToRename being used elsewhere. So can you replace this chunk
with 
> >     if (fs.exists(outPaths[idx]) && !fs.rename(outPaths[idx], finalPaths[idx])
{..}
> >     ?

But that would change the behavior in the standard case.  This way the behavior is only changed
in the update and delete case.  I didn't want to add an extra stat for every type of write.


> On Sept. 5, 2014, 8:21 a.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java, line 556
> > <https://reviews.apache.org/r/25341/diff/1/?file=676838#file676838line556>
> >
> >     Can you add a comment what is happening here? Are you just stripping off the
_attemptId from taskId_attemptId? If so can you use Utilities.getTaskIdFromFilename() instead?

Switched to Utilities.getTaskIdFromFilename() as requested.


- Alan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25341/#review52339
-----------------------------------------------------------


On Sept. 4, 2014, 3:49 p.m., Alan Gates wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25341/
> -----------------------------------------------------------
> 
> (Updated Sept. 4, 2014, 3:49 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Bugs: HIVE-7078
>     https://issues.apache.org/jira/browse/HIVE-7078
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Changes FileSinkOperator to use RecordUpdater in cases where an ACID write is being done.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java d4e61d8 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java f584926 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java c3a83d4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java 301dde5 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/25341/diff/
> 
> 
> Testing
> -------
> 
> Added a new unit test TestFileSinkOperator that tests writing of standard (non-ACID)
data via RecordWriter and acid data via RecordUpdater, in both partitioned and non-partitioned
cases.
> 
> 
> Thanks,
> 
> Alan Gates
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message