hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Dere <jd...@hortonworks.com>
Subject Review Request 63544: HIVE-17963: Fix for HIVE-17113 can be improved for non-blobstore filesystems
Date Fri, 03 Nov 2017 19:36:17 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63544/
-----------------------------------------------------------

Review request for hive, Ashutosh Chauhan and Owen O'Malley.


Bugs: HIVE-17963
    https://issues.apache.org/jira/browse/HIVE-17963


Repository: hive-git


Description
-------

For non-blobstore FS:
1) Move the temp directory to a new directory name, to prevent additional files from being
added by any runaway processes.
2) Run removeTempOrDuplicateFiles() on this renamed temp directory
3) Run renameOrMoveFiles() to move the renamed temp directory to the final location.

For blobstore FS:
1) Remove duplicates from tmpPath
2) Use moveSpecifiedFiles() to perform a file-by-file move of the de-duped files to specPath

Also remove hive.exec.move.files.from.source.dir setting - the above changes will always be
enabled.


Diffs
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbe4de5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1be7eab 
  ql/src/test/queries/clientpositive/insert_with_move_files_from_source_dir.q 0117755 
  ql/src/test/queries/clientpositive/skewjoin.q 9ad4b77 


Diff: https://reviews.apache.org/r/63544/diff/1/


Testing
-------


Thanks,

Jason Dere


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message