Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Thu, 2 Nov 2017 02:05:01 +0000 (UTC)
From: "Jason Dere (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.13115521.1509588281000.127533.1509588301042@Atlassian.JIRA>
In-Reply-To: <JIRA.13115521.1509588281000@Atlassian.JIRA>
References: <JIRA.13115521.1509588281000@Atlassian.JIRA> <JIRA.13115521.1509588281929@jira-lw-us.apache.org>
Subject: [jira] [Assigned] (HIVE-17963) Fix for HIVE-17113 can be improved
 for non-blobstore filesystems
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 02 Nov 2017 02:05:07 -0000


     [ https://issues.apache.org/jira/browse/HIVE-17963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Dere reassigned HIVE-17963:
---------------------------------


> Fix for HIVE-17113 can be improved for non-blobstore filesystems
> ----------------------------------------------------------------
>
>                 Key: HIVE-17963
>                 URL: https://issues.apache.org/jira/browse/HIVE-17963
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Jason Dere
>            Assignee: Jason Dere
>            Priority: Major
>
> HIVE-17113/HIVE-17813 fix the duplicate file issue by performing file moves on a file-by-file basis. For non-blobstore filesystems this results in many more filesystem/namenode operations compared to the previous Utilities.mvFileToFinalPath() behavior (dedup files in src dir, rename src dir to final dir).
> For non-blobstore filesystems, a better solution would be the one described [here|https://issues.apache.org/jira/browse/HIVE-17113?focusedCommentId=16100564&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16100564]:
> 1) Move the temp directory to a new directory name, to prevent additional files from being added by any runaway processes.
> 2) Run removeTempOrDuplicateFiles() on this renamed temp directory
> 3) Run renameOrMoveFiles() to move the renamed temp directory to the final location.
> This results in only one additional file operation in non-blobstore FSes compared to the original Utilities.mvFileToFinalPath() behavior.
> The proposal is to do away with the config setting hive.exec.move.files.from.source.dir and always have behavior that should take care of the duplicate file issue described in HIVE-17113. For non-blobstore filesystems we will do steps 1-3 described above. For blobstore filesystems we will do the solution done in HIVE-17113/HIVE-17813 which does the file-by-file copy - this should have the same number of file operations as doing a rename directory on blobstore, which effectively results in file moves on a file-by-file basis.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)