hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <>
Subject [jira] [Created] (HIVE-14886) File deduplication in FSOP is not used correctly for list bucketing
Date Tue, 04 Oct 2016 17:57:21 GMT
Sergey Shelukhin created HIVE-14886:

             Summary: File deduplication in FSOP is not used correctly for list bucketing
                 Key: HIVE-14886
             Project: Hive
          Issue Type: Bug
            Reporter: Sergey Shelukhin

I am making things work for MM tables, so I noticed this after adding the logging to removeTempOrDuplicateFiles/2
method that is called from FSOP:
      } else /* sershe: means "if !isTempPath(one)" */ {
        String taskId = getPrefixedTaskIdFromFilename(one.getPath().getName());"removeTempOrDuplicateFiles pondering " + one.getPath() +
", taskId " + taskId);
This is called from FSOP jobCloseOp, via Utilities.mvFileToFinalPath, then via non-dynpart
path in removeTempOrDuplicateFiles/4.
The files passed are from the root destination, disregarding list bucketing, so what happens
is this:
2016-10-03T19:01:38,615  INFO [912dde0f-91af-4a27-b358-5d782897ed1d main] Log14535: removeTempOrDuplicateFiles
pondering hdfs://localhost:63026/build/ql/test/data/warehouse/skew_mm/.hive-staging_hive_2016-10-03_19-01-38_324_9113577068018508885-1/_tmp.-ext-10000/HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME,
2016-10-03T19:01:38,616  INFO [912dde0f-91af-4a27-b358-5d782897ed1d main] Log14535: removeTempOrDuplicateFiles
pondering hdfs://localhost:63026/build/ql/test/data/warehouse/skew_mm/.hive-staging_hive_2016-10-03_19-01-38_324_9113577068018508885-1/_tmp.-ext-10000/k1=0,
taskId 0 [sershe: this is only true by coincidence, task if comes from k1 value]

When I started calling the method correctly on MM path, it started deleting files for different
LB directories thinking they are the same stuff... so, some special logic may be needed for
this similar to dpCtx.

This message was sent by Atlassian JIRA

View raw message