impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bharath v (JIRA)" <j...@apache.org>
Subject [jira] [Created] (IMPALA-5325) Update HdfsTable#totalHdfsBytes_ and HdfsTable#numHdfsFiles_ atomically.
Date Mon, 15 May 2017 17:33:04 GMT
bharath v created IMPALA-5325:
---------------------------------

             Summary: Update HdfsTable#totalHdfsBytes_ and HdfsTable#numHdfsFiles_ atomically.
                 Key: IMPALA-5325
                 URL: https://issues.apache.org/jira/browse/IMPALA-5325
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog
            Reporter: bharath v
             Fix For: Impala 2.8.0, Impala 2.9.0


In the following snippet from HdfsTable.java

{noformat}
 private void loadPartitionFileMetadata(StorageDescriptor storageDescriptor,
      HdfsPartition partition) throws Exception {
    Preconditions.checkNotNull(storageDescriptor);
    Preconditions.checkNotNull(partition);
    Path partDirPath = new Path(storageDescriptor.getLocation());
    FileSystem fs = partDirPath.getFileSystem(CONF);
    if (!fs.exists(partDirPath)) return;

    numHdfsFiles_ -= partition.getNumFileDescriptors();
    totalHdfsBytes_ -= partition.getSize();
    Preconditions.checkState(numHdfsFiles_ >= 0 && totalHdfsBytes_ >= 0);
    refreshFileMetadata(partition);
  }
{noformat}

If {{refreshFileMetadata}} throws an exception before updating {{numHdfsFiles_}} and {{totalHdfsBytes_}},
the table could get into an inconsistent state and could possibly hit a preconditions check
in the next block metadata update.

Fix: Update {{totalHdfsBytes_}} and {{numHdfsFiles_}} in an atomic fashion.

Workaround: Run "invalidate table" to clear the full table state.

This bug also affects older versions around method {{updatePartitionFds}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message