impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bharath v (JIRA)" <>
Subject [jira] [Created] (IMPALA-5325) Update HdfsTable#totalHdfsBytes_ and HdfsTable#numHdfsFiles_ atomically.
Date Mon, 15 May 2017 17:33:04 GMT
bharath v created IMPALA-5325:

             Summary: Update HdfsTable#totalHdfsBytes_ and HdfsTable#numHdfsFiles_ atomically.
                 Key: IMPALA-5325
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog
            Reporter: bharath v
             Fix For: Impala 2.8.0, Impala 2.9.0

In the following snippet from

 private void loadPartitionFileMetadata(StorageDescriptor storageDescriptor,
      HdfsPartition partition) throws Exception {
    Path partDirPath = new Path(storageDescriptor.getLocation());
    FileSystem fs = partDirPath.getFileSystem(CONF);
    if (!fs.exists(partDirPath)) return;

    numHdfsFiles_ -= partition.getNumFileDescriptors();
    totalHdfsBytes_ -= partition.getSize();
    Preconditions.checkState(numHdfsFiles_ >= 0 && totalHdfsBytes_ >= 0);

If {{refreshFileMetadata}} throws an exception before updating {{numHdfsFiles_}} and {{totalHdfsBytes_}},
the table could get into an inconsistent state and could possibly hit a preconditions check
in the next block metadata update.

Fix: Update {{totalHdfsBytes_}} and {{numHdfsFiles_}} in an atomic fashion.

Workaround: Run "invalidate table" to clear the full table state.

This bug also affects older versions around method {{updatePartitionFds}}

This message was sent by Atlassian JIRA

View raw message