impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Behm (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-5429: Multi threaded block metadata loading
Date Sat, 21 Oct 2017 00:12:39 GMT
Alex Behm has posted comments on this change. ( )

Change subject: IMPALA-5429: Multi threaded block metadata loading

Patch Set 7:

File fe/src/main/java/org/apache/impala/catalog/
PS7, Line 221:     // are excluded because they are considered hidden from Impala's perspecitve

// Number of hidden files excluded from file metadata loading . More details at isValidDataFile().
PS7, Line 226:     // Number of files skipped while computeing the block metadata information.

// Number of files skipped from file metadata loading because the the files have not changed
since the last load. More details at hasFileChanged().
PS7, Line 248:     // If set to true, reloads the block metadata only when the files in this
reloads the file metadata (let's try to use the new "file metadata" terminology consistently)
PS7, Line 249:     // has changed since last load (more details in hasFileChanged()).
have changed
PS7, Line 785:    * much higher throughput of RPC calls for listStatus/listFiles. Based on
... RPC calls for listStatus/listFiles. For simplicity, the filesystem type is determined
based on the table's root path and not for each partition individually. Based on our experiments,
S3 showed ...
PS7, Line 791:    * This method is not optimized for tables with mixed partition types (partitions
I don'd think we need this much detail, how about folding a simple sentence into the existing
paragraph above, see suggestion above.
PS7, Line 794:    * This may not work well with the mixed partition types and needs to fixed.
Don't think this is needed.
PS7, Line 819:     // For tables without partitions, we have no block metadata to load.
remove ","
PS7, Line 841:           LOG.error("Encountered an error loading block metadata for table:
" +
Could this flood the log when HDFS is under pressure or there is an intermittent Kerberos
issue? I'm thinking we should aggregate these into a single log message. Do you think having
every single path helps with supportability? My feeling is it's more important to note how
many tasks failed, but I'll defer to you if you think having the paths is useful.

To view, visit
To unsubscribe, visit

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I07eaa7151dfc4d56da8db8c2654bd65d8f808481
Gerrit-Change-Number: 8235
Gerrit-PatchSet: 7
Gerrit-Owner: Bharath Vissapragada <>
Gerrit-Reviewer: Alex Behm <>
Gerrit-Reviewer: Bharath Vissapragada <>
Gerrit-Reviewer: Dimitris Tsirogiannis <>
Gerrit-Reviewer: Jim Apple <>
Gerrit-Reviewer: Mostafa Mokhtar <>
Gerrit-Reviewer: Vuk Ercegovac <>
Gerrit-Comment-Date: Sat, 21 Oct 2017 00:12:39 +0000
Gerrit-HasComments: Yes

  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message