hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Enis Soztutar (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-16288) HFile intermediate block level indexes might recurse forever creating multi TB files
Date Tue, 26 Jul 2016 17:59:20 GMT
Enis Soztutar created HBASE-16288:
-------------------------------------

             Summary: HFile intermediate block level indexes might recurse forever creating
multi TB files
                 Key: HBASE-16288
                 URL: https://issues.apache.org/jira/browse/HBASE-16288
             Project: HBase
          Issue Type: Bug
            Reporter: Enis Soztutar
            Assignee: Enis Soztutar


Mighty [~elserj] was debugging an opentsdb cluster where some region directory ended up having
5TB+ files under <regiondir>/.tmp/ 

Further debugging and analysis, we were able to reproduce the problem locally where we never
we recursing in this code path for writing intermediate level indices: 

{code:title=HFileBlockIndex.java}
if (curInlineChunk != null) {
        while (rootChunk.getRootSize() > maxChunkSize) {
          rootChunk = writeIntermediateLevel(out, rootChunk);
          numLevels += 1;
        }
      }
{code}

The problem happens if we end up with a very large rowKey (larger than "hfile.index.block.max.size"
being the first key in the block, then moving all the way to the root-level index building.
We will keep writing and building the next level of intermediate level indices with a single
very-large key. This can happen in flush / compaction / region recovery causing cluster inoperability
due to ever-growing files. 

Seems the issue was also reported earlier, with a temporary workaround: 
https://github.com/OpenTSDB/opentsdb/issues/490



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message