accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher Tubbs (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ACCUMULO-4536) Infinite loop creating empty WAL files when disk space is low
Date Sat, 17 Dec 2016 00:39:58 GMT
Christopher Tubbs created ACCUMULO-4536:
-------------------------------------------

             Summary: Infinite loop creating empty WAL files when disk space is low
                 Key: ACCUMULO-4536
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-4536
             Project: Accumulo
          Issue Type: Bug
          Components: tserver
    Affects Versions: 1.6.6
            Reporter: Christopher Tubbs
            Priority: Minor


Saw this on 1.6.6 with a small disk for testing (32GB disk). The default walog size is around
1GB, and only 3.4GB were left available on each data node.

The namenode reported that no data nodes had space available when trying to write the first
block, so the tserver failed to write the file. It kept retrying, resulting in the namenode
filling up with thousands of zero-length WAL files.

The fix was to lower the {{tserver.walog.max.size}} to {{100M}}. Another solution would be
to use a larger disk.

The infinite loop problem, constantly creating new empty WAL files is still a problem, but
it should only happen when low on disk space, which is likely going to cause other, more serious
problems... and could be avoided with good system monitoring.

I have not tested on versions newer than 1.6.6, but I imagine it's still a problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message