Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 144FB200BEF for ; Tue, 20 Dec 2016 15:44:00 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 12E47160B1B; Tue, 20 Dec 2016 14:44:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5CFBF160B29 for ; Tue, 20 Dec 2016 15:43:59 +0100 (CET) Received: (qmail 59505 invoked by uid 500); 20 Dec 2016 14:43:58 -0000 Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: jira@apache.org Delivered-To: mailing list notifications@accumulo.apache.org Received: (qmail 59430 invoked by uid 99); 20 Dec 2016 14:43:58 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Dec 2016 14:43:58 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 6C2802C1F54 for ; Tue, 20 Dec 2016 14:43:58 +0000 (UTC) Date: Tue, 20 Dec 2016 14:43:58 +0000 (UTC) From: "Michael Miller (JIRA)" To: notifications@accumulo.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (ACCUMULO-4536) Infinite loop creating empty WAL files when disk space is low MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 20 Dec 2016 14:44:00 -0000 [ https://issues.apache.org/jira/browse/ACCUMULO-4536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miller updated ACCUMULO-4536: ------------------------------------- Affects Version/s: (was: 1.8.0) > Infinite loop creating empty WAL files when disk space is low > ------------------------------------------------------------- > > Key: ACCUMULO-4536 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4536 > Project: Accumulo > Issue Type: Bug > Components: tserver > Affects Versions: 1.6.6, 1.7.2 > Reporter: Christopher Tubbs > Assignee: Michael Miller > Priority: Minor > > Saw this on 1.6.6 with a small disk for testing (32GB disk). The default walog size is around 1GB, and only 3.4GB were left available on each data node. > The namenode reported that no data nodes had space available when trying to write the first block, so the tserver failed to write the file. It kept retrying, resulting in the namenode filling up with thousands of zero-length WAL files. > The fix was to lower the {{tserver.walog.max.size}} to {{100M}}. Another solution would be to use a larger disk. > The infinite loop problem, constantly creating new empty WAL files is still a problem, but it should only happen when low on disk space, which is likely going to cause other, more serious problems... and could be avoided with good system monitoring. > I have not tested on versions newer than 1.6.6, but I imagine it's still a problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)