Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
From: Andrew Purtell <apurtell@apache.org>
Date: Mon, 6 Oct 2014 20:55:24 -0700
Message-ID: 
 <CA+RK=_AQzrUt7YABkXfppbGLBCcha1hvu1ZxD1+nCGux4fPRPw@mail.gmail.com>
Subject: An important read
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
Content-Type: multipart/alternative; boundary=047d7b3435eaafd16c0504cd2dc7

--047d7b3435eaafd16c0504cd2dc7
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf

Simple Testing Can Prevent Most Critical Failures: An Analysis of
Production Failures in Distributed Data-intensive Systems
Yuan et. al, University of Toronto

Large, production quality distributed systems still fail periodically, and
do so sometimes catastrophically, where most or all users experience an
outage or data loss. We present the result of a comprehensive study
investigating 198 randomly selected, user-reported failures that occurred
on Cassandra, HBase, Hadoop Distributed File System (HDFS), Hadoop
MapReduce, and Redis, with the goal of understanding how one or multiple
faults eventually evolve into a user-visible failure. We found that from a
testing point of view, almost all failures require only 3 or fewer nodes to
reproduce, which is good news considering that these services typically run
on a very large number of nodes. However, multiple inputs are needed to
trigger the failures with the order between them being important. Finally,
we found the error logs of these systems typically contain sufficient data
on both the errors and the input events that triggered the failure,
enabling the diagnose and the reproduction of the production failures.

We found the majority of catastrophic failures could easily have been
prevented by performing simple testing on error handling code =E2=80=93 the=
 last
line of defense =E2=80=93 even without an understanding of the software des=
ign. We
extracted three simple rules from the bugs that have lead to some of the
catastrophic failures, and developed a static checker, Aspirator, capable
of locating these bugs. Over 30% of the catastrophic failures would have
been prevented had Aspirator been used and the identified bugs fixed.
Running Aspirator on the code of 9 distributed systems located 143 bugs and
bad practices that have been fixed or confirmed by the developers.


=E2=80=8BThis is an interesting benefit of open source and open development
process. Please read this detailed analysis of availability and data loss
bugs resulting from improper error handling, in HBase and other systems.
The authors focus on a particular pattern of defect and cause. The point is
well taken. It would be worth taking time where possible to revisit
exception handling, especially where we have low test coverage.

Also, consider HBASE-11912. The static analyses mentioned in this paper
could likely be implemented with error-prone. Development and code review
will always be uneven in a volunteer open source project. However if we
agree on some baseline practices, and those are amenable to static
analysis, then we could build validation of those practices into the
compiler, in effect.

--=20
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

--047d7b3435eaafd16c0504cd2dc7--