Return-Path: X-Original-To: apmail-hbase-dev-archive@www.apache.org Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 470C917E31 for ; Sat, 25 Oct 2014 18:50:04 +0000 (UTC) Received: (qmail 7355 invoked by uid 500); 25 Oct 2014 18:50:03 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 7266 invoked by uid 500); 25 Oct 2014 18:50:03 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 7254 invoked by uid 99); 25 Oct 2014 18:50:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Oct 2014 18:50:02 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ndimiduk@gmail.com designates 209.85.214.182 as permitted sender) Received: from [209.85.214.182] (HELO mail-ob0-f182.google.com) (209.85.214.182) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Oct 2014 18:49:58 +0000 Received: by mail-ob0-f182.google.com with SMTP id nt9so1013321obb.41 for ; Sat, 25 Oct 2014 11:49:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=tKJpQI+jMJl8bR5PC2UJ/1kCnKYQrqNJXHcaW0O2vDI=; b=Aa6Mip734rCqQAOiJpZhOPAiVV/jay3gcmqn+xosZU0RHUiaKF4M5Q+rDUu46seYPv whhoE6mO2cjxknYDI67SwHYjuBmrUp5lxmBk7MvIIV2jzex+OEdbfnPWhrX+XBTzxaRK 0F5yTPB5b/wYVp6CHIur5mp0V3fPUvtV5hoaKDnmMPfazu53WosEoNr6DlAHUnQjPgVY zhiM4EHvV5ChgticfLW/X+3N12Hm3mOYTEhDd4tATmej4uwZB5WGUPC5k+stVcAUoJcn 30pNomPxC6JWuhbRlz9X3yLaiMX1Rhksm7U5Tcww1IfrNSJDq/JudGmaweloT4DoWCwg wSHw== MIME-Version: 1.0 X-Received: by 10.60.45.206 with SMTP id p14mr3847556oem.38.1414262942777; Sat, 25 Oct 2014 11:49:02 -0700 (PDT) Received: by 10.182.80.164 with HTTP; Sat, 25 Oct 2014 11:49:02 -0700 (PDT) In-Reply-To: References: Date: Sat, 25 Oct 2014 11:49:02 -0700 Message-ID: Subject: Re: Reworking the use of log levels From: Nick Dimiduk To: "dev@hbase.apache.org" Content-Type: multipart/alternative; boundary=001a11c2019a52c737050643c029 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2019a52c737050643c029 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable We have the ability to alter log levels at runtime. This would allow an operator to temporarily increase log level for afflicted components, even in production. Doing this on a server-by-server basis should have minimal impact on overall cluster performance. Maybe this needs to be better documented? Maybe we need a script that makes this easier, or could be managed via a new shell command? On Saturday, October 25, 2014, Andrew Purtell wrote: > =E2=80=8B > On Sat, Oct 25, 2014 at 6:34 AM, Sean Busbey > wrote: > > > Even if debug is disabled in production, it could be enabled on a > > non-production system for reproducing the problem, no? > > > > =E2=80=8BIn my experience, often enough, no.=E2=80=8B > > I do hear the complaint that Hadoop ecosystem projects are quite operator > unfriendly because error messages most often come in the form of a > stacktrace. It's a totally valid point. I think we could certainly improv= e > the exception message printed ahead of the stacktrace in a large number o= f > cases. > > > > On Sat, Oct 25, 2014 at 6:34 AM, Sean Busbey > wrote: > > > Even if debug is disabled in production, it could be enabled on a > > non-production system for reproducing the problem, no? > > > > -- > > Sean > > On Oct 25, 2014 7:11 AM, "Qiang Tian" = > > wrote: > > > > > perhaps case by case is better. stacktrace is one of most important > > problem > > > determination methods. debug is mostly disabled in production, we ma= y > > lose > > > important clues. > > > > > > > > > On Sat, Oct 25, 2014 at 1:14 PM, Sean Busbey > > > wrote: > > > > > > > Hi! > > > > > > > > Right now we have many failure paths where we send stack traces to > log > > > > files at ERROR / WARN. In an effort to make things easier to operat= e, > > I'd > > > > like to propose we move towards: > > > > > > > > * INFO/WARN/ERROR : description of failure and if possible an actio= n > an > > > > operator could take to fix/diagnose > > > > * DEBUG : information needed to handle failures that require > developer > > > > action, i.e. stack traces > > > > > > > > I figure this can go as one or more subtasks off of HBASE-12341, bu= t > > > wanted > > > > to float things here before I get started. > > > > > > > > -- > > > > Sean > > > > > > > > > > > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) > --001a11c2019a52c737050643c029--