hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vrodio...@carrieriq.com>
Subject RE: HBase 0.90.4 missing data in production
Date Thu, 03 Nov 2011 20:03:08 GMT
I would say that they are identical

This is Nov-2 RegionServer1:

2011-11-02 00:12:29,352 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:12:35,220 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:12:37,639 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:12:42,865 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:12:49,710 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:12:54,079 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:12:55,692 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:13:01,779 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:15:03,044 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:15:05,686 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:16:07,621 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:16:19,499 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:17:23,085 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:17:29,428 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:18:23,436 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:18:25,183 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:19:02,230 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:19:04,624 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:19:06,577 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:20:22,306 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:20:25,645 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:20:27,662 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:21:31,655 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:22:13,565 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 00:22:15,892 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 06:05:11,911 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 06:10:00,359 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 06:10:02,612 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 06:10:31,625 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 06:21:06,955 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 06:23:33,967 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 06:23:37,270 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 06:24:45,313 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 06:24:48,087 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 06:26:49,583 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 06:27:52,948 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 06:28:54,638 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 06:29:20,036 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 20:03:02,271 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 20:03:04,420 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 20:04:20,810 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 20:05:48,922 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 20:06:35,397 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 20:07:07,572 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 20:07:27,270 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 20:12:04,725 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 20:12:28,737 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 20:13:19,586 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 20:14:12,752 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 20:15:27,228 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 20:16:01,354 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 23:02:46,540 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 23:02:48,722 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 23:02:54,242 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 23:03:13,952 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 23:03:34,459 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 23:06:48,307 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 23:07:01,707 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 23:09:20,909 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 23:09:34,632 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-02 23:11:38,399 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog

This is Nov-3 RegionServer1:

2011-11-03 03:28:08,934 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-03 03:28:22,234 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-03 03:41:20,823 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-03 06:20:22,524 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-03 06:29:40,429 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-03 06:35:22,349 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-03 18:43:52,802 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-03 18:46:09,616 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-03 18:47:33,136 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-03 18:52:10,282 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-03 18:52:39,507 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-03 18:52:56,564 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-03 18:53:06,166 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog
2011-11-03 18:53:15,147 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append.
Requesting close of hlog

Less but the cluster size is approx the same - 8-10 FATAL errors per cluster

I can not find any errors (ERROR) in RS log files except few, related to the above FATAL.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com

________________________________________
From: saint.ack@gmail.com [saint.ack@gmail.com] On Behalf Of Stack [stack@duboce.net]
Sent: Thursday, November 03, 2011 12:00 PM
To: dev@hbase.apache.org
Subject: Re: HBase 0.90.4 missing data in production

On Thu, Nov 3, 2011 at 11:56 AM, Vladimir Rodionov
<vrodionov@carrieriq.com> wrote:
> I can not reproduce the bug today. No missing rows after cluster restart. I will answer
> Stack's question as soon as I hit this problem again.
>

Can you compare logs from the bad and good runs Vladimir?
St.Ack

Confidentiality Notice:  The information contained in this message, including any attachments
hereto, may be confidential and is intended to be read only by the individual or entity to
whom this message is addressed. If the reader of this message is not the intended recipient
or an agent or designee of the intended recipient, please note that any review, use, disclosure
or distribution of this message or its attachments, in any form, is strictly prohibited. 
If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com
and delete or destroy any copy of this message and its attachments.

Mime
View raw message