Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 62A5918EC8 for ; Tue, 12 May 2015 21:09:16 +0000 (UTC) Received: (qmail 6806 invoked by uid 500); 12 May 2015 21:09:14 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 6742 invoked by uid 500); 12 May 2015 21:09:14 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 6728 invoked by uid 99); 12 May 2015 21:09:14 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 May 2015 21:09:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 03CC4C1241 for ; Tue, 12 May 2015 21:09:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.23 X-Spam-Level: X-Spam-Status: No, score=0.23 tagged_above=-999 required=6.31 tests=[KAM_LOTSOFHASH=0.25, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id qG6rr6F2eAZ7 for ; Tue, 12 May 2015 21:09:01 +0000 (UTC) Received: from nk11p18im-asmtp001.me.com (nk11p18im-asmtp001.me.com [17.158.120.160]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 89ECC20D8A for ; Tue, 12 May 2015 21:09:01 +0000 (UTC) Received: from [10.154.75.180] (host-95-199-11-180.mobileonline.telia.com [95.199.11.180]) by nk11p18im-asmtp001.me.com (Oracle Communications Messaging Server 7.0.5.35.0 64bit (built Dec 4 2014)) with ESMTPSA id <0NO900MGK9ESG040@nk11p18im-asmtp001.me.com> for user@hbase.apache.org; Tue, 12 May 2015 21:08:55 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.14.151,1.0.33,0.0.0000 definitions=2015-05-12_07:2015-05-12,2015-05-12,1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1412110000 definitions=main-1505120256 Content-type: text/plain; charset=windows-1251 Content-transfer-encoding: quoted-printable From: Akmal Abbasov MIME-version: 1.0 (1.0) Subject: Re: HBase snapshots and Compaction Message-id: <58643EA9-AC7B-4F76-9A4D-0CDB22D8BA31@icloud.com> Date: Tue, 12 May 2015 23:06:31 +0200 References: <8D6E150D-548B-481E-AC2F-627054A18CD4@icloud.com> In-reply-to: To: "user@hbase.apache.org" X-Mailer: iPhone Mail (12F70) Hi Ted, Thank you for reply. I am running with the default settings. Sent from my iPhone > On 12 May 2015, at 22:02, Ted Yu wrote: >=20 > Can you show us compaction related parameters you use ? >=20 > e.g. hbase.hregion.majorcompaction , hbase.hregion.majorcompaction.jitter ,= > etc >=20 > On Tue, May 12, 2015 at 9:52 AM, Akmal Abbasov > wrote: >=20 >> HI, >> I am using HBase 0.98.7. >> I am using HBase snapshots to backup data. I create snapshot of tables >> each our. >> Each create snapshot process will cause the flush of the memstore, and >> creation of hfiles. >> When the number of hfiles will reach 3 the MINOR compaction process will >> start for each CF. >> Ok, I was expecting that the compaction will process only small hfiles, >> and I won=92t have problems with moving all data to archive folder >> each time after compaction process ends. >> But most of the times, the minor compaction is promoted to major(more tha= n >> 100 in 24 hours without loads). >> As far as I know, the only possibility for this is that all hfiles are >> eligible for compaction. >> But when I tested the archive folder for a CF I see the strange situation= >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 06:04 >> /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/36d= c06f4c34242daadc343d857a35734 >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 06:04 >> /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/7e8= b993f97b84f4594542144f15b0a1e >> -rw-r--r-- 3 akmal supergroup 1.1 K 2015-05-10 06:04 >> /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/b9a= fc64792ba4bf99a08f34033cc46ac >> -rw-r--r-- 3 akmal supergroup 638.4 K 2015-05-10 06:04 >> /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/dff= 846ae4fc24d418289a95322b35d46 >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 08:50 >> /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/228= eee22c32e458e8eb7f5d031f64b58 >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 08:50 >> /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/529= 257432308466f971e41db49ecffdf >> -rw-r--r-- 3 akmal supergroup 638.5 K 2015-05-10 08:50 >> /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/839= d3a6fc523435d8b44f63315fd11b8 >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 08:50 >> /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/8c2= 45e8661b140439e719f69a535d57f >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 11:37 >> /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/234= 97a31d3e721fe9b63c58fbe0224d5 >> -rw-r--r-- 3 akmal supergroup 638.7 K 2015-05-10 11:37 >> /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/8c9= af0357d164221ad46b336cd660b30 >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 11:37 >> /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/8eb= 55b43c22d434954e2e0bfda656018 >> -rw-r--r-- 3 akmal supergroup 1.0 K 2015-05-10 11:37 >> /hbase/archive/data/default/table1/0e8e3bf44a2ea5dfaa8a9c58d99b92e6/c/b8b= 6210d9e6d4ec2344238c6e9c17ddf >>=20 >> As I understood this files were copied to archive folder after compaction= . >> The part I didn=92t understand is, why the file with 638 K was also selec= ted >> for compaction? >> Any ideas? >> Thank you. >>=20 >> Kind regards, >> Akmal Abbasov >>=20 >>=20 >>=20