Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2C6BC183E5 for ; Sat, 24 Oct 2015 03:08:34 +0000 (UTC) Received: (qmail 24764 invoked by uid 500); 24 Oct 2015 03:08:32 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 24693 invoked by uid 500); 24 Oct 2015 03:08:32 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 24679 invoked by uid 99); 24 Oct 2015 03:08:31 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Oct 2015 03:08:31 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 13E35C6393 for ; Sat, 24 Oct 2015 03:08:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.901 X-Spam-Level: ** X-Spam-Status: No, score=2.901 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 3-P4_o3wlChg for ; Sat, 24 Oct 2015 03:08:24 +0000 (UTC) Received: from mail-ig0-f171.google.com (mail-ig0-f171.google.com [209.85.213.171]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 01A7120634 for ; Sat, 24 Oct 2015 03:08:24 +0000 (UTC) Received: by igbkq10 with SMTP id kq10so45787920igb.0 for ; Fri, 23 Oct 2015 20:08:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=ltbHts7s4nhYd+nU0kgvDkIUQjfR0jDCCqbRJ+SGTHc=; b=d4+pTZQsKGmsndrU95jI7Bjczjyk7svCERVDMJMDOdDcpavhJZwDnLGiWxA6pAjnkC QvFLKefncL2Q3fo+hmz5j0WBj4eSo/oi7LyvgwE8fNXBZwhOYJ3uH67CyMoqeYGFAwC6 WgP9WIvFIFecmIpONLnQOjhlrA2MNO9sE0Gg4tZ9GbF3BvyiRUdplLrc9RS+Q/8rUQ7A tBQ4J2Blw7tOqnWMAhyR4VNHX7KdXfcX/7tbCAB2MCxS52kDlkKCUk91zfymNshqMC3w SmoR+8oDTc1ujjYHdMLQlOJkI0DZ++19KsFQMg3GhHfLsnw4mdtYYWTOcQCnjS5suqGY cD2Q== MIME-Version: 1.0 X-Received: by 10.50.67.79 with SMTP id l15mr8303998igt.9.1445656096986; Fri, 23 Oct 2015 20:08:16 -0700 (PDT) Received: by 10.107.157.82 with HTTP; Fri, 23 Oct 2015 20:08:16 -0700 (PDT) In-Reply-To: References: Date: Fri, 23 Oct 2015 23:08:16 -0400 Message-ID: Subject: Re: Disk usage drops after RegionServer restart? (0.98) From: =?UTF-8?Q?Otis_Gospodneti=C4=87?= To: "user@hbase.apache.org" Content-Type: multipart/alternative; boundary=047d7bd7571020dc3d0522d10a10 --047d7bd7571020dc3d0522d10a10 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Ted, 0.98.6-cdh5.3.0 I did actually try to use lsof, but I didn't see anything unusual there. Is there something specific I should look for? Things owned by hbase user or hdfs or yarn? Hm, here, I don't really see anything interesting $ sudo lsof| grep '/mnt' <=3D=3D this is where all data lives and where dis= k usage drops after RS restart java 2654 hdfs 1w REG 202,16 89487 44042562 /mnt/hadoop-hdfs/log/hadoop-hdfs-datanode-spm-hbase-slave11.prod.sematext.o= ut java 2654 hdfs 2w REG 202,16 89487 44042562 /mnt/hadoop-hdfs/log/hadoop-hdfs-datanode-spm-hbase-slave11.prod.sematext.o= ut java 2654 hdfs 286w REG 202,16 108938205 44044137 /mnt/hadoop-hdfs/log/hadoop-hdfs-datanode-spm-hbase-slave11.prod.sematext.l= og java 2654 hdfs 289w REG 202,16 0 44040203 /mnt/hadoop-hdfs/log/SecurityAuth-hdfs.audit java 2654 hdfs 314w REG 202,16 261462 44040213 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/dnc= p_block_verification.log.curr java 2654 hdfs 316r REG 202,16 134217728 44045060 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir74/subdir58/blk_1078606358 java 2654 hdfs 318r REG 202,16 134217728 44057015 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir74/subdir224/blk_1078648930 java 2654 hdfs 319uW REG 202,16 36 44042741 /mnt/hadoop-hdfs/data/in_use.lock java 2654 hdfs 321r REG 202,16 1048583 44042793 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir7/blk_1078658889_4918820.meta java 2654 hdfs 330u REG 202,16 352563 44048279 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/rbw/blk_1078675432_4935363.meta java 2654 hdfs 333r REG 202,16 134217728 44055769 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir9/blk_1078659381 java 2654 hdfs 335u REG 202,16 45127168 44048273 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/rbw/blk_1078675432 java 2654 hdfs 340r REG 202,16 134217728 44042791 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir7/blk_1078658889 java 2654 hdfs 343r REG 202,16 13882119 44048053 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir71/blk_1078675385 java 2654 hdfs 345u REG 202,16 485059 44048209 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/rbw/blk_1078675399_4935330.meta java 2654 hdfs 346r REG 202,16 134217728 44053723 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir4/blk_1078658098 java 2654 hdfs 347u REG 202,16 371455 44047931 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/rbw/blk_1078675364_4935295.meta java 2654 hdfs 348u REG 202,16 47545282 44047927 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/rbw/blk_1078675364 java 2654 hdfs 354u REG 202,16 20386405 44047875 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir8/blk_1078659266 java 2654 hdfs 355r REG 202,16 134217728 44042762 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir74/subdir243/blk_1078653797 java 2654 hdfs 357r REG 202,16 134217728 44042535 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir66/blk_1078674123 java 2654 hdfs 359u REG 202,16 1839 44045445 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/rbw/blk_1078674506_4934437.meta java 2654 hdfs 360u REG 202,16 234130 44045440 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/rbw/blk_1078674506 java 2654 hdfs 363r REG 202,16 20629437 44046774 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir17/blk_1078661533 java 2654 hdfs 369r REG 202,16 18304945 44047599 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir71/blk_1078675270 java 2654 hdfs 370r REG 202,16 62086413 44048199 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/rbw/blk_1078675399 java 2654 hdfs 379r REG 202,16 134217728 44050035 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir3/blk_1078657983 java 2654 hdfs 390u REG 202,16 20857780 44050270 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir8/blk_1078659267 java 2654 hdfs 408r REG 202,16 115453375 44042299 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir66/blk_1078674120 java 2654 hdfs 415r REG 202,16 20253192 44053520 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir60/blk_1078672624 java 2654 hdfs 423r REG 202,16 18382878 44047547 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir71/blk_1078675257 java 2654 hdfs 424r REG 202,16 19555559 44040692 /mnt/hadoop-hdfs/data/current/BP-282774069-10.123.212.150-1419335230604/cur= rent/finalized/subdir75/subdir65/blk_1078673801 bash 15005 ec2-user cwd DIR 202,16 4096 2 /mnt sudo 16055 root cwd DIR 202,16 4096 2 /mnt grep 16056 ec2-user cwd DIR 202,16 4096 2 /mnt sed 16057 ec2-user cwd DIR 202,16 4096 2 /mnt lsof 16058 root cwd DIR 202,16 4096 2 /mnt lsof 16059 root cwd DIR 202,16 4096 2 /mnt bash 18748 hbase 1w REG 202,16 12843 4980744 /mnt/hbase/log/hbase-hbase-regionserver-spm-hbase-slave11.prod.sematext.out bash 18748 hbase 2w REG 202,16 12843 4980744 /mnt/hbase/log/hbase-hbase-regionserver-spm-hbase-slave11.prod.sematext.out java 18761 hbase 1w REG 202,16 12843 4980744 /mnt/hbase/log/hbase-hbase-regionserver-spm-hbase-slave11.prod.sematext.out java 18761 hbase 2w REG 202,16 12843 4980744 /mnt/hbase/log/hbase-hbase-regionserver-spm-hbase-slave11.prod.sematext.out java 18761 hbase 338w REG 202,16 117537786 4980753 /mnt/hbase/log/hbase-hbase-regionserver-spm-hbase-slave11.prod.sematext.log java 18761 hbase 339w REG 202,16 0 4980741 /mnt/hbase/log/SecurityAuth.audit java 29057 yarn 1w REG 202,16 130105 51380228 /mnt/hadoop-yarn/log/yarn-yarn-nodemanager-spm-hbase-slave11.prod.sematext.= out java 29057 yarn 2w REG 202,16 130105 51380228 /mnt/hadoop-yarn/log/yarn-yarn-nodemanager-spm-hbase-slave11.prod.sematext.= out java 29057 yarn 286w REG 202,16 103611255 51380852 /mnt/hadoop-yarn/log/yarn-yarn-nodemanager-spm-hbase-slave11.prod.sematext.= log I don't see anything big there... Thanks, Otis -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ On Fri, Oct 23, 2015 at 10:26 PM, Ted Yu wrote: > Which specific release of 0.98 are you using ? > > Have you used lsof to see which files were being held onto ? > > Thanks > > On Fri, Oct 23, 2015 at 7:21 PM, Otis Gospodneti=C4=87 < > otis.gospodnetic@gmail.com> wrote: > > > Hello, > > > > Is/was there a known issue with HBase 0.98 "holding onto" files? > > > > We noticed the used disk space metric going up, up and up and we could > not > > stop it with major compaction. > > But we noticed that if we restart a RegionServer 2 things happen: > > 1) its disk usage immediately drops a lot > > 2) the disk usage of other RegionServers drops some as well > > > > Have a look at this chart: > > https://apps.sematext.com/spm-reports/s/Ssy4ViFGHq > > > > At 1:54 we restarted the first RS (blue line) > > At 2:03 we restarted the second RS (dark green line) > > > > Is/was this a known HBase 0.98 issue? > > > > Thanks, > > Otis > > -- > > Monitoring - Log Management - Alerting - Anomaly Detection > > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > --047d7bd7571020dc3d0522d10a10--