Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 82CA1200B71 for ; Wed, 31 Aug 2016 16:38:07 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 80084160AB4; Wed, 31 Aug 2016 14:38:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C6F61160AA7 for ; Wed, 31 Aug 2016 16:38:06 +0200 (CEST) Received: (qmail 97123 invoked by uid 500); 31 Aug 2016 14:38:05 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 97083 invoked by uid 99); 31 Aug 2016 14:38:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Aug 2016 14:38:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id A165AC9C0B for ; Wed, 31 Aug 2016 14:38:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.878 X-Spam-Level: * X-Spam-Status: No, score=1.878 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=d2l.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id fWic5godTB6b for ; Wed, 31 Aug 2016 14:38:02 +0000 (UTC) Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0070.outbound.protection.outlook.com [104.47.33.70]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id DEC525F239 for ; Wed, 31 Aug 2016 14:38:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=d2l.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=cYxfrkd6fl0b55jJ238b/KJ3WgSLhsJSzNpm1L9/vdw=; b=JdQjSzPo5yfJ57aaDgyvFX0uaM6J90j4Cb98jYB/OCnSLskbYfkvIyW2aimQ7WwZxXBgbWLLhr+V8g3UBWXFV0kyvI8tQs9Ri/7OuFfH0supejyFmHhSkFQoM2y3bnLAMRyeBvpuKYKdgnuQ4etVGXF4tASZBqprndqUE4NR21I= Received: from SN1PR0101MB1503.prod.exchangelabs.com (10.163.128.152) by SN1PR0101MB1501.prod.exchangelabs.com (10.163.128.150) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.587.13; Wed, 31 Aug 2016 14:37:52 +0000 Received: from SN1PR0101MB1503.prod.exchangelabs.com ([10.163.128.152]) by SN1PR0101MB1503.prod.exchangelabs.com ([10.163.128.152]) with mapi id 15.01.0587.009; Wed, 31 Aug 2016 14:37:52 +0000 From: Mark Tse To: "user@hbase.apache.org" Subject: Can adding additional DataNodes / RegionServers cause a spike in RegionServer requests? Thread-Topic: Can adding additional DataNodes / RegionServers cause a spike in RegionServer requests? Thread-Index: AdIDlRuk6cDDbXY3S+WXD/AIwrS52w== Date: Wed, 31 Aug 2016 14:37:51 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Mark.Tse@D2L.com; x-originating-ip: [216.16.228.6] x-ms-office365-filtering-correlation-id: 03947ed2-1957-433c-0fce-08d3d1ac6352 x-microsoft-exchange-diagnostics: 1;SN1PR0101MB1501;6:lXoWFEduCy0mL+JWOdIQbqE51zO0IxC4NcNMOMZfPr4W3BFVgLb4VdW4gLICe253NX9XPeovhBpGwlPEn97GcD2KAHbrQPGsDplXN1B0KNj77zayRtVtSmh1hKuIl6sbc7qbxrGzHkgwLIcJOjG2ZfuyvkfAOAjCftbTeNGmP0ADSPCbMq2MP/Q73cKLwHkopfsWZB5pVOl927OFFacir1tJCmT+xpyNPMXxSIwIRLO27g8WNeaCGRB/OQoxBUiMVdMnUomMP7jNoqcPFzGHfx/GUNxH+yselvfo3F0hwCs=;5:VEX6MBeIK3kXchjDPYC835eAk9CwBos2b1ua48TPQCMs6J6ZmMXuE2/pTTi8+DNUwbGUD8FDceuSMEWkAJ9DEdLgnRKDcQs0JWZI17pXmJA5O6VJ3JantpBlUL5JrnU7QMRXUNOpkK/M1hxYtZzCcA==;24:QDEnQlcvmxp78uOlZfL91QG4dfBNkuPnAQ3N/lDxFu78PmAA5/tBMp+GUSGzG83SW7Coow/UvMvvlJr+mL1v4Y6xle+MpPHMqdwlHqBhCHo=;7:55ZSCA0GJEe0HHXV7DQnYi40+/KS3zCEa34feBkB+XkFIACGX4wUhTYA8g5M/y+nCHgr0rrl1ZVH2XyrzfpVoYpIdJya57Sdoccjo70ASXwdsCHBJZFwc8yq0pGUmVubdpjUl2WOkSWxc5Vevl6cYcRoLFPORpNCK0EE408tH352/9XLBw1+FtAlynLut2c8aldPXnu/eA5Odxp2a4uWilxW4lgIJbg5mO2knqioY7n/Yc6XiN15pEzm/qNCvIYX x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:SN1PR0101MB1501; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(209352067349851)(21748063052155); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040176)(2401047)(8121501046)(5005006)(3002001)(10201501046);SRVR:SN1PR0101MB1501;BCL:0;PCL:0;RULEID:;SRVR:SN1PR0101MB1501; x-forefront-prvs: 00514A2FE6 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(6009001)(7916002)(199003)(189002)(92566002)(5002640100001)(86362001)(9686002)(586003)(2900100001)(81156014)(3660700001)(15975445007)(8676002)(8936002)(81166006)(74316002)(54356999)(450100001)(2906002)(7696003)(68736007)(33656002)(7846002)(1730700003)(7736002)(16236675004)(101416001)(2501003)(122556002)(105586002)(50986999)(5660300001)(77096005)(19300405004)(189998001)(10400500002)(3846002)(66066001)(2351001)(87936001)(229853001)(107886002)(102836003)(19580395003)(6116002)(110136002)(790700001)(97736004)(11100500001)(19625215002)(3280700002)(106356001);DIR:OUT;SFP:1101;SCL:1;SRVR:SN1PR0101MB1501;H:SN1PR0101MB1503.prod.exchangelabs.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; received-spf: None (protection.outlook.com: D2L.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/alternative; boundary="_000_SN1PR0101MB1503AF1122B9BF2B91487599D6E30SN1PR0101MB1503_" MIME-Version: 1.0 X-OriginatorOrg: d2l.com X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Aug 2016 14:37:51.7792 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 74bbca6d-410b-45b3-9b51-2a6aa6477079 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR0101MB1501 archived-at: Wed, 31 Aug 2016 14:38:07 -0000 --_000_SN1PR0101MB1503AF1122B9BF2B91487599D6E30SN1PR0101MB1503_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I have an HBase cluster with 4 RegionServers that was serving ~3600 request= s per second. After adding two RegionServers / DataNodes at once, two of th= e pre-existing RegionServers went up to ~55000 and ~96000 requests per seco= nd, and totaling about ~153000 requests per second across all 6 RegionServe= rs. The newly added RegionServers had the lowest requests per second. After leaving it overnight, the total number of requests has gone back down= to ~3600 requests per second across all 6 RegionServers. As well, HDFS has= distributed the blocks evenly so that the new DataNodes now host a similar= amount of blocks as the pre-existing DataNodes. Not sure if relevant, but the "hbase:meta" table regions was not hosted on = one of the RegionServers that had the spike in requests per second. Is it normal behaviour for existing RegionServers to have to work harder wh= en new RegionServers come online? If so, what is the reason for the (massiv= e) increase? Thanks, Mark --_000_SN1PR0101MB1503AF1122B9BF2B91487599D6E30SN1PR0101MB1503_--