Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (nike.apache.org: domain of
 prvs=195018b424=matt.dickson@defence.gov.au designates 203.6.68.1 as
 permitted sender)
From: "Dickson, Matt MR" <matt.dickson@defence.gov.au>
To: "'user@accumulo.apache.org'" <user@accumulo.apache.org>
Date: Mon, 2 Sep 2013 12:59:08 +1000
Subject: RE: High Ingest on a single server [SEC=UNOFFICIAL]
Thread-Topic: High Ingest on a single server [SEC=UNOFFICIAL]
Thread-Index: Ac6nh3p5UL0kCDBhS6mzMUpl6vRQ8QAAKTXw
Message-ID: <24070BEF0A3F684489AA943FD3439EF20586FA05E3@CARRXM06.drn.mil.au>
References: <24070BEF0A3F684489AA943FD3439EF2058102AB9E@CARRXM06.drn.mil.au>
 <CADxc9Bm9LvDb9sZXn5HnOzvHOBUX33whfYOELpxWHD2m6cp+ZA@mail.gmail.com>
 <24070BEF0A3F684489AA943FD3439EF20586FA05E1@CARRXM06.drn.mil.au>
 <CADczPYTLO-dvRtVs4LJrw1kvLHKS4_Gevo8f9i=qpvGKFO3LmQ@mail.gmail.com>
 <5223FD58.9030204@gmail.com>
In-Reply-To: <5223FD58.9030204@gmail.com>
Accept-Language: en-US, en-AU
Content-Language: en-US
acceptlanguage: en-US, en-AU
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

UNOFFICIAL

Just checked and there are a lot of 'not balancing because there are unhost=
ed tablets' debug messages.  Is this the same issue?

-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com]
Sent: Monday, 2 September 2013 12:52
To: user@accumulo.apache.org
Subject: Re: High Ingest on a single server [SEC=3DUNOFFICIAL]

To verify that this what you're running into, you should see a message in t=
he master*.debug.log that matches "not balancing because.*"

On 09/01/2013 09:54 PM, John Vines wrote:
> Try restarting the master. A few releases had a big where it would get=20
> stuck.
>
> Sent from my phone, please pardon the typos and brevity.
>
> On Sep 1, 2013 6:12 PM, "Dickson, Matt MR"=20
> <matt.dickson@defence.gov.au <mailto:matt.dickson@defence.gov.au>> wrote:
>
>     __
>
>     *UNOFFICIAL*
>
>     Thanks Eric.
>     The tablet of concern has 2000 tablets while all others have 1000 so
>     it looks like the balancers aren't evening out the tablets per node
>     as expected.
>     Is there a way to force the balancer to run or rectify this
>     situation by moving tablets to alternate nodes?
>
>     ---------------------------------------------------------------------=
---
>     *From:* Eric Newton [mailto:eric.newton@gmail.com
>     <mailto:eric.newton@gmail.com>]
>     *Sent:* Thursday, 29 August 2013 23:23
>     *To:* user@accumulo.apache.org <mailto:user@accumulo.apache.org>
>     *Subject:* Re: High Ingest on a single server [SEC=3DUNOFFICIAL]
>
>     The balancers that ship with accumulo attempt to keep an equal
>     number of tablets on each server.  An empty tablet, will be balanced
>     with the same weight as a 50G tablet.
>
>     You can write a new balancer to take advantage of the properties of
>     the tablets, and any expected hotspots you have.
>
>
>
>     On Thu, Aug 29, 2013 at 1:39 AM, Dickson, Matt MR
>     <matt.dickson@defence.gov.au <mailto:matt.dickson@defence.gov.au>>
>     wrote:
>
>         __
>
>         *UNOFFICIAL*
>
>         We are seeing a single server that has less entries than all the
>         other nodes in the cluster.  Accumulo now appears to be
>         directing higher ingest tablets to this node and its now getting
>         7 times the ingest entries than all other nodes and is slowing
>         or load.  Does Accumulo attempt to balance disk usage across the
>         nodes for a table by moving tablets and that is why we are
>         seeing this node ingesting more?
>         If not, is it possible to make accumulo rebalance the ingest
>         across all servers during a load?
>         Matt
>
>