Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 43AA310ED2 for ; Wed, 15 Jan 2014 13:49:43 +0000 (UTC) Received: (qmail 7115 invoked by uid 500); 15 Jan 2014 13:49:42 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 6801 invoked by uid 500); 15 Jan 2014 13:49:39 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 6778 invoked by uid 99); 15 Jan 2014 13:49:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jan 2014 13:49:38 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,LOTS_OF_MONEY,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of eric.newton@gmail.com designates 209.85.216.174 as permitted sender) Received: from [209.85.216.174] (HELO mail-qc0-f174.google.com) (209.85.216.174) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jan 2014 13:49:34 +0000 Received: by mail-qc0-f174.google.com with SMTP id x13so938602qcv.19 for ; Wed, 15 Jan 2014 05:49:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=vFGYlSXDwJblGLGdHLldgbu622GCtiQuJ7HHCaOIb8k=; b=VsaFXc8KmIgFTylH0Mi08UwuDLmM/vnidOMfKNqVy8bn+stXtIk08zn+7cI82aCCn7 DNwfyxgUv5fYqNuCu1fRGOdPvaFp299krVTf9HiuhVWyzdeSRWximYIKw+/IvYpEednT 8gIqOdYTo7onU68ckpb6W0pbsm9ExslOEuSVMdsiDSIhTaPUIKoLfIPOYmcwnSWyVImP iivJU5s8MvzvE/wfFDoAzGqz5jqp1Dx7QvS0XNZknjVUkN1LojqYVtJ4+RDsVvbdiII2 xciCYl7b/iKUM2D+Q3EWCH+EjJGFj/u2/lP1wmCKPSYkH9PZzXmRorhjRKOefDJV0/Jp +UcA== MIME-Version: 1.0 X-Received: by 10.224.30.133 with SMTP id u5mr4487593qac.47.1389793753705; Wed, 15 Jan 2014 05:49:13 -0800 (PST) Received: by 10.96.149.229 with HTTP; Wed, 15 Jan 2014 05:49:13 -0800 (PST) In-Reply-To: References: Date: Wed, 15 Jan 2014 08:49:13 -0500 Message-ID: Subject: Re: Bulk ingest losing tablet server From: Eric Newton To: "user@accumulo.apache.org" Content-Type: multipart/alternative; boundary=047d7bea44520037aa04f002934d X-Virus-Checked: Checked by ClamAV on apache.org --047d7bea44520037aa04f002934d Content-Type: text/plain; charset=ISO-8859-1 When a tablet server (lets call it A) bulk imports a file, it makes a few bookkeeping entries in the !METADATA table. The tablet server that is serving the !METADATA table (lets call it B) checks a constraint: tablet server A must still have its zookeeper lock. This constraint is being violated because A has lost its lock. Tablet server A should have died. The native map is used for live data ingest and exist outside of the java heap. The caches live in the heap. -Eric On Wed, Jan 15, 2014 at 8:19 AM, Anthony F wrote: > Just checked on the native mem maps . . . looks like it is set to 1GB. Do > the index and data caches reside in native mem maps if available or is > native mem used for something else? > > I just repeated an ingest . . . this time I did not lose any tablet > servers but my logs are filling up with the following messages: > > 2014-01-15 08:16:41,643 [constraints.MetadataConstraints] DEBUG: violating > metadata mutation : b;74~thf > 2014-01-15 08:16:41,643 [constraints.MetadataConstraints] DEBUG: update: > file:/b-00012bq/I00012cj.rf value 20272720,0,1389757684543 > 2014-01-15 08:16:41,643 [constraints.MetadataConstraints] DEBUG: update: > loaded:/b-00012bq/I00012cj.rf value 2675766456963732003 > 2014-01-15 08:16:41,643 [constraints.MetadataConstraints] DEBUG: update: > srv:time value M1389757684543 > 2014-01-15 08:16:41,643 [constraints.MetadataConstraints] DEBUG: update: > srv:lock value tservers/ > 192.168.2.231:9997/zlock-0000000002$2438da698db13b4 > > > > On Mon, Jan 13, 2014 at 2:44 PM, Sean Busbey wrote: > >> >> On Mon, Jan 13, 2014 at 12:02 PM, Anthony F wrote: >> >>> Yes, system swappiness is set to 0. I'll run again and gather more logs. >>> >>> Is there a zookeeper timeout setting that I can adjust to avoid this >>> issue and is that advisable? Basically, the tservers are colocated with >>> HDFS datanodes and Hadoop nodemanagers. The machines are overallocated in >>> terms of RAM. So, I have a feeling that when a map-reduce job is kicked >>> off, it causes the tserver to page out to swap space. Once the map-reduce >>> job finishes and the bulk ingest is kicked off, the tserver is paged back >>> in and the ZK timeout causes a shutdown. >>> >>> >>> >> You should not overallocate the amount of memory on the machines. >> Generally, you should provide memory limits under teh assumption that >> everything will be on at once. >> >> Many parts of Hadoop (not just Accumulo) will degrade or malfunction in >> the presence of memory swapping. >> >> How much of hte 12GB for Accumulo is for native memmaps? >> > > --047d7bea44520037aa04f002934d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
When a tablet server (lets call it A) bulk imports a file,= it makes a few bookkeeping entries in the !METADATA table. The tablet serv= er that is serving the !METADATA table (lets call it B) checks a constraint= : tablet server A must still have its zookeeper lock. =A0This constraint is= being violated because A has lost its lock.

Tablet server A should have died.

=
The native map is used for live data ingest and exist outs= ide of the java heap. =A0The caches live in the heap.

-Eric


On Wed, Jan 15, 2014 at 8:19 AM, Anthony F <afccri@g= mail.com> wrote:
Just checked on the native = mem maps . . . looks like it is set to 1GB. =A0Do the index and data caches= reside in native mem maps if available or is native mem used for something= else?

I just repeated an ingest . . . this time I did not lose any= tablet servers but my logs are filling up with the following messages:

2014-01-15 08:16:41,643 [constraints.MetadataConst= raints] DEBUG: violating metadata mutation : b;74~thf
2014-01-15 = 08:16:41,643 [constraints.MetadataConstraints] DEBUG: =A0update: file:/b-00= 012bq/I00012cj.rf value 20272720,0,1389757684543
2014-01-15 08:16:41,643 [constraints.MetadataConstraints] DEBUG: =A0up= date: loaded:/b-00012bq/I00012cj.rf value 2675766456963732003
201= 4-01-15 08:16:41,643 [constraints.MetadataConstraints] DEBUG: =A0update: sr= v:time value M1389757684543
2014-01-15 08:16:41,643 [constraints.MetadataConstraints] DEBUG: =A0up= date: srv:lock value tservers/192.168.2.231:9997/zlock-000000= 0002$2438da698db13b4



On Mon, Jan 13, 2014 = at 2:44 PM, Sean Busbey <busbey+lists@cloudera.com> = wrote:

On Mon, Jan 13, 2014 at 12:02 PM, Antho= ny F <afccri@gmail.com> wrote:
Yes, system swappiness is s= et to 0. =A0I'll run again and gather more logs.

Is = there a zookeeper timeout setting that I can adjust to avoid this issue and= is that advisable? =A0Basically, the tservers are colocated with HDFS data= nodes and Hadoop nodemanagers. =A0The machines are overallocated in terms o= f RAM. =A0So, I have a feeling that when a map-reduce job is kicked off, it= causes the tserver to page out to swap space. =A0Once the map-reduce job f= inishes and the bulk ingest is kicked off, the tserver is paged back in and= the ZK timeout causes a shutdown.



You should not overallocate the amount of = memory on the machines. Generally, you should provide memory limits under t= eh assumption that everything will be on at once.

Many parts of Hadoop (not just Accumulo) will degrade o= r malfunction in the presence of memory swapping.

= How much of hte 12GB for Accumulo is for native memmaps?=A0


--047d7bea44520037aa04f002934d--