Return-Path: 
 <user-return-3736-apmail-accumulo-user-archive=accumulo.apache.org@accumulo.apache.org>
X-Original-To: apmail-accumulo-user-archive@www.apache.org
Delivered-To: apmail-accumulo-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 321B310EC9
	for <apmail-accumulo-user-archive@www.apache.org>;
 Mon, 17 Feb 2014 00:59:54 +0000 (UTC)
Received: (qmail 62928 invoked by uid 500); 17 Feb 2014 00:59:52 -0000
Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org
Received: (qmail 62876 invoked by uid 500); 17 Feb 2014 00:59:51 -0000
Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@accumulo.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@accumulo.apache.org>
List-Post: <mailto:user@accumulo.apache.org>
List-Id: <user.accumulo.apache.org>
Reply-To: user@accumulo.apache.org
Delivered-To: mailing list user@accumulo.apache.org
Received: (qmail 62862 invoked by uid 99); 17 Feb 2014 00:59:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Feb 2014 00:59:51 +0000
X-ASF-Spam-Status: No, hits=-5.0 required=5.0
	tests=RCVD_IN_DNSWL_HI,SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of
 prvs=1118b8ef5b=matt.dickson@defence.gov.au designates 203.6.68.1 as
 permitted sender)
Received: from [203.6.68.1] (HELO defence.gov.au) (203.6.68.1)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Feb 2014 00:59:46 +0000
From: "Dickson, Matt MR" <matt.dickson@defence.gov.au>
To: "'user@accumulo.apache.org'" <user@accumulo.apache.org>
Date: Mon, 17 Feb 2014 11:59:16 +1100
Subject: RE: Failing to BulkIngest [SEC=UNOFFICIAL]
Thread-Topic: Failing to BulkIngest [SEC=UNOFFICIAL]
Thread-Index: Ac8rd8YsEBgpdaXMTVGL5mAfvjMwigAAKudA
Message-ID: <24070BEF0A3F684489AA943FD3439EF205ACA78445@CARRXM06.drn.mil.au>
References: <24070BEF0A3F684489AA943FD3439EF205ACA78444@CARRXM06.drn.mil.au>
 <53015887.8070606@gmail.com>
In-Reply-To: <53015887.8070606@gmail.com>
Accept-Language: en-US, en-AU
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-protective-marking: VER=2012.3, NS=gov.au, SEC=UNOFFICIAL,
 ORIGIN=matt.dickson@defence.gov.au
x-tituslabs-classifications-30: TLPropertyRoot=Titus;SEC=UNOFFICIAL;
x-tituslabs-classificationhash-30: 
 VgNFIFU9Hx+/nZJb9Kg7IlN0CMTbBDRmxMA7ZmKK2DefFPwHVDfnvjSKkQanxyUkeclisG+BxoKurT399K9ubMBYYgjyQRmy5W4pl3njDjct19txn7Gae019bdcKA7FXZlaO2m2bNK2ERBBHIUy+yQkxcMmxvh+kNCooq5+68C3+uqtnWDDWJTWnjrd7me52j3S2mZw86U4xiO+waojIYw==
x-titus-version: 3.5.8.4
x-tituslabs-subjectpostlabel: [SEC=UNOFFICIAL]
acceptlanguage: en-US, en-AU
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 17 Feb 2014 00:59:20.0665 (UTC)
 FILETIME=[7DAF5C90:01CF2B7B]
X-TM-AS-Product-Ver: SMEX-10.2.0.3262-7.500.1017-20504.004
X-TM-AS-Result: No--7.450100-8.000000-31
X-TM-AS-User-Approved-Sender: No
X-TM-AS-User-Blocked-Sender: No
X-Virus-Checked: Checked by ClamAV on apache.org

UNOFFICIAL

Josh,

Zookeepr - 3.4.5-cdh4.3.0
Accumulo - 1.5.0
Hadoop - cdh 4.3.0

In the accumulo console getting

ERROR RemoteException(...LeaseExpiredException): Lease mismatch on /accumul=
o/wal/<ip>+9997/<uid> owned by DFSClient_NONMAPREDUCE_699577321_12 but is a=
ccessed by DFSClient_NONMAPREDUCE_903051502_12

We can scan the table without issues and can load rows directly, ie not usi=
ng bulk import. =20

A bit more information - we recently extended how we manage old tablets in =
the system. We load data by date, creating splits for each day and then age=
off using the ageoff filters.  This leaves empty tablets so we now merge th=
ese old tablets together to effectively remove them.  I mention it because =
I'm not sure if this might have introduced another issue. =20

Matt

-----Original Message-----
From: Josh Elser [mailto:josh.elser@gmail.com]
Sent: Monday, 17 February 2014 11:32
To: user@accumulo.apache.org
Subject: Re: Failing to BulkIngest [SEC=3DUNOFFICIAL]

Matt,

Can you provide Hadoop, ZK and Accumulo versions? Does the cluster appear t=
o be functional otherwise (can you scan that table you're bulk importing to=
? any other errors on the monitor? etc)

On 2/16/14, 7:07 PM, Dickson, Matt MR wrote:
> *UNOFFICIAL*
>
> I have a situation where bulk ingests are failing with a "Thread "shell"
> stuck on IO to xxx:9999:99999 ...
>  From the management console the table we are loading to has no=20
> compactions running, yet we ran "./accumulo=20
> org.apache.accumulo.server.fate.Admin print and can see 50,000 lines=20
> stating
> txid: xxxx     status:IN_PROGRESS op: CompactRange     locked: []
> locking: [R:3n]     top: Compact:Range
> Does this mean there are actually compactions running or old=20
> comapaction locks still hanging around that will be preventing the builk =
ingest to run?
> Thanks in advance,
> Matt