From user-return-7231-archive-asf-public=cust-asf.ponee.io@accumulo.apache.org  Fri Feb  9 20:26:58 2018
Return-Path: <user-return-7231-archive-asf-public=cust-asf.ponee.io@accumulo.apache.org>
X-Original-To: archive-asf-public@eu.ponee.io
Delivered-To: archive-asf-public@eu.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by mx-eu-01.ponee.io (Postfix) with ESMTP id 30C95180654
	for <archive-asf-public@eu.ponee.io>; Fri,  9 Feb 2018 20:26:58 +0100 (CET)
Received: by cust-asf.ponee.io (Postfix)
	id 203DC160C4C; Fri,  9 Feb 2018 19:26:58 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id E5050160C3C
	for <archive-asf-public@cust-asf.ponee.io>; Fri,  9 Feb 2018 20:26:56 +0100 (CET)
Received: (qmail 4226 invoked by uid 500); 9 Feb 2018 19:26:55 -0000
Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@accumulo.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@accumulo.apache.org>
List-Post: <mailto:user@accumulo.apache.org>
List-Id: <user.accumulo.apache.org>
Reply-To: user@accumulo.apache.org
Delivered-To: mailing list user@accumulo.apache.org
Received: (qmail 4216 invoked by uid 99); 9 Feb 2018 19:26:55 -0000
Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Feb 2018 19:26:55 +0000
Received: from localhost (localhost [127.0.0.1])
	by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 5A93CC00E0
	for <user@accumulo.apache.org>; Fri,  9 Feb 2018 19:26:55 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: 1.979
X-Spam-Level: *
X-Spam-Status: No, score=1.979 tagged_above=-999 required=6.31
	tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2,
	RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01,
	RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled
Authentication-Results: spamd4-us-west.apache.org (amavisd-new);
	dkim=pass (2048-bit key) header.d=phemi-com.20150623.gappssmtp.com
Received: from mx1-lw-us.apache.org ([10.40.0.8])
	by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024)
	with ESMTP id GR8xjsjdNTPn for <user@accumulo.apache.org>;
	Fri,  9 Feb 2018 19:26:51 +0000 (UTC)
Received: from mail-wm0-f53.google.com (mail-wm0-f53.google.com [74.125.82.53])
	by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id A09A65F178
	for <user@accumulo.apache.org>; Fri,  9 Feb 2018 19:26:50 +0000 (UTC)
Received: by mail-wm0-f53.google.com with SMTP id 141so18135684wme.3
        for <user@accumulo.apache.org>; Fri, 09 Feb 2018 11:26:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=phemi-com.20150623.gappssmtp.com; s=20150623;
        h=mime-version:from:date:message-id:subject:to;
        bh=o8vWN5l5SZLPoUKCDplLXVZ9RQUI+W1K9CJ4oFvgn90=;
        b=ezq1doaB1oGbRn3ktxJ+4e4K7HWHZj1CwG7pWTTDni7PnUMhLyO53BXBdlVSiZvvKc
         G7NZY0WM9BBR9j0CWYNNj4ZB02xAbV60n8CYsmL/Hi92srxKKIU3Q2G56YL0JWQJSNAz
         XP4uhqu+PtbPPRLChUSGQqK5seTYcg6HC0sxSsPZ18dFXGxu89VLic6ojI0Zx3BmjOa+
         BgShio1Az0TzK+xJpgERTvbSymcpkpxJlrbEC+4GDf0Ah3mt1RJN+iR42wrTlrOuTmlx
         fDwBXzuHzXdu8JjSOGZHSvs50f4PhlOhhgU9PqTIi8Cf01oInLzwmsHFMshqbY6W0KIF
         blWg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
        bh=o8vWN5l5SZLPoUKCDplLXVZ9RQUI+W1K9CJ4oFvgn90=;
        b=YOCA3vxrllLo8/kS+NBTBJE9C3P4xsTxgjipopHKEmdy+rVlXmPHtxZ2XlQ2h/7N1Z
         C8DJi56f5/4S5PithrD9S7Go+c8nkcegGaseon7LRLqlp1u6t//wiJio3AaoB+3yPUZK
         vwYA33OoIDiDfXo++dL/woqXoXDe5gQBy1vDC/0c4Tkkh8J4Kbyzxtr1hnVbTWk8xk8d
         G90AOOEOHytFDL4DEMljQ+xcVH9UKr9LhLNzadYymqIdiYevnXgI3H1GK+0KKqkWC7t9
         ThfkjrYmPmKl2VZza/y+jmAMnkgLL4YF14j09KwYOO4MNARlMFNhTqk26EJCETquLJKM
         eiXA==
X-Gm-Message-State: APf1xPBTvc2vtUNpGMZFrdJJu6f0tKbMFkTJWx3lkmymcjsXUWSatoDA
	JU6znxsfCsHu72gajfPJl03h9LL61ycF4sAIPnvGSg==
X-Google-Smtp-Source: AH8x225HHWBKBmNtFzl/I995YmnQRnd5f8zw46Pn3XGK3rlD3ozZsedRczRefqx8R7gR+pgFoM8P9WVW89SvSt8KGyI=
X-Received: by 10.80.146.134 with SMTP id k6mr5019838eda.107.1518204408862;
 Fri, 09 Feb 2018 11:26:48 -0800 (PST)
MIME-Version: 1.0
From: Dong Zhou <dzhou@phemi.com>
Date: Fri, 09 Feb 2018 19:26:38 +0000
Message-ID: <CAJQ-rOK5wKQw77Jx6QJ5o9BdmymsF0_V7rWJzC8S_xG=vB2Q-Q@mail.gmail.com>
Subject: Failed to assign map files to tablets during Bulk Import
To: user@accumulo.apache.org
Content-Type: multipart/alternative; boundary="f403045c1c4e7cafec0564cc8256"

--f403045c1c4e7cafec0564cc8256
Content-Type: text/plain; charset="UTF-8"

Hi All,

We were trying to write some data into an Accumulo table that contains
roughly 3.7 Trillion entries using Bulk Import.

Our code generates RFiles in distributed fashion, and most of them go into
the table perfectly, excepting for 1 RFile.

The bulk import failed with following error message. Please note that I
have shortened the error message and also masked out some of the data.

2018-02-05 21:47:07,244 [client.BulkImporter] ERROR: Encountered unknown
exception in assignMapFiles.
org.apache.thrift.TApplicationException: Internal error processing
bulkImport
        at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
        at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_bulkImport(TabletClientService.java:594)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.bulkImport(TabletClientService.java:577)
        at
org.apache.accumulo.server.client.BulkImporter.assignMapFiles(BulkImporter.java:600)
        at
org.apache.accumulo.server.client.BulkImporter.access$400(BulkImporter.java:77)
        at
org.apache.accumulo.server.client.BulkImporter$AssignmentTask.run(BulkImporter.java:479)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
...
...
...
2018-02-05 21:47:07,494 [impl.Writer] ERROR: error sending update to
<tserver ip address>:<tserver port>:
ConstraintViolationException(violationSummaries:[TConstraintViolationSummary(constrainClass:org.apache.accumulo.server.constraints.MetadataConstraints,
violationCode:8, violationDescription:Bulk load transaction no longer
running, numberOfViolatingMutations:1)])
2018-02-05 21:47:07,494 [util.MetadataTableUtil] ERROR: null
ConstraintViolationException(violationSummaries:[TConstraintViolationSummary(constrainClass:org.apache.accumulo.server.constraints.MetadataConstraints,
violationCode:8, violationDescription:Bulk load transaction no longer
running, numberOfViolatingMutations:1)])
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result$update_resultStandardScheme.read(TabletClientService.java:15928)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result$update_resultStandardScheme.read(TabletClientService.java:15896)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result.read(TabletClientService.java:15830)
        at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_update(TabletClientService.java:468)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.update(TabletClientService.java:451)
        at
org.apache.accumulo.core.client.impl.Writer.updateServer(Writer.java:72)
        at
org.apache.accumulo.core.client.impl.Writer.update(Writer.java:98)
        at
org.apache.accumulo.server.util.MetadataTableUtil.update(MetadataTableUtil.java:153)
        at
org.apache.accumulo.server.util.MetadataTableUtil.update(MetadataTableUtil.java:145)
        at
org.apache.accumulo.server.util.MetadataTableUtil.updateTabletDataFile(MetadataTableUtil.java:196)
        at
org.apache.accumulo.tserver.tablet.Tablet.updatePersistedTime(Tablet.java:2657)
        at
org.apache.accumulo.tserver.tablet.DatafileManager.importMapFiles(DatafileManager.java:268)
        at
org.apache.accumulo.tserver.tablet.Tablet.importMapFiles(Tablet.java:2376)
        at
org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.bulkImport(TabletServer.java:435)
        at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
        at
org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:74)
        at com.sun.proxy.$Proxy22.bulkImport(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
        at
org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
        at com.sun.proxy.$Proxy22.bulkImport(Unknown Source)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2585)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2569)
        at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at
org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
        at
org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)
...
...
...
2018-02-05 21:47:07,526 [client.BulkImporter] ERROR: Encountered unknown
exception in assignMapFiles.
org.apache.thrift.TApplicationException: Internal error processing
bulkImport
        at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
        at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_bulkImport(TabletClientService.java:594)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.bulkImport(TabletClientService.java:577)
        at
org.apache.accumulo.server.client.BulkImporter.assignMapFiles(BulkImporter.java:600)
        at
org.apache.accumulo.server.client.BulkImporter.access$400(BulkImporter.java:77)
        at
org.apache.accumulo.server.client.BulkImporter$AssignmentTask.run(BulkImporter.java:479)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)
...
...
...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...

--f403045c1c4e7cafec0564cc8256
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><p>Hi All,</p>
<p>We were trying to write some data into an Accumulo table that contains r=
oughly 3.7 Trillion entries using Bulk Import.</p><p>Our code generates RFi=
les in distributed fashion, and most of them go into the table perfectly, e=
xcepting for 1 RFile.</p>
<p>The bulk import failed with following error message. Please note that I =
have shortened the error message and also masked out some of the data.</p>
<font face=3D"Times New Roman, Times, serif"><div><font face=3D"Times New R=
oman, Times, serif"><br></font></div>2018-02-05 21:47:07,244 [client.BulkIm=
porter] ERROR: Encountered unknown exception in assignMapFiles.</font><br>
<font face=3D"Times New Roman, Times, serif">org.apache.thrift.TApplication=
Exception: Internal error processing bulkImport</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.thrift.TApplicationException.read(TApplicationEx=
ception.java:111)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.thrift.TServiceClient.receiveBase(TServiceClient=
.java:71)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.tabletserver.thrift.TabletClientSe=
rvice$Client.recv_bulkImport(TabletClientService.java:594)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.tabletserver.thrift.TabletClientSe=
rvice$Client.bulkImport(TabletClientService.java:577)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.client.BulkImporter.assignMapFil=
es(BulkImporter.java:600)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.client.BulkImporter.access$400(B=
ulkImporter.java:77)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.client.BulkImporter$AssignmentTa=
sk.run(BulkImporter.java:479)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.util.concurrent.Executors$RunnableAdapter.call(Executo=
rs.java:511)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.util.concurrent.FutureTask.run(FutureTask.java:266)</f=
ont><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoo=
lExecutor.java:1142)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPo=
olExecutor.java:617)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRu=
nnable.java:35)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.lang.Thread.run(Thread.java:745)</font><br>
<font face=3D"Times New Roman, Times, serif">2018-02-05 21:47:07,244 [clien=
t.BulkImporter] INFO : Could not assign 1 map files to tablet o;SESSION_ID|=
****;SESSION_ID|**** because : org.apache.thrift.TApplicationException: Int=
ernal error processing bulkImport .=C2=A0 Will retry ...</font><br>
<font face=3D"Times New Roman, Times, serif">2018-02-05 21:47:07,244 [clien=
t.BulkImporter] INFO : Could not assign 1 map files to tablet o;SESSION_ID|=
****;SESSION_ID|**** because : org.apache.thrift.TApplicationException: Int=
ernal error processing bulkImport .=C2=A0 Will retry ...</font><br>
<font face=3D"Times New Roman, Times, serif">2018-02-05 21:47:07,244 [clien=
t.BulkImporter] INFO : Could not assign 1 map files to tablet o;SESSION_ID|=
****;SESSION_ID|**** because : org.apache.thrift.TApplicationException: Int=
ernal error processing bulkImport .=C2=A0 Will retry ...</font><br>
<font face=3D"Times New Roman, Times, serif">2018-02-05 21:47:07,244 [clien=
t.BulkImporter] INFO : Could not assign 1 map files to tablet o;SESSION_ID|=
****;SESSION_ID|**** because : org.apache.thrift.TApplicationException: Int=
ernal error processing bulkImport .=C2=A0 Will retry ...</font><br>
<font face=3D"Times New Roman, Times, serif">2018-02-05 21:47:07,244 [clien=
t.BulkImporter] INFO : Could not assign 1 map files to tablet o;SESSION_ID|=
****;SESSION_ID|**** because : org.apache.thrift.TApplicationException: Int=
ernal error processing bulkImport .=C2=A0 Will retry ...</font><br>
<font face=3D"Times New Roman, Times, serif">...</font><br>
<font face=3D"Times New Roman, Times, serif">...</font><br>
<font face=3D"Times New Roman, Times, serif">...</font><br>
<font face=3D"Times New Roman, Times, serif">2018-02-05 21:47:07,494 [impl.=
Writer] ERROR: error sending update to &lt;tserver ip address&gt;:&lt;tserv=
er port&gt;: ConstraintViolationException(violationSummaries:[TConstraintVi=
olationSummary(constrainClass:org.apache.accumulo.server.constraints.Metada=
taConstraints, violationCode:8, violationDescription:Bulk load transaction =
no longer running, numberOfViolatingMutations:1)])</font><br>
<font face=3D"Times New Roman, Times, serif">2018-02-05 21:47:07,494 [util.=
MetadataTableUtil] ERROR: null</font><br>
<font face=3D"Times New Roman, Times, serif">ConstraintViolationException(v=
iolationSummaries:[TConstraintViolationSummary(constrainClass:org.apache.ac=
cumulo.server.constraints.MetadataConstraints, violationCode:8, violationDe=
scription:Bulk load transaction no longer running, numberOfViolatingMutatio=
ns:1)])</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.tabletserver.thrift.TabletClientSe=
rvice$update_result$update_resultStandardScheme.read(TabletClientService.ja=
va:15928)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.tabletserver.thrift.TabletClientSe=
rvice$update_result$update_resultStandardScheme.read(TabletClientService.ja=
va:15896)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.tabletserver.thrift.TabletClientSe=
rvice$update_result.read(TabletClientService.java:15830)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.thrift.TServiceClient.receiveBase(TServiceClient=
.java:78)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.tabletserver.thrift.TabletClientSe=
rvice$Client.recv_update(TabletClientService.java:468)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.tabletserver.thrift.TabletClientSe=
rvice$Client.update(TabletClientService.java:451)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.client.impl.Writer.updateServer(Wr=
iter.java:72)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.client.impl.Writer.update(Writer.j=
ava:98)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.util.MetadataTableUtil.update(Me=
tadataTableUtil.java:153)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.util.MetadataTableUtil.update(Me=
tadataTableUtil.java:145)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.util.MetadataTableUtil.updateTab=
letDataFile(MetadataTableUtil.java:196)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.tserver.tablet.Tablet.updatePersistedTi=
me(Tablet.java:2657)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.tserver.tablet.DatafileManager.importMa=
pFiles(DatafileManager.java:268)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.tserver.tablet.Tablet.importMapFiles(Ta=
blet.java:2376)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.tserver.TabletServer$ThriftClientHandle=
r.bulkImport(TabletServer.java:435)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source=
)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingM=
ethodAccessorImpl.java:43)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.lang.reflect.Method.invoke(Method.java:498)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.trace.wrappers.RpcServerInvocation=
Handler.invoke(RpcServerInvocationHandler.java:46)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapp=
er.java:74)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at com.sun.proxy.$Proxy22.bulkImport(Unknown Source)</font><br=
>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source=
)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingM=
ethodAccessorImpl.java:43)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.lang.reflect.Method.invoke(Method.java:498)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocati=
onHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)</fon=
t><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocati=
onHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at com.sun.proxy.$Proxy22.bulkImport(Unknown Source)</font><br=
>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.tabletserver.thrift.TabletClientSe=
rvice$Processor$bulkImport.getResult(TabletClientService.java:2585)</font><=
br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.tabletserver.thrift.TabletClientSe=
rvice$Processor$bulkImport.getResult(TabletClientService.java:2569)</font><=
br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.thrift.ProcessFunction.process(ProcessFunction.j=
ava:39)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.jav=
a:39)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.rpc.UGIAssumingProcessor.process=
(UGIAssumingProcessor.java:102)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.rpc.TimedProcessor.process(Timed=
Processor.java:63)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.ru=
n(TThreadPoolServer.java:225)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoo=
lExecutor.java:1142)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPo=
olExecutor.java:617)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRu=
nnable.java:35)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.lang.Thread.run(Thread.java:745)</font><br>
<font face=3D"Times New Roman, Times, serif">...</font><br>
<font face=3D"Times New Roman, Times, serif">...</font><br>
<font face=3D"Times New Roman, Times, serif">...</font><br>
<font face=3D"Times New Roman, Times, serif">2018-02-05 21:47:07,526 [clien=
t.BulkImporter] ERROR: Encountered unknown exception in assignMapFiles.</fo=
nt><br>
<font face=3D"Times New Roman, Times, serif">org.apache.thrift.TApplication=
Exception: Internal error processing bulkImport</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.thrift.TApplicationException.read(TApplicationEx=
ception.java:111)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.thrift.TServiceClient.receiveBase(TServiceClient=
.java:71)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.tabletserver.thrift.TabletClientSe=
rvice$Client.recv_bulkImport(TabletClientService.java:594)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.core.tabletserver.thrift.TabletClientSe=
rvice$Client.bulkImport(TabletClientService.java:577)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.client.BulkImporter.assignMapFil=
es(BulkImporter.java:600)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.client.BulkImporter.access$400(B=
ulkImporter.java:77)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.server.client.BulkImporter$AssignmentTa=
sk.run(BulkImporter.java:479)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.util.concurrent.Executors$RunnableAdapter.call(Executo=
rs.java:511)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.util.concurrent.FutureTask.run(FutureTask.java:266)</f=
ont><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoo=
lExecutor.java:1142)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPo=
olExecutor.java:617)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRu=
nnable.java:35)</font><br>
<font face=3D"Times New Roman, Times, serif">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at java.lang.Thread.run(Thread.java:745)</font><br>
<font face=3D"Times New Roman, Times, serif">...</font><br>
<font face=3D"Times New Roman, Times, serif">...</font><br>
<font face=3D"Times New Roman, Times, serif">...</font><br>
<font face=3D"Times New Roman, Times, serif">2018-02-05 21:47:08,542 [clien=
t.BulkImporter] INFO : Could not assign 1 map files to tablet o;checksum|**=
**;checksum|**** because : org.apache.thrift.TApplicationException: Interna=
l error processing bulkImport .=C2=A0 Will retry ...</font><br>
<font face=3D"Times New Roman, Times, serif">2018-02-05 21:47:08,542 [clien=
t.BulkImporter] INFO : Could not assign 1 map files to tablet o;checksum|**=
**;checksum|**** because : org.apache.thrift.TApplicationException: Interna=
l error processing bulkImport .=C2=A0 Will retry ...</font><br>
<font face=3D"Times New Roman, Times, serif">2018-02-05 21:47:08,542 [clien=
t.BulkImporter] INFO : Could not assign 1 map files to tablet o;checksum|**=
**;checksum|**** because : org.apache.thrift.TApplicationException: Interna=
l error processing bulkImport .=C2=A0 Will retry ...</font><br>
<font face=3D"Times New Roman, Times, serif">2018-02-05 21:47:08,542 [clien=
t.BulkImporter] INFO : Could not assign 1 map files to tablet o;checksum|**=
**;checksum|**** because : org.apache.thrift.TApplicationException: Interna=
l error processing bulkImport .=C2=A0 Will retry ...</font><br>
<font face=3D"Times New Roman, Times, serif">2018-02-05 21:47:08,542 [clien=
t.BulkImporter] INFO : Could not assign 1 map files to tablet o;checksum|**=
**;checksum|**** because : org.apache.thrift.TApplicationException: Interna=
l error processing bulkImport .=C2=A0 Will retry ...</font><br><div><font f=
ace=3D"Times New Roman, Times, serif"><br></font></div><div><br></div></div=
>

--f403045c1c4e7cafec0564cc8256--