accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dong Zhou <dz...@phemi.com>
Subject Failed to assign map files to tablets during Bulk Import
Date Fri, 09 Feb 2018 19:26:38 GMT
Hi All,

We were trying to write some data into an Accumulo table that contains
roughly 3.7 Trillion entries using Bulk Import.

Our code generates RFiles in distributed fashion, and most of them go into
the table perfectly, excepting for 1 RFile.

The bulk import failed with following error message. Please note that I
have shortened the error message and also masked out some of the data.

2018-02-05 21:47:07,244 [client.BulkImporter] ERROR: Encountered unknown
exception in assignMapFiles.
org.apache.thrift.TApplicationException: Internal error processing
bulkImport
        at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
        at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_bulkImport(TabletClientService.java:594)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.bulkImport(TabletClientService.java:577)
        at
org.apache.accumulo.server.client.BulkImporter.assignMapFiles(BulkImporter.java:600)
        at
org.apache.accumulo.server.client.BulkImporter.access$400(BulkImporter.java:77)
        at
org.apache.accumulo.server.client.BulkImporter$AssignmentTask.run(BulkImporter.java:479)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;SESSION_ID|****;SESSION_ID|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
...
...
...
2018-02-05 21:47:07,494 [impl.Writer] ERROR: error sending update to
<tserver ip address>:<tserver port>:
ConstraintViolationException(violationSummaries:[TConstraintViolationSummary(constrainClass:org.apache.accumulo.server.constraints.MetadataConstraints,
violationCode:8, violationDescription:Bulk load transaction no longer
running, numberOfViolatingMutations:1)])
2018-02-05 21:47:07,494 [util.MetadataTableUtil] ERROR: null
ConstraintViolationException(violationSummaries:[TConstraintViolationSummary(constrainClass:org.apache.accumulo.server.constraints.MetadataConstraints,
violationCode:8, violationDescription:Bulk load transaction no longer
running, numberOfViolatingMutations:1)])
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result$update_resultStandardScheme.read(TabletClientService.java:15928)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result$update_resultStandardScheme.read(TabletClientService.java:15896)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$update_result.read(TabletClientService.java:15830)
        at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_update(TabletClientService.java:468)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.update(TabletClientService.java:451)
        at
org.apache.accumulo.core.client.impl.Writer.updateServer(Writer.java:72)
        at
org.apache.accumulo.core.client.impl.Writer.update(Writer.java:98)
        at
org.apache.accumulo.server.util.MetadataTableUtil.update(MetadataTableUtil.java:153)
        at
org.apache.accumulo.server.util.MetadataTableUtil.update(MetadataTableUtil.java:145)
        at
org.apache.accumulo.server.util.MetadataTableUtil.updateTabletDataFile(MetadataTableUtil.java:196)
        at
org.apache.accumulo.tserver.tablet.Tablet.updatePersistedTime(Tablet.java:2657)
        at
org.apache.accumulo.tserver.tablet.DatafileManager.importMapFiles(DatafileManager.java:268)
        at
org.apache.accumulo.tserver.tablet.Tablet.importMapFiles(Tablet.java:2376)
        at
org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.bulkImport(TabletServer.java:435)
        at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
        at
org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:74)
        at com.sun.proxy.$Proxy22.bulkImport(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
        at
org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
        at com.sun.proxy.$Proxy22.bulkImport(Unknown Source)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2585)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$bulkImport.getResult(TabletClientService.java:2569)
        at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at
org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
        at
org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)
...
...
...
2018-02-05 21:47:07,526 [client.BulkImporter] ERROR: Encountered unknown
exception in assignMapFiles.
org.apache.thrift.TApplicationException: Internal error processing
bulkImport
        at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
        at
org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_bulkImport(TabletClientService.java:594)
        at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.bulkImport(TabletClientService.java:577)
        at
org.apache.accumulo.server.client.BulkImporter.assignMapFiles(BulkImporter.java:600)
        at
org.apache.accumulo.server.client.BulkImporter.access$400(BulkImporter.java:77)
        at
org.apache.accumulo.server.client.BulkImporter$AssignmentTask.run(BulkImporter.java:479)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)
...
...
...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...
2018-02-05 21:47:08,542 [client.BulkImporter] INFO : Could not assign 1 map
files to tablet o;checksum|****;checksum|**** because :
org.apache.thrift.TApplicationException: Internal error processing
bulkImport .  Will retry ...

Mime
View raw message