phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian Thomas (JIRA)" <j...@apache.org>
Subject [jira] [Created] (PHOENIX-4075) Executor already shutdown building secondary index
Date Tue, 08 Aug 2017 13:02:01 GMT
Brian Thomas created PHOENIX-4075:
-------------------------------------

             Summary: Executor already shutdown building secondary index
                 Key: PHOENIX-4075
                 URL: https://issues.apache.org/jira/browse/PHOENIX-4075
             Project: Phoenix
          Issue Type: Bug
         Environment: 3 Node HBase cluster running version 0.98
Phoenix version 4.7.0 using thin client + query server
            Reporter: Brian Thomas


I started to experience an issue trying to insert some data into our HBase cluster using phoenix.
This seems to have start occurring after I added a secondary index on one of my columns. 

The issue seems to resolve temporarily with an HBase restart. However, it eventually falls
back into this state and once it gets into this state, it's not possible to insert data anymore.
When my application is running, it tries to insert 80k rows and then issues a single commit.
However, the batch size doesn't seem to affect it as I cannot even insert a single row when
it gets into this state. The full stack trace I see is

Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 131563
actions: org.apache.phoenix.hbase.index.builder.IndexBuildingFailureException: Failed to build
index for unexp
ected reason!
        at org.apache.phoenix.hbase.index.util.IndexManagementUtil.rethrowIndexingException(IndexManagementUtil.java:180)
        at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:205)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$35.call(RegionCoprocessorHost.java:1007)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preBatchMutate(RegionCoprocessorHost.java:1003)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3088)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2875)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2817)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:751)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:713)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2142)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.RejectedExecutionException: Executor already shutdown
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.startTask(MoreExecutors.java:327)
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:251)
        at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:56)
        at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:58)
        at org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
        at org.apache.phoenix.hbase.index.builder.IndexBuildManager.getIndexUpdate(IndexBuildManager.java:143)
        at org.apache.phoenix.hbase.index.Indexer.preBatchMutateWithExceptions(Indexer.java:273)
        at org.apache.phoenix.hbase.index.Indexer.preBatchMutate(Indexer.java:202)
        ... 17 more
: 131563 times, 
        at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:247)
        at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:227)
        at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1663)
        at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:982)
        at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:996)
        at org.apache.phoenix.execute.MutationState.send(MutationState.java:951)
        ... 22 more


        at org.apache.calcite.avatica.remote.Service$ErrorResponse.toException(Service.java:2315)
        at org.apache.calcite.avatica.remote.RemoteProtobufService._apply(RemoteProtobufService.java:46)
        at org.apache.calcite.avatica.remote.ProtobufService.apply(ProtobufService.java:107)
        at org.apache.calcite.avatica.remote.RemoteMeta$17.call(RemoteMeta.java:377)
        at org.apache.calcite.avatica.remote.RemoteMeta$17.call(RemoteMeta.java:375)
        at org.apache.calcite.avatica.AvaticaConnection.invokeWithRetries(AvaticaConnection.java:666)
        at org.apache.calcite.avatica.remote.RemoteMeta.commit(RemoteMeta.java:375)
        at org.apache.calcite.avatica.AvaticaConnection.commit(AvaticaConnection.java:175)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message