phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-3159) CachingHTableFactory may close HTable during eviction even if it is getting used for writing by another thread.
Date Mon, 03 Oct 2016 14:55:20 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542601#comment-15542601
] 

Hudson commented on PHOENIX-3159:
---------------------------------

SUCCESS: Integrated in Jenkins build Phoenix-master #1428 (See [https://builds.apache.org/job/Phoenix-master/1428/])
PHOENIX-3159 CachingHTableFactory may close HTable during eviction even (ankitsinghal59: rev
4c0aeb0d530852bc12e5fcd930e336fb19434397)
* (edit) phoenix-core/src/test/java/org/apache/phoenix/hbase/index/write/TestParalleIndexWriter.java
* (edit) phoenix-core/src/test/java/org/apache/phoenix/hbase/index/write/TestParalleWriterIndexCommitter.java
* (edit) phoenix-core/src/test/java/org/apache/phoenix/hbase/index/write/TestIndexWriter.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/hbase/index/write/recovery/TrackingParallelWriterIndexCommitter.java
* (edit) phoenix-core/src/test/java/org/apache/phoenix/hbase/index/write/TestCachingHTableFactory.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/hbase/index/table/CachingHTableFactory.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/hbase/index/table/HTableFactory.java
* (edit) phoenix-core/src/test/java/org/apache/phoenix/hbase/index/write/FakeTableFactory.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/hbase/index/table/CoprocessorHTableFactory.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/hbase/index/write/IndexWriterUtils.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/hbase/index/write/ParallelWriterIndexCommitter.java


> CachingHTableFactory may close HTable during eviction even if it is getting used for
writing by another thread.
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-3159
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3159
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Ankit Singhal
>            Assignee: Ankit Singhal
>             Fix For: 4.9.0, 4.8.2
>
>         Attachments: PHOENIX-3159.patch, PHOENIX-3159_v1.patch, PHOENIX-3159_v2.patch,
PHOENIX-3159_v3.patch, PHOENIX-3159_v4.patch, PHOENIX-3159_v5.patch
>
>
> CachingHTableFactory may close HTable during eviction even if it is getting used for
writing by another thread which results in writing thread to fail and index is disabled.
> LRU eviction closing HTable or underlying connection when cache is full and new HTable
is requested.
> {code}
> 2016-08-04 13:45:21,109 DEBUG [nat-s11-4-ioss-phoenix-1-5.openstacklocal,16020,1470297472814-index-writer--pool11-t35]
client.ConnectionManager$HConnectionImplementation: Closing HConnection (debugging purposes
only)
> java.lang.Exception
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.internalClose(ConnectionManager.java:2423)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.close(ConnectionManager.java:2447)
>         at org.apache.hadoop.hbase.client.CoprocessorHConnection.close(CoprocessorHConnection.java:41)
>         at org.apache.hadoop.hbase.client.HTableWrapper.internalClose(HTableWrapper.java:91)
>         at org.apache.hadoop.hbase.client.HTableWrapper.close(HTableWrapper.java:107)
>         at org.apache.phoenix.hbase.index.table.CachingHTableFactory$HTableInterfaceLRUMap.removeLRU(CachingHTableFactory.java:61)
>         at org.apache.commons.collections.map.LRUMap.addMapping(LRUMap.java:256)
>         at org.apache.commons.collections.map.AbstractHashedMap.put(AbstractHashedMap.java:284)
>         at org.apache.phoenix.hbase.index.table.CachingHTableFactory.getTable(CachingHTableFactory.java:100)
>         at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:160)
>         at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:136)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> {code}
> But the IndexWriter was using this old connection to write to the table which was closed
during LRU eviction
> {code}
> 016-08-04 13:44:59,553 ERROR [htable-pool659-t1] client.AsyncProcess: Cannot get replica
0 location for {"totalColumns":1,"row":"\\xC7\\x03\\x04\\x06X\\x1C)\\x00\\x80\\x07\\xB0X","families":{"0":[{"qualifier":"_0","vlen":2,"tag":[],"timestamp":1470318296425}]}}
> java.io.IOException: hconnection-0x21f468be closed
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1153)
>         at org.apache.hadoop.hbase.client.CoprocessorHConnection.locateRegion(CoprocessorHConnection.java:41)
>         at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncProcess.java:949)
>         at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:866)
>         at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1195)
>         at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveGlobalFailure(AsyncProcess.java:1162)
>         at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1100(AsyncProcess.java:584)
>         at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:727)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> {code}
> Although the workaround is to the cache size(index.tablefactory.cache.size). But still
we should handle the closing of working HTables to avoid index write failures (which in turn
disables index).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message