phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Taylor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-3649) After PHOENIX-3271 higher memory consumption on RS leading to OOM/abort on immutable index creation with multiple regions on single RS
Date Fri, 03 Mar 2017 08:26:45 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893908#comment-15893908
] 

James Taylor commented on PHOENIX-3649:
---------------------------------------

Thanks for the patch, [~ankit@apache.org]. So the upserts run on the server-side are using
the query compilation timestamp, right? There is a race condition in the original code, so
perhaps we're just losing the race now when the upsert select is run server-side.

I can't think of a better solution for non transactional tables than yours. The only problem
is that the index will be active, but it won't be consistent with the data table until the
buildIndex you added is complete. Perhaps we should mark the index as inactive while the buildIndex
is running?

> After PHOENIX-3271 higher memory consumption on RS leading to OOM/abort on immutable
index creation with multiple regions on single RS
> --------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-3649
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-3649
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.9.0
>            Reporter: Mujtaba Chohan
>            Assignee: Ankit Singhal
>            Priority: Blocker
>             Fix For: 4.9.1, 4.10.0
>
>         Attachments: PHOENIX-3649.patch, PHOENIX-3649_v1.patch, PHOENIX-3649_v2.patch
>
>
> *Configuration*
> hbase-0.98.23 standalone
> Heap 5GB
> *When*
> Verified that this happens after PHOENIX-3271 Distribute UPSERT SELECT across cluster.
https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commitdiff;h=accd4a276d1085e5d1069caf93798d8f301e4ed6
> To repro
> {noformat}
> CREATE TABLE INDEXED_TABLE (HOST CHAR(2) NOT NULL,DOMAIN VARCHAR NOT NULL, FEATURE VARCHAR
NOT NULL,DATE DATE NOT NULL,USAGE.CORE BIGINT,USAGE.DB BIGINT,STATS.ACTIVE_VISITOR INTEGER
CONSTRAINT PK PRIMARY KEY (HOST, DOMAIN, FEATURE, DATE)) IMMUTABLE_ROWS=true,MAX_FILESIZE=30485760
> {noformat}
> Upsert 2M rows (CSV is available at https://goo.gl/OsTSKB) that will create ~4 regions
on a single RS and then create index with data present
> {noformat}
> CREATE INDEX idx5 ON INDEXED_TABLE (CORE) INCLUDE (DB,ACTIVE_VISITOR)
> {noformat}
> From RS log
> {noformat}
> 2017-02-02 13:29:06,899 WARN  [rs,51371,1486070044538-HeapMemoryChore] regionserver.HeapMemoryManager:
heapOccupancyPercent 0.97875696 is above heap occupancy alarm watermark (0.95)
> 2017-02-02 13:29:18,198 INFO  [SessionTracker] server.ZooKeeperServer: Expiring session
0x15a00ad4f300001, timeout of 10000ms exceeded
> 2017-02-02 13:29:18,231 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause
in JVM or host machine (eg GC): pause of approximately 10581ms
> GC pool 'ParNew' had collection(s): count=4 time=139ms
> 2017-02-02 13:29:19,669 FATAL [RS:0;rs:51371-EventThread] regionserver.HRegionServer:
ABORTING region server rs,51371,1486070044538: regionserver:51371-0x15a00ad4f300001, quorum=localhost:2181,
baseZNode=/hbase regionserver:51371-0x15a00ad4f300001 received expired from ZooKeeper, aborting
> {noformat}
> Prior to the change index creation succeeds with as little as 2GB heap.
> [~ankit@apache.org]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message