Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@phoenix.apache.org
Date: Thu, 14 Dec 2017 18:25:00 +0000 (UTC)
From: "Lars Hofhansl (JIRA)" <jira@apache.org>
To: dev@phoenix.apache.org
Message-ID: <JIRA.13125011.1513273313000.461605.1513275900217@Atlassian.JIRA>
In-Reply-To: <JIRA.13125011.1513273313000@Atlassian.JIRA>
References: <JIRA.13125011.1513273313000@Atlassian.JIRA> <JIRA.13125011.1513273313772@jira-lw-us.apache.org>
Subject: [jira] [Commented] (PHOENIX-4460) High GC / RS shutdown when we use
 select query with "IN" clause using 4.10 phoenix client on 4.13 phoenix
 server
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 14 Dec 2017 18:25:06 -0000


    [ https://issues.apache.org/jira/browse/PHOENIX-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291300#comment-16291300 ] 

Lars Hofhansl commented on PHOENIX-4460:
----------------------------------------

That's it? Can you explain? :)

> High GC / RS shutdown when we use select query with "IN" clause using 4.10 phoenix client on 4.13 phoenix server
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-4460
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4460
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>            Assignee: James Taylor
>            Priority: Blocker
>             Fix For: 4.14.0, 4.13.2
>
>         Attachments: PHOENIX-4460.patch
>
>
> We were able to reproduce the High GC / RS shutdown / phoenix KeyRange query high object count issue on cluster today. 
> Main observation is that this is reproducible when firing lots of query    select from xyz where abc in (?, ?, ...)  of this type with 4.10 phoenix client hitting 4.13 phoenix on HBase server side
>  (4.10 client/4.10 server works fine, 4.13 client with 4.13 server works fine)
> We wrote a loader client (attached) with the below table/query , upserted ~100 million rows and fired the query in parallel using 4-5 loader clients with 16 threads each
> {code}
> TABLE:  = "CREATE TABLE " + TABLE_NAME_TEMPLATE 
>      + " (\n" + " TestKey varchar(255) PRIMARY KEY, TestVal1 varchar(200), TestVal2 varchar(200), "  + "TestValue varchar(10000))";
> QUERY: = "SELECT * FROM " +  TABLE_NAME_TEMPLATE + " WHERE TestKey IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
> {code}
> After running this client immediately within a min or two we see the phoenix.query.KeyRange object count immediately going up to several lakhs and keeps on increasing continuously. This count doesn't seem to come down even after shutting down the clients 
> {code}
> -bash-4.1$ ~/current/bigdata-util/tools/Linux/jdk/jdk1.8.0_102_x64/bin/jmap -histo:live 90725 | grep KeyRange
>   47:        274852        6596448  org.apache.phoenix.query.KeyRange
> 1851:             2             48  org.apache.phoenix.query.KeyRange$Bound
> 2434:             1             24  [Lorg.apache.phoenix.query.KeyRange$Bound;
> 3411:             1             16  org.apache.phoenix.query.KeyRange$1
> 3412:             1             16  org.apache.phoenix.query.KeyRange$2
> {code}
> After some time we also started seeing High GC issues and RegionServers crashing
> Experiment Summary:
> - 4.13 client/4.13 Server --- Issue not reproducible (we do see KeyRange count increasing upto few 100's)
> - 4.10 client/4.10 Server --- Issue not reproducible (we do see KeyRange count increasing upto few 100's)
> - 4.10 client/4.13 Server --- Issue reproducible as described above


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)