phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-4460) High GC / RS shutdown when we use select query with "IN" clause using 4.10 phoenix client on 4.13 phoenix server
Date Thu, 14 Dec 2017 19:59:00 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291490#comment-16291490
] 

Hadoop QA commented on PHOENIX-4460:
------------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12902112/PHOENIX-4460.patch
  against master branch at commit 5cb02da74c15b0ae7c0fb4c880d60a2d1b6d18aa.
  ATTACHMENT ID: 12902112

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:red}-1 tests included{color}.  The patch doesn't appear to include any new or modified
tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:red}-1 release audit{color}.  The applied patch generated 1 release audit warnings
(more than the master's current 0 warnings).

    {color:green}+1 lineLengths{color}.  The patch does not introduce lines longer than 100

     {color:red}-1 core tests{color}.  The patch failed these unit tests:
     ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.AlterSessionIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.AggregateIT

Test results: https://builds.apache.org/job/PreCommit-PHOENIX-Build/1665//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-PHOENIX-Build/1665//artifact/patchprocess/patchReleaseAuditWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-PHOENIX-Build/1665//console

This message is automatically generated.

> High GC / RS shutdown when we use select query with "IN" clause using 4.10 phoenix client
on 4.13 phoenix server
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: PHOENIX-4460
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-4460
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: James Taylor
>            Assignee: James Taylor
>            Priority: Blocker
>             Fix For: 4.14.0, 4.13.2
>
>         Attachments: PHOENIX-4460.patch
>
>
> We were able to reproduce the High GC / RS shutdown / phoenix KeyRange query high object
count issue on cluster today. 
> Main observation is that this is reproducible when firing lots of query    select from
xyz where abc in (?, ?, ...)  of this type with 4.10 phoenix client hitting 4.13 phoenix on
HBase server side
>  (4.10 client/4.10 server works fine, 4.13 client with 4.13 server works fine)
> We wrote a loader client (attached) with the below table/query , upserted ~100 million
rows and fired the query in parallel using 4-5 loader clients with 16 threads each
> {code}
> TABLE:  = "CREATE TABLE " + TABLE_NAME_TEMPLATE 
>      + " (\n" + " TestKey varchar(255) PRIMARY KEY, TestVal1 varchar(200), TestVal2 varchar(200),
"  + "TestValue varchar(10000))";
> QUERY: = "SELECT * FROM " +  TABLE_NAME_TEMPLATE + " WHERE TestKey IN (?, ?, ?, ?, ?,
?, ?, ?, ?, ?)"
> {code}
> After running this client immediately within a min or two we see the phoenix.query.KeyRange
object count immediately going up to several lakhs and keeps on increasing continuously. This
count doesn't seem to come down even after shutting down the clients 
> {code}
> -bash-4.1$ ~/current/bigdata-util/tools/Linux/jdk/jdk1.8.0_102_x64/bin/jmap -histo:live
90725 | grep KeyRange
>   47:        274852        6596448  org.apache.phoenix.query.KeyRange
> 1851:             2             48  org.apache.phoenix.query.KeyRange$Bound
> 2434:             1             24  [Lorg.apache.phoenix.query.KeyRange$Bound;
> 3411:             1             16  org.apache.phoenix.query.KeyRange$1
> 3412:             1             16  org.apache.phoenix.query.KeyRange$2
> {code}
> After some time we also started seeing High GC issues and RegionServers crashing
> Experiment Summary:
> - 4.13 client/4.13 Server --- Issue not reproducible (we do see KeyRange count increasing
upto few 100's)
> - 4.10 client/4.10 Server --- Issue not reproducible (we do see KeyRange count increasing
upto few 100's)
> - 4.10 client/4.13 Server --- Issue reproducible as described above



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message