phoenix-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-5131) Make spilling to disk for order/group by configurable
Date Thu, 07 Mar 2019 05:23:00 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-5131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786391#comment-16786391
] 

Hadoop QA commented on PHOENIX-5131:
------------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12961445/PHOENIX-5131-master.patch
  against master branch at commit afb191d16f6a155162c1a76d5ec20d5a48ba457e.
  ATTACHMENT ID: 12961445

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 6 new or modified
tests.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:red}-1 release audit{color}.  The applied patch generated 1 release audit warnings
(more than the master's current 0 warnings).

    {color:red}-1 lineLengths{color}.  The patch introduces the following lines longer than
100:
    +    public static void serializeIntoScan(Scan scan, int limit, List<OrderByExpression>
orderByExpressions, int estimatedRowSize) {
+            return new OrderedResultIterator(scanner, Collections.<OrderByExpression>singletonList(orderByExpression),
spoolingEnabled, threshold);
+                    iterator = new OrderedResultIterator(iterator, keyExpressionOrderBy,
spoolingEnabled, thresholdBytes, null, null, projector.getEstimatedRowByteSize());
+            resultScanner = new OrderedAggregatingResultIterator(aggResultIterator, orderBy.getOrderByExpressions(),
spoolingEnabled, thresholdBytes, limit, offset);
+            iterator = new OrderedResultIterator(iterator, orderBy.getOrderByExpressions(),
spoolingEnabled, thresholdBytes, limit,
+                e.setNextException(new SQLException("Also encountered exception while closing
queue", t));
+    final OrderedResultIterator iterator = deserializeFromScan(scan, innerScanner, spoolingEnabled,
thresholdBytes);
+  private static OrderedResultIterator deserializeFromScan(Scan scan, RegionScanner s, boolean
spoolingEnabled, long thresholdBytes) {
+      return new OrderedResultIterator(inner, orderByExpressions, spoolingEnabled, thresholdBytes,
limit >= 0 ? limit : null, null,
+            boolean spoolingEnabled, long thresholdBytes, Integer limit, Integer offset,int
estimatedRowSize) {

     {color:red}-1 core tests{color}.  The patch failed these unit tests:
     ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.TableDDLPermissionsIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.MutableIndexSplitReverseScanIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.LocalImmutableTxIndexIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ConcurrentMutationsIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.join.SortMergeJoinNoSpoolingIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ChangePermissionsIT
./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.MutableIndexSplitForwardScanIT

Test results: https://builds.apache.org/job/PreCommit-PHOENIX-Build/2395//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-PHOENIX-Build/2395//artifact/patchprocess/patchReleaseAuditWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-PHOENIX-Build/2395//console

This message is automatically generated.

> Make spilling to disk for order/group by configurable
> -----------------------------------------------------
>
>                 Key: PHOENIX-5131
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-5131
>             Project: Phoenix
>          Issue Type: Improvement
>            Reporter: Abhishek Singh Chouhan
>            Assignee: Abhishek Singh Chouhan
>            Priority: Major
>             Fix For: 4.15.0, 5.1.0
>
>         Attachments: PHOENIX-5131-master.patch
>
>
> We've observed that large queries, doing order/group by leading to issues on the regionserver
(crashes/long gc pauses/file handler exhaustion etc.). We should make spilling to disk configurable
and in case its disabled, fail the query once it hits the spilling limit on any of the region
servers. Also make spooling threshold server-side property only to prevent clients from controlling
memory allocation on the rs side.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message