Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8CC4C200CD2 for ; Thu, 13 Jul 2017 02:59:47 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 8B15F16A543; Thu, 13 Jul 2017 00:59:47 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id AB6C316A541 for ; Thu, 13 Jul 2017 02:59:46 +0200 (CEST) Received: (qmail 63375 invoked by uid 500); 13 Jul 2017 00:59:45 -0000 Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list dev@phoenix.apache.org Received: (qmail 63340 invoked by uid 99); 13 Jul 2017 00:59:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jul 2017 00:59:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 71B99196253 for ; Thu, 13 Jul 2017 00:59:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id dnBYnDUX_wTy for ; Thu, 13 Jul 2017 00:59:31 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 529CA618AC for ; Thu, 13 Jul 2017 00:59:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 41D9FE0059 for ; Thu, 13 Jul 2017 00:59:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 55C0C24741 for ; Thu, 13 Jul 2017 00:59:00 +0000 (UTC) Date: Thu, 13 Jul 2017 00:59:00 +0000 (UTC) From: "Hadoop QA (JIRA)" To: dev@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (PHOENIX-3997) UngroupedAggregateRegionObserver.commitBatchWithHTable() should not check the memstore size and wait for flush. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 13 Jul 2017 00:59:47 -0000 [ https://issues.apache.org/jira/browse/PHOENIX-3997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085002#comment-16085002 ] Hadoop QA commented on PHOENIX-3997: ------------------------------------ {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12876962/PHOENIX-3997-v2.patch against master branch at commit 06c10dc391220da8e54f0191bef1c9b8b702f651. ATTACHMENT ID: 12876962 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 50 warning messages. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: + private void commitBatch(Region region, List mutations, long blockingMemstoreSize) throws IOException { + private void setIndexAndTransactionProperties(List mutations, byte[] indexUUID, byte[] indexMaintainersPtr, byte[] txState, boolean useIndexProto) { + m.setAttribute(useIndexProto ? PhoenixIndexCodec.INDEX_PROTO_MD : PhoenixIndexCodec.INDEX_MD, indexMaintainersPtr); + isPKChanging = ExpressionUtil.isPkPositionChanging(new TableRef(projectedTable), selectExpressions); + commit(region, mutations, indexUUID, blockingMemStoreSize, indexMaintainersPtr, + setIndexAndTransactionProperties(indexMutations, indexUUID, indexMaintainersPtr, txState, useIndexProto); + private void commit(Region region, List mutations, byte[] indexUUID, long blockingMemStoreSize, + setIndexAndTransactionProperties(mutations, indexUUID, indexMaintainersPtr, txState, useIndexProto); + separateLocalAndRemoteMutations(targetHTable, region, mutations, localRegionMutations, remoteRegionMutations, + private void separateLocalAndRemoteMutations(HTable targetHTable, Region region, List mutations, {color:red}-1 core tests{color}. The patch failed these unit tests: ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.rpc.PhoenixServerRpcIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.RowValueConstructorIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ArrayIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.DerivedTableIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.DistinctCountIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.txn.MutableRollbackIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ArithmeticQueryIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.UseSchemaIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.AggregateIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.SequenceIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.TopNIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.SpillableGroupByIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ProductMetricsIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.VariableLengthPKIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.DynamicColumnIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.TenantSpecificViewIndexIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.TenantSpecificTablesDMLIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.txn.TxWriteFailureIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.DropColumnIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.QueryDatabaseMetaDataIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.CsvBulkLoadToolIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.CursorWithRowValueConstructorIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.AlterTableWithViewsIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.SequenceBulkAllocationIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.BinaryRowKeyIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.KeyOnlyIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ExtendedQueryExecIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ViewIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.NativeHBaseTypesIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.LocalIndexIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ReadIsolationLevelIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.UpsertValuesIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ToNumberFunctionIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.IndexToolForPartialBuildIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.ImmutableIndexIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.DeleteIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.FunkyNamesIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.TruncateFunctionIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.TableSnapshotReadsMapReduceIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.monitoring.PhoenixMetricsIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.NamespaceSchemaMappingIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.MappingTableDataTypeIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.IndexToolForPartialBuildWithNamespaceEnabledIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.rpc.UpdateCacheWithScnIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.salted.SaltedTableIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.RegexBulkLoadToolIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.AlterTableIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.IndexExpressionIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.AutoPartitionViewsIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.CustomEntityDataIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.ColumnProjectionOptimizationIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.IndexIT ./phoenix-core/target/failsafe-reports/TEST-org.apache.phoenix.end2end.index.ViewIndexIT Test results: https://builds.apache.org/job/PreCommit-PHOENIX-Build/1204//testReport/ Javadoc warnings: https://builds.apache.org/job/PreCommit-PHOENIX-Build/1204//artifact/patchprocess/patchJavadocWarnings.txt Console output: https://builds.apache.org/job/PreCommit-PHOENIX-Build/1204//console This message is automatically generated. > UngroupedAggregateRegionObserver.commitBatchWithHTable() should not check the memstore size and wait for flush. > ---------------------------------------------------------------------------------------------------------------- > > Key: PHOENIX-3997 > URL: https://issues.apache.org/jira/browse/PHOENIX-3997 > Project: Phoenix > Issue Type: Bug > Reporter: Thomas D'Silva > Assignee: Geoffrey Jacoby > Fix For: 4.12.0 > > Attachments: PHOENIX-3997.patch, PHOENIX-3997-v2.patch > > > [~ankit.singhal] > In UngroupedAggregateRegionObserver.commitBatchWithHTable() do we need to check the memstore size and wait for the flush. We are using a htable to write the mutations. > {code} > // When memstore size reaches blockingMemstoreSize we are waiting 3 seconds for the > // flush happen which decrease the memstore size and then writes allowed on the region. > for (int i = 0; region.getMemstoreSize().get() > blockingMemstoreSize && i < 30; i++) { > try { > checkForRegionClosing(); > Thread.sleep(100); > } catch (InterruptedException e) { > Thread.currentThread().interrupt(); > throw new IOException(e); > } > } > logger.debug("Committing batch of " + mutations.size() + " mutations for " + table); > try { > table.batch(mutations); > } catch (InterruptedException e) { > throw new RuntimeException(e); > } > {code} > FYI [~jamestaylor] [~apurtell] -- This message was sent by Atlassian JIRA (v6.4.14#64029)