From dev-return-49574-archive-asf-public=cust-asf.ponee.io@phoenix.apache.org Mon Feb 19 22:12:07 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id A1E2018067E for ; Mon, 19 Feb 2018 22:12:06 +0100 (CET) Received: (qmail 56342 invoked by uid 500); 19 Feb 2018 21:12:04 -0000 Mailing-List: contact dev-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@phoenix.apache.org Delivered-To: mailing list dev@phoenix.apache.org Received: (qmail 56200 invoked by uid 99); 19 Feb 2018 21:12:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Feb 2018 21:12:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 3FAB0C07C7 for ; Mon, 19 Feb 2018 21:12:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.311 X-Spam-Level: X-Spam-Status: No, score=-110.311 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id UNJAsi2inhJd for ; Mon, 19 Feb 2018 21:12:03 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id B8F325F5F7 for ; Mon, 19 Feb 2018 21:12:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id BF0DCE0230 for ; Mon, 19 Feb 2018 21:12:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 471482255B for ; Mon, 19 Feb 2018 21:12:00 +0000 (UTC) Date: Mon, 19 Feb 2018 21:12:00 +0000 (UTC) From: "James Taylor (JIRA)" To: dev@phoenix.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (PHOENIX-4619) Process transactional updates to local index on server-side MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PHOENIX-4619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated PHOENIX-4619: ---------------------------------- Description: For local indexes, we'll want to continue to process updates on the server-side. After PHOENIX-4278, updates even for local indexes are occurring on the client-side. The reason is that we know the updates to the index table will be a local write and we can generate the write on the server side. Having a separate RPC and sending the updates across the wire would be tremendously inefficient. On top of that, we need the region boundary information which we have already in the coprocessor, but would need to retrieve it on the client side (with a likely race condition too if a split occurs after we retrieve it). To fix this, we need to modify PhoenixTxnIndexMutationGenerator such that it can be use on the server-side as well. The main change will be to change this method signature to pass through an IndexMaintainer instead of a PTable (which isn't available on the server-side): {code} public List getIndexUpdates(final PTable table, PTable index, List dataMutations) throws IOException, SQLException { {code} I think this can be changed to the following instead and be used both client and server side: {code} public List getIndexUpdates(final IndexMaintainer maintainer, byte[] dataTableName, List dataMutations) throws IOException, SQLException { {code} We can tweak the code that makes PhoenixTransactionalIndexer a noop for clients >= 4.14 to have it execute if the index is a local index. The one downside is that if there's a mix of local and global indexes on the same table, the index update calculation will be done twice. I think having a mix of index types would be rare, though, and we should advise against it. There's also this code in UngroupedAggregateRegionObserver which needs to be updated to write shadow cells for Omid: {code} } else if (buildLocalIndex) { for (IndexMaintainer maintainer : indexMaintainers) { if (!results.isEmpty()) { result.getKey(ptr); ValueGetter valueGetter = maintainer.createGetterFromKeyValues( ImmutableBytesPtr.copyBytesIfNecessary(ptr), results); Put put = maintainer.buildUpdateMutation(kvBuilder, valueGetter, ptr, results.get(0).getTimestamp(), env.getRegion().getRegionInfo().getStartKey(), env.getRegion().getRegionInfo().getEndKey()); indexMutations.add(put); } } result.setKeyValues(results); {code} This is the code that builds a local index initially (unlike the global index code path which executes an UPSERT SELECT on the client side to do this initial population). was: For local indexes, we'll want to continue to process updates on the server-side. After PHOENIX-4278, updates even for local indexes are occurring on the client-side. The reason is that we know the updates to the index table will be a local write and we can generate the write on the server side. Having a separate RPC and sending the updates across the wire would be tremendously inefficient. On top of that, we need the region boundary information which we have already in the coprocessor, but would need to retrieve it on the client side (with a likely race condition too if a split occurs after we retrieve it). To fix this, we need to modify PhoenixTxnIndexMutationGenerator such that it can be use on the server-side as well. The main change will be to change this method signature to pass through an IndexMaintainer instead of a PTable (which isn't available on the server-side): {code} public List getIndexUpdates(final PTable table, PTable index, List dataMutations) throws IOException, SQLException { {code} I think this can be changed to the following instead and be used both client and server side: {code} public List getIndexUpdates(final IndexMaintainer maintainer, byte[] dataTableName, List dataMutations) throws IOException, SQLException { {code} We can tweak the code that makes PhoenixTransactionalIndexer a noop for clients >= 4.14 to have it execute if the index is a local index. The one downside is that if there's a mix of local and global indexes on the same table, the index update calculation will be done twice. I think having a mix of index types would be rare, though, and we should advise against it. > Process transactional updates to local index on server-side > ----------------------------------------------------------- > > Key: PHOENIX-4619 > URL: https://issues.apache.org/jira/browse/PHOENIX-4619 > Project: Phoenix > Issue Type: Bug > Reporter: James Taylor > Priority: Major > > For local indexes, we'll want to continue to process updates on the server-side. After PHOENIX-4278, updates even for local indexes are occurring on the client-side. The reason is that we know the updates to the index table will be a local write and we can generate the write on the server side. Having a separate RPC and sending the updates across the wire would be tremendously inefficient. On top of that, we need the region boundary information which we have already in the coprocessor, but would need to retrieve it on the client side (with a likely race condition too if a split occurs after we retrieve it). > To fix this, we need to modify PhoenixTxnIndexMutationGenerator such that it can be use on the server-side as well. The main change will be to change this method signature to pass through an IndexMaintainer instead of a PTable (which isn't available on the server-side): > {code} > public List getIndexUpdates(final PTable table, PTable index, List dataMutations) throws IOException, SQLException { > {code} > I think this can be changed to the following instead and be used both client and server side: > {code} > public List getIndexUpdates(final IndexMaintainer maintainer, byte[] dataTableName, List dataMutations) throws IOException, SQLException { > {code} > We can tweak the code that makes PhoenixTransactionalIndexer a noop for clients >= 4.14 to have it execute if the index is a local index. The one downside is that if there's a mix of local and global indexes on the same table, the index update calculation will be done twice. I think having a mix of index types would be rare, though, and we should advise against it. > There's also this code in UngroupedAggregateRegionObserver which needs to be updated to write shadow cells for Omid: > {code} > } else if (buildLocalIndex) { > for (IndexMaintainer maintainer : indexMaintainers) { > if (!results.isEmpty()) { > result.getKey(ptr); > ValueGetter valueGetter = > maintainer.createGetterFromKeyValues( > ImmutableBytesPtr.copyBytesIfNecessary(ptr), > results); > Put put = maintainer.buildUpdateMutation(kvBuilder, > valueGetter, ptr, results.get(0).getTimestamp(), > env.getRegion().getRegionInfo().getStartKey(), > env.getRegion().getRegionInfo().getEndKey()); > indexMutations.add(put); > } > } > result.setKeyValues(results); > {code} > This is the code that builds a local index initially (unlike the global index code path which executes an UPSERT SELECT on the client side to do this initial population). -- This message was sent by Atlassian JIRA (v7.6.3#76005)