Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id ADC46200D3E for ; Thu, 16 Nov 2017 08:31:12 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id AC434160BE6; Thu, 16 Nov 2017 07:31:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 00086160BE5 for ; Thu, 16 Nov 2017 08:31:11 +0100 (CET) Received: (qmail 94046 invoked by uid 500); 16 Nov 2017 07:31:11 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 94037 invoked by uid 99); 16 Nov 2017 07:31:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Nov 2017 07:31:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 5A21CC30CA for ; Thu, 16 Nov 2017 07:31:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id DF3XcEaVXxkN for ; Thu, 16 Nov 2017 07:31:06 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 511B460F64 for ; Thu, 16 Nov 2017 07:31:05 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 387E4E2572 for ; Thu, 16 Nov 2017 07:31:03 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id AF91F240E6 for ; Thu, 16 Nov 2017 07:31:01 +0000 (UTC) Date: Thu, 16 Nov 2017 07:31:01 +0000 (UTC) From: "zhoukang (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (SPARK-22539) Add second order for rangepartitioner since partition number may be small if the specified key is skewed MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 16 Nov 2017 07:31:12 -0000 [ https://issues.apache.org/jira/browse/SPARK-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-22539: ----------------------------- Description: The rangepartitioner generated from shuffle exchange may cause partiton skew if sort key is skewed. We can add second order for rangepartitioner since partition number may be small if the specified key is skewed. This improvement comes from real case. was: The rangepartitioner generated from shuffle exchange may cause partiton skew if sort key is skewed. We can add second order for rangepartitioner since partition number may be small if the specified key is skewed. > Add second order for rangepartitioner since partition number may be small if the specified key is skewed > -------------------------------------------------------------------------------------------------------- > > Key: SPARK-22539 > URL: https://issues.apache.org/jira/browse/SPARK-22539 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.1.0 > Reporter: zhoukang > > The rangepartitioner generated from shuffle exchange may cause partiton skew if sort key is skewed. > We can add second order for rangepartitioner since partition number may be small if the specified key is skewed. > This improvement comes from real case. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org