Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BED49200D5A for ; Thu, 14 Dec 2017 14:16:04 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id BD6A4160C25; Thu, 14 Dec 2017 13:16:04 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0FAC5160C01 for ; Thu, 14 Dec 2017 14:16:03 +0100 (CET) Received: (qmail 40133 invoked by uid 500); 14 Dec 2017 13:16:03 -0000 Mailing-List: contact dev-help@datafu.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@datafu.incubator.apache.org Delivered-To: mailing list dev@datafu.incubator.apache.org Received: (qmail 40122 invoked by uid 99); 14 Dec 2017 13:16:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Dec 2017 13:16:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 837E6C185E for ; Thu, 14 Dec 2017 13:16:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.011 X-Spam-Level: X-Spam-Status: No, score=-100.011 tagged_above=-999 required=6.31 tests=[SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id liEI0WHfogcH for ; Thu, 14 Dec 2017 13:16:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 3F7095F3FE for ; Thu, 14 Dec 2017 13:16:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 76986E0EF1 for ; Thu, 14 Dec 2017 13:16:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 3522E212FD for ; Thu, 14 Dec 2017 13:16:00 +0000 (UTC) Date: Thu, 14 Dec 2017 13:16:00 +0000 (UTC) From: "Eyal Allweil (JIRA)" To: dev@datafu.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Closed] (DATAFU-116) Make SetIntersect and SetDifference implement Accumulator MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 14 Dec 2017 13:16:04 -0000 [ https://issues.apache.org/jira/browse/DATAFU-116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eyal Allweil closed DATAFU-116. ------------------------------- Resolution: Won't Fix Since it seems like Pig doesn't use the Accumulator interface when there are multiple bags in the input, this improvement isn't relevant for these UDF's. > Make SetIntersect and SetDifference implement Accumulator > --------------------------------------------------------- > > Key: DATAFU-116 > URL: https://issues.apache.org/jira/browse/DATAFU-116 > Project: DataFu > Issue Type: Improvement > Affects Versions: 1.3.0 > Reporter: Eyal Allweil > > SetIntersect and SetDifference accept only sorted bags, and the output is always smaller than the inputs. Therefore an accumulator implementation should be possible and it will improve memory usage (somewhat) and allow Pig to optimize loops with these operations better. -- This message was sent by Atlassian JIRA (v6.4.14#64029)