Return-Path: X-Original-To: apmail-datafu-dev-archive@minotaur.apache.org Delivered-To: apmail-datafu-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 61EC5113FE for ; Fri, 12 Sep 2014 00:20:28 +0000 (UTC) Received: (qmail 74281 invoked by uid 500); 12 Sep 2014 00:20:28 -0000 Delivered-To: apmail-datafu-dev-archive@datafu.apache.org Received: (qmail 74227 invoked by uid 500); 12 Sep 2014 00:20:28 -0000 Mailing-List: contact dev-help@datafu.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@datafu.incubator.apache.org Delivered-To: mailing list dev@datafu.incubator.apache.org Received: (qmail 74216 invoked by uid 99); 12 Sep 2014 00:20:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Sep 2014 00:20:27 +0000 X-ASF-Spam-Status: No, hits=-1999.5 required=5.0 tests=ALL_TRUSTED,HTML_MESSAGE,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 12 Sep 2014 00:20:26 +0000 Received: (qmail 74003 invoked by uid 99); 12 Sep 2014 00:20:06 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Sep 2014 00:20:06 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 9524B1DD6AC; Fri, 12 Sep 2014 00:20:04 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============2649671574339067866==" MIME-Version: 1.0 Subject: Review Request 25564: DATAFU-69: Create ChooseFieldByValue UDF - which, given a field who's value contains a field name, and *, returns the value of the field referenced by the field name From: "Russell Jurney" To: "Russell Jurney" , "DataFu" Date: Fri, 12 Sep 2014 00:20:04 -0000 Message-ID: <20140912002004.23923.67773@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Russell Jurney" X-ReviewGroup: DataFu X-ReviewRequest-URL: https://reviews.apache.org/r/25564/ X-Sender: "Russell Jurney" Reply-To: "Russell Jurney" X-ReviewRequest-Repository: datafu X-Virus-Checked: Checked by ClamAV on apache.org --===============2649671574339067866== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25564/ ----------------------------------------------------------- Review request for DataFu. Repository: datafu Description ------- Example use: group_fields = LOAD '/e8/smalldata/group_fields.txt' AS (groupField:chararray); with_group = CROSS group_fields, hour_rounded; with_group = FOREACH with_group GENERATE group_fields::groupField AS groupField, hour_rounded::sourceNameOrIp AS sourceNameOrIp, hour_rounded::destinationNameOrIp AS destinationNameOrIp, ...; with_value_substitution = FOREACH with_group GENERATE ChooseFieldByValue(groupField, *) AS groupValue:tuple(value:chararray), *; with_value_substitution = FOREACH with_value_substitution GENERATE FLATTEN(groupValue) AS groupValue:chararray, groupField, foo, bar, ...; all_success = FOREACH (GROUP with_value_substitution BY (groupField, groupValue, day)) GENERATE FLATTEN(group) AS (seriesType, groupValue, day), (int)COUNT_STAR(with_value_substitution) AS connections:int; Diffs ----- datafu-pig/src/main/java/datafu/pig/util/ChooseFieldByValue.java PRE-CREATION datafu-pig/src/test/java/datafu/test/pig/util/ChooseFieldByValueTest.java PRE-CREATION Diff: https://reviews.apache.org/r/25564/diff/ Testing ------- This UDF was used to replace a very inefficient pig script where macros that did many individual GROUP BY's took many minutes to plan. Testing: unit tests and used on real data on a cluster. Thanks, Russell Jurney --===============2649671574339067866==--