Return-Path: X-Original-To: apmail-datafu-dev-archive@minotaur.apache.org Delivered-To: apmail-datafu-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4FB9711B1C for ; Fri, 21 Feb 2014 04:46:53 +0000 (UTC) Received: (qmail 82987 invoked by uid 500); 21 Feb 2014 04:46:52 -0000 Delivered-To: apmail-datafu-dev-archive@datafu.apache.org Received: (qmail 82958 invoked by uid 500); 21 Feb 2014 04:46:51 -0000 Mailing-List: contact dev-help@datafu.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@datafu.incubator.apache.org Delivered-To: mailing list dev@datafu.incubator.apache.org Received: (qmail 82950 invoked by uid 99); 21 Feb 2014 04:46:50 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Feb 2014 04:46:50 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of russell.jurney@gmail.com designates 209.85.215.182 as permitted sender) Received: from [209.85.215.182] (HELO mail-ea0-f182.google.com) (209.85.215.182) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Feb 2014 04:46:46 +0000 Received: by mail-ea0-f182.google.com with SMTP id r15so1306841ead.27 for ; Thu, 20 Feb 2014 20:46:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=N2hhmVaT5yzDWqP+0oB17gewCz4CUpcS+UXmxwPQ4hA=; b=tg04QGjzaHXiucIdJNa6JcIYif4ZeF84lmUk0+HBnTxCccTBaGEZ/RqPxEsuNmRYBU PS5Ndyc3M3m9rYGv/mLUCS2sF8FBAhkYPG8nhpIuR+Cr7KA+jG0xNz7oF2qB/VAtTAky tBZPlSsArOT57ewRAj2FtFMdNPLNW4QnxOc37xEHyDz+kXvXarKnNvP8sI0G5/5mEnrf 64GR01SREeLEhtfwlPpoLdW0/isE89mm84lr/VbqF/OdCcE05HF/z54J7sYeG1DOMKC/ ia9vXOXQStstueapxJknp3DsdAuH1+YQeXuPmYogBv6yhHJiTp/gOi95jagzZHtfy5p6 DwyA== MIME-Version: 1.0 X-Received: by 10.15.61.134 with SMTP id i6mr5841822eex.106.1392957985359; Thu, 20 Feb 2014 20:46:25 -0800 (PST) Received: by 10.14.98.129 with HTTP; Thu, 20 Feb 2014 20:46:25 -0800 (PST) In-Reply-To: References: Date: Thu, 20 Feb 2014 20:46:25 -0800 Message-ID: Subject: Re: Macros in DataFu From: Russell Jurney To: "dev@datafu.incubator.apache.org" , Jacob Perkins Content-Type: multipart/alternative; boundary=089e01681676e78ed704f2e34d49 X-Virus-Checked: Checked by ClamAV on apache.org --089e01681676e78ed704f2e34d49 Content-Type: text/plain; charset=ISO-8859-1 Oh, one issue worth raising... if the macros are inside a jar, it is really cool that we can reference UDFs in that jar from the macro. No extra loading needed. Jacob: would you be interested in contributing the varaha TF-IDF UDF to DataFu? On Thu, Feb 20, 2014 at 8:37 PM, Russell Jurney wrote: > Actually, this one by Jacob Perkins is better than mine: > https://github.com/thedatachef/varaha/blob/master/macros/nlp/tfidf.pig > > I rely on default_parallel with macros. I don't see another way if they > are inside a jar. We could make sure the macro source itself has high > visibility for customization/pasting to tune PARALLEL. > > > On Thu, Feb 20, 2014 at 6:47 PM, Sam Shah wrote: > >> Can you paste your TFIDF macro? How do you handle parallel statements? >> >> >> On Thu, Feb 20, 2014 at 6:36 PM, Russell Jurney > >wrote: >> >> > I would like to add macros to DataFu. I have a TFIDF macro and a couple >> > others I'd like to contribute. >> > >> > What do people think? Any issues that need to be figured out? >> > >> > Russ >> > >> > >> > -- >> > Russell Jurney twitter.com/rjurney russell.jurney@gmail.com >> > datasyndrome.com >> > >> > > > > -- > Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome. > com > -- Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com --089e01681676e78ed704f2e34d49--