Mailing-List: contact user-help@pig.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@pig.apache.org
Received-SPF: pass (athena.apache.org: domain of jcoveney@gmail.com designates
 209.85.214.49 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=vgNV30KwhrG3/0HFNbp9eLAQ53kht6rXBwK2SdCFPe0cpi9hzagvrBjn9E79sJuCke
         wm9jL6dtOfMlk49j0thH+0yG2bAtkXtGwO8qtpS/+hD3Y+2fElWgxYrl753p7F3HKzKy
         8kXysa3wuZE4EnU14qCrPC4dF1mvkYotYUT8k=
MIME-Version: 1.0
Date: Thu, 3 Feb 2011 15:30:02 -0500
Message-ID: <AANLkTinSmk0u2B5L3b9YSTCsL=U0Z6yiU4e3rh5qBABk@mail.gmail.com>
Subject: the nuts and bolts on how algebraic UDFs are invoked? // would this
 UDF be efficient?
From: Jonathan Coveney <jcoveney@gmail.com>
To: user@pig.apache.org
Content-Type: multipart/alternative; boundary=001636e1ecff53c300049b66a12a

--001636e1ecff53c300049b66a12a
Content-Type: text/plain; charset=ISO-8859-1

Howdy, I am curious about how algebraic UDFs are invoked. I know about how
to write them, but in the case where your initial step is to create a costly
datastructure, how can you ensure that this is created as few times as
possible?

What I have in mind is this: I have a huge set of data, and a big one. I
essentially want to join the two. I want to create a balanced binary search
tree of the big set of data, and then do the join against that... I imagine
that you could construct the search tree in every initial algebraic call,
and on the intermediate and final you simply will be passed the constructed
search tress. HOWEVER, if the initial call will be on EVERY row of data, it
might be faster to make it an accumulator? Is there is a difference in this
case?

Would this sort of strategyfor  creating a "join" via UDF be efficient? Is
there a more efficient way I want to avoid implementing something on the
lower pig level if possible (in large part because I have little experience
with parsers etc), but am not against that if I have to (in a way, this
could be a trial run for possible trying to implement some sort of band
join).

I'd appreciate your thoughts
Jon

--001636e1ecff53c300049b66a12a--