pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheolsoo Park" <cheol...@cloudera.com>
Subject Review Request: PIG-2904 Scripting UDFs should allow DEFINE statements to pass parameters to the UDF's constructor
Date Tue, 25 Sep 2012 00:59:26 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7217/
-----------------------------------------------------------

Review request for pig and Julien Le Dem.


Description
-------

Currently, it is not possible to pass arguments to scripting udf's constructor via define
statements. However, this will be useful to support function closure in scripting udfs. Take
the following udf for example,

//-----

# udf that returns a closure
@outputSchema("log:double")
def logn(base):
    def log(x):
        return math.log(x, float(base))
    return log

//-----

With this patch, now we can do:

//-----

REGISTER 'scriptingudf.py' using jython as myfuncs;

DEFINE log2 myfuncs.logn('2');
DEFINE log10 myfuncs.logn('10');

out2 = FOREACH in GENERATE log2($0);
out10 = FOREACH in GENERATE log10($0);

//-----


Please note that constructor arguments specified in define statements are interpreted as closure
parameters, so it is not valid to pass constructor arguments to a udf that does not return
a closure. For example, the following script will throw an exception:

//-----

# udf that returns a closure
@outputSchema("log:double")
def log2(x):
    return math.log(x, 2)

//-----

DEFINE log2 myfuncs.log2('2');

//-----


The changes include:
- Update LogicalPlanGenerator grammar so that jython functions can be re-registered with constructor
parameters in define statement.
- Change JythonFunction's constructor so that it can take a list of string parameters.
- Change JythonFunction's exec method so that it can obtain a closure when constructor parameters
are present.
- Update Pig.java so that all this also works with embedded scripts.

Note that this patch only includes work for Jython. Work for other supported scripting languages
is yet to be added.


This addresses bug PIG-2904.
    https://issues.apache.org/jira/browse/PIG-2904


Diffs
-----

  src/org/apache/pig/parser/LogicalPlanGenerator.g 9b9c099 
  src/org/apache/pig/scripting/Pig.java 1e4065f 
  src/org/apache/pig/scripting/jython/JythonFunction.java 15f936d 
  test/e2e/pig/tests/nightly.conf ebcd66e 
  test/e2e/pig/tests/turing_jython.conf d800d78 
  test/e2e/pig/udfs/python/scriptingudf.py 54129b6 

Diff: https://reviews.apache.org/r/7217/diff/


Testing
-------

Added 3 test cases to e2e test suite:
1) Test for UDF that returns a closure with closure parameters in define statements. (positive
test)
2) Test for UDF that doesn't returns a closure with closure parameters in define statements.
(negative test)
3) Test for define statements in embedded scripts.

Ran ant test-commit.
Ran relevant e2e tests: Scripting_*, Jython_*.


Thanks,

Cheolsoo Park


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message