hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy V. Ryaboy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1427) Monitor and kill runaway UDFs
Date Thu, 27 May 2010 18:18:36 GMT

    [ https://issues.apache.org/jira/browse/PIG-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872314#action_12872314
] 

Dmitriy V. Ryaboy commented on PIG-1427:
----------------------------------------

bq. You didnt pay heed to my request for incrementing counter when udf times out or throws
an exception  I think that will be pretty useful for user to know how many faulty records
there are in the dataset which can't be processed by the UDF.

Right, I forgot to comment on that --  I think the counters no longer being available is a
separate issue; will file a ticket for that, and when the fix goes in, will use that here.


bq. In the getDefaultValue() there seems to be a inconsistency among different if statements.
I guess you need to make a distinction between Integer[] and Integer return type and then
return appropriate return value.

Whoops, good catch. Those are all supposed to be [0]. So, the trick there is that you can't
have null values as defaults for annotation properties, but you can have empty arrays; and
the first element of an array can be populated by simply setting nameOfArray = "foo" (the
first element becomes "foo").  So I am using that here. 

bq. Doing svn co; patch -p0 < monitoredUDF.patch; ant jar results in build failure. It
seems ivy is not pulling guava lib.

Yeah I spent 2 hours wrestling with maven last night until I convinced myself that the guava
maven deploy is broken. We need to drop the jar in lib/ for now. See here for reference: http://code.google.com/p/guava-libraries/issues/detail?id=354

bq. Since its user facing new interface, having stability/visibility tag would really be useful.

Will do.

bq. Since it spawns a new thread for every exec() call, I assume it will have some overhead.
If you have done some comparison or have numbers for that, it will be great if you can share
that. 

Simple enough. To measure the overhead, I created two equivalent UDFs that do nothing (they
fetch the current time, just to create an illusion of work and avoid potential java optimizations);
one is monitored, the other unmonitored. I then called them in loops of varying number of
repetition.  I first warmed up the JVM by running through the procedures without timing them,
and then repeated the same, this time grabbing the time at the beginning and end of the loop.
As you can see, there is almost no overhead -- it almost looks like we should default to monitoring,
it's so cheap.

{noformat}
Warming up.
Warmed up. Timing.
Reps: 1000 monitored: 0 unmonitored: 0
Reps: 10000 monitored: 1 unmonitored: 1
Reps: 100000 monitored: 12 unmonitored: 10
Reps: 1000000 monitored: 60 unmonitored: 62
Reps: 10000000 monitored: 587 unmonitored: 573
Reps: 1000 monitored: 0 unmonitored: 0
Reps: 10000 monitored: 0 unmonitored: 1
Reps: 100000 monitored: 6 unmonitored: 5
Reps: 1000000 monitored: 73 unmonitored: 57
Reps: 10000000 monitored: 574 unmonitored: 583
Reps: 1000 monitored: 0 unmonitored: 0
Reps: 10000 monitored: 1 unmonitored: 0
Reps: 100000 monitored: 6 unmonitored: 5
Reps: 1000000 monitored: 57 unmonitored: 58
Reps: 10000000 monitored: 594 unmonitored: 628
Reps: 1000 monitored: 0 unmonitored: 0
Reps: 10000 monitored: 1 unmonitored: 1
Reps: 100000 monitored: 6 unmonitored: 5
Reps: 1000000 monitored: 58 unmonitored: 62
Reps: 10000000 monitored: 594 unmonitored: 591
Reps: 1000 monitored: 0 unmonitored: 0
Reps: 10000 monitored: 0 unmonitored: 1
Reps: 100000 monitored: 6 unmonitored: 5
Reps: 1000000 monitored: 58 unmonitored: 69
Reps: 10000000 monitored: 587 unmonitored: 586
{noformat}



> Monitor and kill runaway UDFs
> -----------------------------
>
>                 Key: PIG-1427
>                 URL: https://issues.apache.org/jira/browse/PIG-1427
>             Project: Pig
>          Issue Type: New Feature
>    Affects Versions: 0.8.0
>            Reporter: Dmitriy V. Ryaboy
>            Assignee: Dmitriy V. Ryaboy
>         Attachments: monitoredUdf.patch, monitoredUdf.patch
>
>
> As a safety measure, it is sometimes useful to monitor UDFs as they execute. It is often
preferable to return null or some other default value instead of timing out a runaway evaluation
and killing a job. We have in the past seen complex regular expressions lead to job failures
due to just half a dozen (out of millions) particularly obnoxious strings.
> It would be great to give Pig users a lightweight way of enabling UDF monitoring.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message