crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph Adler (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-279) Allow DoFn.process to throw exceptions
Date Thu, 10 Oct 2013 17:01:45 GMT


Joseph Adler commented on CRUNCH-279:

Well, exceptions are the standard Java mechanism for failure. Among other things, they give
you a meaningful stack trace.

I like the idea of an alternative failure mechanism (some absolute number of failed operations
or percentage of failed operations) for some use cases, but for other use cases an assertion
failure or bad record should cause the job to fail.

> Allow DoFn.process to throw exceptions
> --------------------------------------
>                 Key: CRUNCH-279
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.7.0
>            Reporter: Joseph Adler
>            Assignee: Josh Wills
> Currently, the DoFn process method doesn't throw any exceptions. This forces me to (a)
catch all exceptions internally (which is good), but then (b) do something unnatural (like
a deliberate null pointer exception) to stop processing and cause the job to fail.
> If only we could just throw an exception from DoFn (like, say, an IOException, which
both and Reducer.reduce can throw), we could have a better way to cause a task
to fail on error.
> If other people think this is useful, I'll submit a patch.

This message was sent by Atlassian JIRA

View raw message