hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Enis Soztutar (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-449) Generalize the SequenceFileInputFilter to apply to any InputFormat
Date Tue, 25 Mar 2008 14:53:24 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581971#action_12581971

Enis Soztutar commented on HADOOP-449:

bq. Even if every job can use the filtering functionality, integrating it into Task/MapTask
limits where it may be applied. If, for example, one were reading from multiple sources, different
sets of filters could be applied to each source. Similarly, a map or a reduce task could use
a filtering record reader to read a subset of records indirectly. If it's limited to the interfaces
you provide to MapTask, then this code can't be reused elsewhere. Again, since weaving it
into core doesn't seem to give you extra functionality- it seems to make it less general-
and there's zero performance hit, making it a library looks laced with win.
please see below
bq. Not exactly. If I apply a RangeFilter to each of my record readers, the join considers
a smaller subset of the records read. Since it's generating the cross of all the matching
records (i.e. sets A, B, C sharing key k and containing values x1, x2 would emit [a1, b1,
c1], [a1, b1, c2], [a1, b2, c1], ... [a2, b2, c2]), my filter would have to reject the cross
of all those records, rather than each individually. Further, if I only want to filter the
records from B in the previous example, the filter in my map would need more state to ensure
I'm not emitting duplicate records to the map (or my map code would have to deal with that).
One can imagine other cases where, again, filtering shouldn't be limited to a single part
of the job, or cases where it might change the result if filters can only be applied at a
certain stage.

What we discuss, differs in just one part : 
As a library : 
  - FilterRecordReader same as before
  - classes under io.filter - same as before
  - FilterInputFormat as a thin wrapper 
  - Configure filters from FilterInputFormat (delegating to FilterEngine)
  - Users may freely extend / use FilterInputFormat, FilterRecordReader, Filter's. 
As integrated to the core 
  - FilterRecordReader same as before
  - classes under io.filter - same as before
  - instead of FilterInputFormat, use FilterRecordReader wrapping MapTask.TrackedRecordReader
if filtering is enabled. 
  - Configure filters from JobConf. (delegating to FilterEngine)
  - Users may freely extend / use FilterRecordReader, Filter's. 

The bulk of the implementation is in the FilterRecordReader, FilterEngine and Filter implementations,
thus this code(and filtering functionality) CAN be reused. Moreover FilterInputFormat from
the previous patch is only a wrapper. Assuming we move out FilterRecordReader to a separate
class, join framework can easily extend its grammar to accept an InputFormat which filters
records from its underlying inputformat (but this is a different issue ). In this sense, the
current patch does not "limit" applicability of filtering to other parts of the system, such
as using more than one inputFormats, filtering map output results etc, but it enables a filtering
framework and enables a frequent use case in which we filter input records to the job. The
other use cases can indeed be implemented on top of the patch (in the case of join, it can
bypass the core filter, and introduce filtering at another layer). 

bq. I disagree, and I cite your previous patch. Its interface was not only easier to understand
than the postfix additions, but specifying the baseInputFormat was very intuitive. For users
seeking to benefit from this, the difficulty delta between the library and Task implementations
is so slight that I doubt it'll actually prevent someone from taking advantage of it.
The postfix additions is irrelevant to whether filtering should be a library or not. The postfix
expressions are a way to specify the filtering expression to use, that part of the API will
not be changed if we had sticked with FilterInputFormat.  

bq. Though Filters will be useful, the semantics of a FunctionFilter aren't so mysterious
that people won't want to write those, too. Again, the purpose of both parameters is easily
explained, and people will decide whether they should employ them or not. It seems premature
to decide that there are only two types of filters, anyway. It sounds like we agree that it's
a cleaner interface with only one signature for the eval; I'm just not sure I see the extensibility
benefit as clearly.
I have separated Filter and FunctionFilter interfaces to enhance encapsulation, not extensibility.
Having one eval method is a cleaner interface to core developers who could understand how
the postfix expression is evaluated using Dijkstra's algorithm in the FilterEngine class,
but it is a cleaner interface to hide all this details from the user if she just  wants to
develop a Filter. 
Think about 3 + 4 or in postfix 3 4 +. The 3 and 4 are values, and + is a function. Clearly,
the Filters(which are values) and FunctionFilters(which are functions) are different in that
case.  However 3,4 and + are all represented as symbols so that they can be interpreted and
passed to a machine evaluating the expression. Similarly FunctionFilter implements Filter(although
it is not a filter) to be passed to the FilterEngine. As most of the users will only implements
Filters, and there is only one boolean binary function (which is XOR) which is not implemented,
I see no loss of generality in hiding the gory details of FunctionFilters. 
Maybe the best way was to construct the hierarchy :
interface ExpressionItem;
interface Filter extends ExpressionItem;
interface Function extends ExpressionItem;
FilterEngine.add(ExpressionItem item);
But had chosen not to do this, to simplify things. Anyway as I said before, now that passing
complete objects are imaginable, thanks to HADOOP-3048, I will change the postfix expressions
and develop a more intuitive way : 
Filter f1 = new RangeFilter(2, 5);
Filter f2 = new RangeFilter (10, 20);
Filter orFilter = new ORFilter(f1, f2);

Well, Chris I share your wisdom about the patch being a library, but I only insist because
I [still] think that this way might be better. Other than that, since the implicit votes are
2v1, I will probably implement the next version of the patch as a library including FilterInputFormat,
unless someone objects. 

> Generalize the SequenceFileInputFilter to apply to any InputFormat
> ------------------------------------------------------------------
>                 Key: HADOOP-449
>                 URL: https://issues.apache.org/jira/browse/HADOOP-449
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.17.0
>            Reporter: Owen O'Malley
>            Assignee: Enis Soztutar
>             Fix For: 0.17.0
>         Attachments: filtering_v2.patch, filtering_v3.patch, filterinputformat_v1.patch
> I'd like to generalize the SequenceFileInputFormat that was introduced in HADOOP-412
so that it can be applied to any InputFormat. To do this, I propose:
> interface WritableFilter {
>    boolean accept(Writable item);
> }
> class FilterInputFormat implements InputFormat {
>   ...
> }
> FilterInputFormat would look in the JobConf for:
>    mapred.input.filter.source = the underlying input format
>    mapred.input.filter.filters = a list of class names that implement WritableFilter
> The FilterInputFormat will work like the current SequenceFilter, but use an internal
RecordReader rather than the SequenceFile. This will require adding a next(key) and getCurrentValue(value)
to the RecordReader interface, but that will be addressed in a different issue.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message