pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pradeep Kamath" <prade...@yahoo-inc.com>
Subject Contract for streaming binaries
Date Fri, 29 Aug 2008 05:57:54 GMT
Hi,

  I am working on integrating the streaming feature (described at
http://wiki.apache.org/pig/PigStreamingFunctionalSpec) which is present
in the trunk into the "types" branch. As part of this I wanted to get
thoughts on the following design choice:

 Given the way the pipeline works in the types branch, we would need to
have a contract that the streaming binary would be invoked only in the
case where there is input to send to it. If either the original input is
entirely empty OR if the input passed on from the preceding operator in
the pipeline (possibly a filter) is entirely empty, then the streaming
binary will not be invoked and the "empty" state will be passed on to
the succeeding operator in the pipeline. By entirely empty I mean that
no tuple is available to pass on to the binary by the end of all map()
calls.

 

I wanted to work with this contract as a precondition for using
streaming and wanted to hear thoughts on this if anyone has an
objection.

 

Thanks,

Pradeep


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message