apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bhupesh Chawda <bhup...@datatorrent.com>
Subject File control tuples
Date Fri, 02 Jun 2017 19:06:05 GMT
​Hi,

​

Emitting ​file
​information
for a file based source like a file input operator
​in malhar ​
seems
​like
a
​good
feature to provide. It is useful information for any downstream operator to
​know
that a data tuple belongs to a certain file
​ for instance​
.


We propose to add capability in the abstract file input operator to emit
file control tuples. These control tuples can include filenames as well as
any metadata that the user wishes to include along with it.

​To link this meta data to each tuple, we can add another port to the input
operator which would carry the meta data along with the actual tuple. We
can try to reduce the amount of meta data that goes with each tuple by
having some sort of meta encoding in the control tuple.


~ Bhupesh​


_______________________________________________________

Bhupesh Chawda

E: bhupesh@datatorrent.com | Twitter: @bhupeshsc

www.datatorrent.com  |  apex.apache.org

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message