beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Romain Manni-Bucau <>
Subject Re: Beam Summit - IO brainstorming
Date Fri, 23 Mar 2018 07:21:33 GMT
+1000 for record metadata (camel headers)

For python it can be interesting to "just" generate python styled IO API
and use jython under the hood to let python users code as they know but
reuse all the beam ecosystem - including runners! The other way around
implies a lot of work for the community but also ops and makes users quite
unhappy to have a partial vision of beam IMHO. Java having the jsr223 we
can plug whatever we want behind including the portable api if you want
instead of putting it everywhere and pollute people not caring much.

Le 23 mars 2018 08:07, "Jean-Baptiste Onofré" <> a écrit :

> Hi all,
> Sorry for the delay, but I got issues with my e-mail provider (I was not
> able to
> send e-mails :( ).
> Last week during Beam Summit, I had the change to participate to the IO
> brainstorming session.
> Here's the minute notes:
> 1. IOs set
> We now have a decent number of IOs in Beam, and new are coming (ParquetIO,
> RabbitMQIO). Users mentioned a new file format you could support: HDF5. It
> would
> be an Python IO.
> I will create the Jira about HDF5.
> Other IOs will also be in preparation, coming along with SDF support.
> 2. IOs and SDKs
> This point was related to the portability layer: how can I use a Java IO in
> Python or the opposite ? Today, most of the IOs are related to Java SDK,
> and
> it's a bit frustrating for Python SDK users. Users are looking forward
> portability layer, however they also expressed some questions about Docker
> requirements. I think we should prepare a clean answer to this point.
> 3. PCollection Headers
> Users want more "dynamic" IOs, maybe that a IO behavior could change
> depending
> of the element they are considering in the PCollection. I introduced what
> we are
> using in Apache Camel: Message Headers. The Camel components endpoints
> (equivalent of Beam IOs) can use the headers: for instance the camel-http
> component can use a Camel.HTTP_URL header. We already discussed about
> PCollection headers/hints/annotation/metadata (whatever the name we give)
> and I
> still think it would be a great feature for both IOs and even the runners.
> I'm proposing to create a Jira about that, I will be more than happy to
> work on
> this one.
> 4. Schema
> As you might know, we are working on adding schema support in PCollection.
> This
> feature can be leveraged by IOs. Especially, I think it would reduce the
> "wrapping" made by IOs (like KafkaRecord, JmsRecord, ...) and easier data
> convert.
> 5. Error Handling
> Users would need a generic error handling in the IOs. Today the error
> handling
> is managed by each IOs. I introduced the error handler we are using in
> Apache
> Camel (sorry again ;)) and especially the default error handler features
> like:
> redelivery policy, recoverable/irrecoverable error handling, onWhen,
> onException, whileTrue, ...
> The error handler is not at component level but at routing engine level. We
> could imagine something similar at pipeline level.
> Thoughts ?
> I hope I didn't forget something ;)
> To summarize:
> - I will create new Jiras for HDF5 and other new IOs
> - We have to work on documentation/explanation about portability layer &
> IOs
> - I will start a separate thread for error handling discussion
> - Nothing to do about schema: it has already started.
> Regards
> JB
> --
> Jean-Baptiste Onofré
> Talend -

View raw message