beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lukasz Cwik <>
Subject Re: Serving web pages with Beam pipelines?
Date Tue, 05 Apr 2016 15:30:55 GMT
Since Beam is all about being able to distribute work, you'll need an HTTP
request/response system where the response can be given from any machine.
This will most likely require some device/service which holds connections
open for you which you can connect back to serve the response. You would
need to have a custom source which "streams" requests from this device and
a custom sink that "streams" back responses. Beam currently is lacking
support for custom unbounded sinks but as always, anything can be done with
ParDo even if it shouldn't.

On Tue, Apr 5, 2016 at 8:28 AM, Jean-Baptiste Onofré <>

> Hi Bertrand,
> it sounds like Camel route, as you don't really manipulate the data, and
> you need service/reply.
> I'm not sure Beam will match
> Anyway, first, we would need an IO exposing a HTTP service, getting the
> PCollections (bounded), doing some Fn, the updated PCollections have to be
> sent back to the HTTP IO, which is not possible AFAIR.
> Let me know if I can help. IMHO, Camel is more appropriate.
> Regards
> JB
> On 04/05/2016 03:28 PM, Bertrand Delacretaz wrote:
>> Hi,
>> Besides being an incubation mentor for Beam (which is not relevant to
>> this message) I'm a long-time Apache Sling committer and PMC member -
>> Sling is a web applications framework [1] where an important task is
>> generating web pages dynamically, by assembling and transforming atoms
>> of content. Sling is very modular and "malleable", uses OSGi services
>> to implement this.
>> In this context, I'm interested in experimenting with Beam to
>> implement an HTTP request -> response pipeline (unless people think
>> this is crazy ;-)
>> I see how that would work for the HTTP key-value pairs: request
>> method, path, parameters, headers as input and HTTP status and headers
>> as output, shouldn't be a problem.
>> What I don't see is how to handle the binary request/response streams
>> - are there any Beam examples which would put me on the right track
>> for accepting a binary stream and returning another one to my HTTP
>> clients?
>> -Bertrand
>> [1]
> --
> Jean-Baptiste Onofré
> Talend -

View raw message