beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofré ...@nanthrax.net>
Subject Re: 答复: Public in-progress I/O Transform list
Date Wed, 12 Apr 2017 06:22:16 GMT
Hi,

Yes, I saw that and already provided some info.

The RestIO I have works as follow:

- Producer Read (working)

pipeline.apply(RestIO.read().from("http://localhost:8080/rest"))

It provides a PCollection<String> where String is JSON or XML. It's also 
possible to provide a data binding with withDataBinding() method.

It's basically a DoFn as it's not possible to do any splitting.

- Producer Write (working)

pipeline.apply()....apply(RestIO.write().to("http://localhost:8080/rest"))

Here, it's basically the same DoFn as producer using input PCollection as REST 
request.

- Consumer Read (work in progress)

pipeline.apply(RestIO.read().bind("http://localhost:9090/foo"))

Here it's an unbounded source that bind a CXF connector (the initialSplit() 
method defines where the connector will be bind).

I'm experiment this as it's more  "ambitious" than the producer mode.

Regards
JB

On 04/12/2017 08:16 AM, Tang Jijun(上海_技术部_数据平台_唐觊隽) wrote:
> Hi Stephen,
> We need restIO module. I am looking forward to your pr about it. For restIO ,I create
a jira https://issues.apache.org/jira/browse/BEAM-1946.
>
> -----邮件原件-----
> 发件人: Jean-Baptiste Onofré [mailto:jb@nanthrax.net]
> 发送时间: 2017年4月12日 14:10
> 收件人: user@beam.apache.org
> 主题: Re: Public in-progress I/O Transform list
>
> Hi Stephen,
>
> Thanks !
>
> I would add Socket IO and REST IO that I  have on local branches (I'm waiting to close
some of my PRs before opening new ones). I also  have (not fully ready for
> PR) CamelIO, IronMQIO, and others (I will update the website  on  the fly ;)).
>
> MQTT IO is missing in the list. Gonna provide a website PR about that.
>
> I also created ExecIO while ago but we didn't get a consensus about what we should do
with it.
>
> Regards
> JB
>
> On 04/12/2017 02:01 AM, Stephen Sisk wrote:
>> Hi!
>>
>> We occasionally get questions about whether or not an I/O is planned
>> to be added to Beam. I've added a list of known in-progress I/O
>> Transforms to the "Built-in Transforms" page (where in-progress is
>> defined as "has a JIRA issue"). The Built-In Transforms page is the
>> publicly visible list of I/O transforms, so it's a natural place to let users know
what's also in-progress.
>>
>> You can find the current list here:
>> https://beam.apache.org/documentation/io/built-in/
>>
>> I may have missed one or two I/Os (there's no good way to find them in
>> JIRA as far as I can tell), so if you're working on an I/O Transform
>> that you intend to contribute to beam that's not listed, feel free to send a PR adding
it to the list.
>>
>> Thanks,
>> Stephen
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Mime
View raw message