beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofré ...@nanthrax.net>
Subject Re: Debugging pipelines
Date Sun, 20 Mar 2016 10:13:21 GMT
By the way, for the "Integration" DSL, in addition of explicit debug 
transform, it would make sense to have an implicit "Tracer". It's 
something that I planned: it would allow us to have sampling on 
PCollection if the pipeline tracer is enabled (like we do in a Camel 
route with the tracer).

Regards
JB

On 03/20/2016 10:14 AM, Ismaël Mejía wrote:
> ​Hello,
>
> I just started playing with Beam and I wanted to debug what happens
> between transforms in pipelines. I wrote a simple 'Debug' transform for
> this.
> The idea is to apply a function based on a predicate to any element in a
> collection without changing the collection, or in other words, a
> transform that
> does not transform but produces side effects.
>
> The idea is better illustrated with this simple example:
>
>      .apply(FlatMapElements.via((String text) ->
> Arrays.asList(text.split(" ")))
>        .withOutputType(new TypeDescriptor<String>() {
>       }))
>      .apply(Debug
>        .when((String s) -> s.startsWith("A"))
>        .with((String s) -> {
>          System.out.println(s);
>          return null;
>        }));
>      .apply(Filter.byPredicate((String text) -> text.length() > 5))
>      .apply(Debug.print());  // sugared method, same as above
>
> I think this can be useful (at least for debugging purposes), is there
> something
> like this already in the SDK ? If this is not the case, can you please
> give me some
> feedback/ideas to improve my transform.
>
> Thanks,
> -Ismael
>
> ps. You can find the code of the first version of the transform here:
> https://github.com/iemejia/beam-playground/blob/master/src/main/java/org/apache/beam/transforms/Debug.java
>
>

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Mime
View raw message