beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <>
Subject [jira] [Commented] (BEAM-2802) TextIO should allow specifying a custom delimiter
Date Fri, 25 Aug 2017 20:34:00 GMT


Eugene Kirpichov commented on BEAM-2802:

What does the custom delimiter look like in practice? E.g. if it's a particular line, then
one could imagine that this line could be delimited from others by \r or \n or \r\n - in that
case, it'd be better to first split into lines, and then group consecutive lines into a record
(as part of the source - not by a transform in the pipeline, because the output of
is unordered so you can't group consecutive lines anymore).

Also, does the user want the consecutive lines given as a List<String> or as a single
String that internally may contain \r, \n or \r\n's ?

> TextIO should allow specifying a custom delimiter
> -------------------------------------------------
>                 Key: BEAM-2802
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-extensions
>            Reporter: Etienne Chauchot
>            Assignee: Etienne Chauchot
>            Priority: Minor
> Currently TextIO use {{\r}} {{\n}} or {{\r\n}} or a mix of the two to split a text file
into PCollection elements. It might happen that a record is spread across more than one line.
In that case we should be able to specify a custom record delimiter to be used in place of
the default ones.

This message was sent by Atlassian JIRA

View raw message