beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Kirpichov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2802) TextIO should allow specifying a custom delimiter
Date Fri, 25 Aug 2017 20:34:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142139#comment-16142139
] 

Eugene Kirpichov commented on BEAM-2802:
----------------------------------------

What does the custom delimiter look like in practice? E.g. if it's a particular line, then
one could imagine that this line could be delimited from others by \r or \n or \r\n - in that
case, it'd be better to first split into lines, and then group consecutive lines into a record
(as part of the source - not by a transform in the pipeline, because the output of TextIO.read
is unordered so you can't group consecutive lines anymore).

Also, does the user want the consecutive lines given as a List<String> or as a single
String that internally may contain \r, \n or \r\n's ?

> TextIO should allow specifying a custom delimiter
> -------------------------------------------------
>
>                 Key: BEAM-2802
>                 URL: https://issues.apache.org/jira/browse/BEAM-2802
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-extensions
>            Reporter: Etienne Chauchot
>            Assignee: Etienne Chauchot
>            Priority: Minor
>
> Currently TextIO use {{\r}} {{\n}} or {{\r\n}} or a mix of the two to split a text file
into PCollection elements. It might happen that a record is spread across more than one line.
In that case we should be able to specify a custom record delimiter to be used in place of
the default ones.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message