beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismaël Mejía (JIRA) <>
Subject [jira] [Assigned] (BEAM-1592) Unify HdfsIO and HadoopInputFormatIO
Date Mon, 26 Feb 2018 10:30:00 GMT


Ismaël Mejía reassigned BEAM-1592:

    Assignee:     (was: Jean-Baptiste Onofré)

> Unify HdfsIO and HadoopInputFormatIO
> ------------------------------------
>                 Key: BEAM-1592
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-hadoop
>            Reporter: Stephen Sisk
>            Priority: Major
>             Fix For: Not applicable
> HIFIO is currently in PR (  and as per discussion
we'd like to check HIFIO in as-is, then unify the two since they share a lot of code. 
> [] has mentioned: "the FileInputFormat reader gets to call some special
APIs that the
> generic InputFormat reader cannot -- so they are not completely redundant. Specifically,
FileInputFormat reader can do size-based splitting." 
> Dan recommended: "See if we can "inline" the FileInputFormat specific parts of HdfsIO
inside of HadoopInputFormatIO via reflection. If so, we can get the best of both worlds with
shared code." 
> This seems reasonable to me. 

This message was sent by Atlassian JIRA

View raw message