beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen Sisk (JIRA)" <>
Subject [jira] [Created] (BEAM-1592) Unify HdfsIO and HadoopInputFormatIO
Date Thu, 02 Mar 2017 18:46:45 GMT
Stephen Sisk created BEAM-1592:

             Summary: Unify HdfsIO and HadoopInputFormatIO
                 Key: BEAM-1592
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-core
            Reporter: Stephen Sisk
            Assignee: Davor Bonaci

HIFIO is currently in PR (  and as per discussion
we'd like to check HIFIO in as-is, then unify the two since they share a lot of code. 

[] has mentioned: "the FileInputFormat reader gets to call some special
APIs that the
generic InputFormat reader cannot -- so they are not completely redundant. Specifically, FileInputFormat
reader can do size-based splitting." 

Dan recommended: "See if we can "inline" the FileInputFormat specific parts of HdfsIO inside
of HadoopInputFormatIO via reflection. If so, we can get the best of both worlds with shared

This seems reasonable to me. 

This message was sent by Atlassian JIRA

View raw message