beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From echauchot <>
Subject [GitHub] beam pull request #3950: [BEAM-2993] AvroIO.write without specifying a schem...
Date Thu, 05 Oct 2017 13:51:30 GMT
GitHub user echauchot opened a pull request:

     [BEAM-2993] AvroIO.write without specifying a schema

    Follow this checklist to help us incorporate your contribution quickly and easily:
     - [X] Make sure there is a [JIRA issue](
filed for the change (usually before you start working on it).  Trivial changes like typos
do not require a JIRA issue.  Your pull request should address just this issue, without pulling
in other changes.
     - [X] Each commit in the pull request should have a meaningful subject line and body.
     - [X] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`,
where you replace `BEAM-XXX` with the appropriate JIRA issue.
     - [X] Write a pull request description that is detailed enough to understand what the
pull request does, how, and why.
     - [X] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.
     - [X] If this contribution is large, please file an Apache [Individual Contributor License
    This PR adds the ability to use `AvroIO.write()` and related methods without specifying
a schema. 
    The schema is determined at the first call of `AvroSink.write()`: the `DataFileWriter`
is lazy initialized (at first write) once we have the value to get the schema from.  
    This PR also makes the schema optional in `ConstantAvroDestination` and depreciate write
methods that take schema as parameter. Tell me if I'm missing something that prevents deprecation
of these methods.
    To use `AvoIO.write()` with no schema, all the elements of the input PCollection must
have the same schema, but it is the same with current AvroIO.write(schema) implementation
because this schema instance is passed to the `TypedWrite` then to the `ConstantAvroDestination`
that is used in `AvroSink`. Please tell me if I'm missing something here.
    My only concern is with empty bundles, `AvroSink.write()` will not be called resulting
in the `DataFileWriter` not being initialized.  
    Please merge the PR bellow before this one because it is used as a base for the tests
    R: @jkff 
    R: @reuvenlax 
    CC: @lukecwik 

You can merge this pull request into a Git repository by running:

    $ git pull AvroIOWriteSchemaLess2

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3950
commit 43ef4d42d7d224b1997278832ec645bccb945792
Author: Etienne Chauchot <>
Date:   2017-10-05T09:45:12Z

    [BEAM-3019] Make AvroIOWriteTransformTest more generic
    make runTestWrite() more generic to be able to use GenericRecord[] as input for writeGenericRecords
test in place of AvroGeneratedUser
    make readAvroFile() generic to be able to read GenericRecords using GenericDatumReader
for writeGenericRecords test

commit 84074e36085d76f569c89d4a29a647fc40b22531
Author: Etienne Chauchot <>
Date:   2017-10-02T15:08:55Z

    [BEAM-2993] AvroIO.write without specifying a schema
    Lazy init (at first write) of the dataFileWriter once we have the value to get the schema
    Make schema optional in ConstantAvroDestination and depreciate write methods that take
schema as parameter

commit d19c2cb3538e5981e8138522d0c2138b455dec46
Author: Etienne Chauchot <>
Date:   2017-10-05T12:08:04Z

    Add tests of the schema less write methods

commit da95342353bd191c55d6d7768d4c052c531b8cf1
Author: Etienne Chauchot <>
Date:   2017-10-05T12:43:29Z




View raw message