beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugene Kirpichov <kirpic...@google.com>
Subject Re: [ANNOUNCE] Apache Beam 2.4.0 released
Date Sat, 24 Mar 2018 00:19:12 GMT
Woohoo, thanks Robert!

A few more notable changes that I'm aware of:

- BigQueryIO.write() now supports column-based partitioning, which makes it
dramatically cheaper and faster to load a bunch of historical data into a
time-partitioned table (1 load job total, instead of 1 load job per
partition).
- Introduces the Wait.on() transform, which allows general-purpose
sequencing of transforms, agnostic to batch vs. streaming. SpannerIO unit
tests are [the pioneering user](
https://github.com/apache/beam/pull/4264/files#diff-f74bb28a007fa6d75d065f14a7c76b38R150
).
- Increases scalability of watching for new files: successfully tested for
reading up to 1 million files, in 1 or 10,000 filepatterns.
- Introduces the TDigest sketch for estimating quantiles.

On Thu, Mar 22, 2018 at 5:23 AM Alexey Romanenko <aromanenko.dev@gmail.com>
wrote:

> Great news! Congrats!
>
> WBR,
> Alexey
>
>
> On 22 Mar 2018, at 10:10, Romain Manni-Bucau <rmannibucau@gmail.com>
> wrote:
>
> congrats guys
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com/> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
> 2018-03-22 9:50 GMT+01:00 Etienne Chauchot <echauchot@apache.org>:
>
>> Great !
>> Le jeudi 22 mars 2018 à 08:24 +0000, Robert Bradshaw a écrit :
>>
>> We are pleased to announce the release of Apache Beam 2.4.0. Thanks goes to
>> the many people who made this possible.
>>
>> Apache Beam is an open source unified programming model to define and
>> execute data processing pipelines, including ETL, batch and stream
>> (continuous) processing. See https://beam.apache.org
>>
>> You can download the release here:
>>
>>          https://beam.apache.org/get-started/downloads/
>>
>> As well as many bugfixes, some notable changes in this release are:
>> - A new Python Direct runner, up to 15x faster than the old one.
>> - Kinesis support for reading and writing in Java
>> - Several refactoring to enable portability (Go/Python <https://goto.google.com/Python>
on Flink/Spark)
>>
>> Full release notes can be found at
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682&projectId=12319527
>>
>> Enjoy!
>>
>>
>
>

Mime
View raw message