beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [beam-site] 01/02: Adds a dependencies guide to Beam Website
Date Fri, 22 Jun 2018 06:00:02 GMT
This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository

commit 37ed2b67537c04c39e01d144ee15206e5b2dd838
Author: <>
AuthorDate: Thu Jun 21 17:04:05 2018 -0700

    Adds a dependencies guide to Beam Website
 src/_includes/section-menu/contribute.html |  1 +
 src/contribute/             | 70 ++++++++++++++++++++++++++++++
 2 files changed, 71 insertions(+)

diff --git a/src/_includes/section-menu/contribute.html b/src/_includes/section-menu/contribute.html
index f956703..ca53156 100644
--- a/src/_includes/section-menu/contribute.html
+++ b/src/_includes/section-menu/contribute.html
@@ -18,6 +18,7 @@
     <li><a href="{{ site.baseurl }}/contribute/portability/">Portability Framework</a></li>
     <li><a href="{{ site.baseurl }}/contribute/docker-images/">Docker images</a></li>
     <li><a href="{{ site.baseurl }}/contribute/design-documents/">Design documents</a></li>
+    <li><a href="{{ site.baseurl }}/contribute/dependencies/">Dependencies guide</a></li>
diff --git a/src/contribute/ b/src/contribute/
new file mode 100644
index 0000000..e580dcf
--- /dev/null
+++ b/src/contribute/
@@ -0,0 +1,70 @@
+layout: section
+title: 'Dependencies Guide'
+section_menu: section-menu/contribute.html
+permalink: /contribute/dependencies/
+# Dependencies Guide
+This document describes policies for keeping Beam dependencies up to date.
+Old dependencies cause user pain and can result in a system being unusable for some users.
Many users do not use Beam in isolation and bundle other dependencies in the same deployment.
These additional dependencies might pull in incompatible dependencies to user’s environment
which can again result in broken Beam pipelines, sometimes with undefined behavior. To prevent
this, users will have to update their deployment environment or worse yet may end up not being
able to use Beam along wi [...]
+Beam Java SDK’s Gradle build defines a set of top level [dependencies](
and various components (runners, IO connectors, etc) can choose to include these dependencies.
Components usually use the versions defined at the top level but may choose to override these
+If a component _X_ chooses to override the version of a dependency _D_ from _a_ to _b_ and
another component _Y_ is incompatible with version _b_ of _D_, deployment of a user that uses
both components _X_ and _Y_ will end up in a broken state.
+A similar issue could arise if two dependencies of Beam depend on a common library but use
incompatible versions of that library.
+Also, users might not use Beam in isolation, a user that depends on Beam as well as other
libraries in the same environment might run into similar issues if Beam and the other library
share a dependency while using incompatible versions.
+Beam Python SDK handles dependencies slightly differently, all dependencies are defined in
a single []( file
and are grouped. One of the groups describes required dependencies while other groups are
for defining dependencies for various optional features. All Python modules have to use the
versions of dependencies defined in [](
file. Additionall [...]
+This picture can become even more complicated during runtime. Runner specific code might
be incompatible with dependencies included by certain modules and if these dependencies leak
into runtime, a pipeline might end up in a broken state.
+The overall issue is not common to Beam and well known in the industry as the Diamond [Dependency
problem \(or Dependency Hell\)](
+One common solution for the diamond dependency problem is [semantic versioning](
The basic idea is that dependencies will be versioned in the form _x.y.z_ where _x_ is the
_major version_, _y_ is the _minor version_, and _z_ is the _patch version_. A major version
change may be backwards incompatible and is expected to be rare. Minor and patch versions
may be released more regularly but are expected to be backwards compatible. But in practice,
important fixes (such a [...]
+## Identifying outdated dependencies
+A big part of keeping dependencies up to date involves identifying outdated dependencies
of Beam that the community should try to upgrade.
+Beam currently executes a weekly Jenkins job that tries to identify outdated dependencies
for various SDKs. This Jenkins job generates a weekly report that is shared in Beam dev list.
In the future we hope to automatically create JIRAs based on this report.
+In addition to this, Beam community members might identify other critical dependency updates
that have to be manually performed. For example,
+* A minor release of a dependency due to a critical security vulnerability. 
+* A dependency conflict that was was triggered by a minor version release of a Beam dependency
(this does not apply to Java SDK that depends on exact minor versions of dependencies).
+These kind of urgently required upgrades might not get automatically picked up by the Jenkins
job for few months. So Beam community has to act to identify such issues and perform upgrades
+## Upgrading identified outdated dependencies
+After outdated dependencies are identified, Beam community has to act to upgrade the dependencies
regularly. Beam community has agreed on following policies regarding upgrading dependencies.
+__Human readable reports on status of Beam dependencies are generated weekly by an automated
Jenkins job and shared with the Beam community through the dev list.__
+These reports should be concise and should highlight the cases where the community has to
act on.
+__Beam components should define dependencies and their versions at the top level. There can
be rare exceptions, but they should come with explanations.__ 
+Components include various Beam runners, IO connectors, etc. Component-level dependency version
declarations should only be performed in rare cases and should come with a comment explaining
the reasoning for overriding the dependency. For example, dependencies specific to a runner
that are unlikely to be utilized by other components might be defined at the runner.  
+__A significantly outdated dependency (identified manually or through the automated Jenkins
job) should result in a JIRA that is a blocker for the next release. Release manager may choose
to push the blocker to the subsequent release or downgrade from a blocker.__
+This will be a blocker for next major and minor version releases of Beam. JIRA may be created
automatically or manually.
+For manually identified critical dependency updates, Beam community members should create
blocking JIRAs for next release. In addition to this Beam community members may trigger patch
releases for any critical dependency fixes that should be made available to users urgently.
+__Dependency declarations may identify owners that are responsible for upgrading respective
+Owners can be mentioned in a comment. Blocking JIRAs will be initially assigned to these
owners (if available). Release manager may choose to re-assign these JIRAs. A dependency may
have more than one declared owner and in this case the JIRA will be assigned to one of the
owners mentioned.
+__Dependencies of Java SDK components that may cause issues to other components if leaked
should be vendored.__
+[Vendoring]( is
the process of creating copies of third party dependencies. Combined with repackaging, vendoring
will allow Beam components to depend on third party libraries without causing conflicts to
other components. Vendoring should be done in a case-by-case basis since this can increase
the total number of dependencies deployed in user's enviroment.
+## Dependency updates and backwards compatibility 
+Beam releases [adhere to]( semantic versioning.
Hence, community members should take care when updating dependencies. Minor version updates
to dependencies should be backwards compatible in most cases. Some updates to dependencies
though may result in backwards incompatible API or functionality changes to Beam. PR reviewers
and committers should take care to detect any dependency updates that could potentially introduce
backwards incompatibl [...]

View raw message