beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (BEAM-2541) ElasticsearchIO breaks for non locally accessible DB
Date Wed, 05 Jul 2017 10:23:00 GMT


ASF GitHub Bot commented on BEAM-2541:

GitHub user echauchot opened a pull request:

    [BEAM-2541] Check Elasticsearch backend version when the pipeline is run not when it is

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
     - [X] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [X] Make sure tests pass via `mvn clean verify`.
     - [X] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [X] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](
    R: @jbonofre

You can merge this pull request into a Git repository by running:

    $ git pull BEAM-2541-move-ES-version-check-to-validate

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3493
commit f64156c4df7de97767adf70b2f02310727d9e902
Author: Etienne Chauchot <>
Date:   2017-07-05T09:35:59Z

    [BEAM-2541] Check Elasticsearch backend version when the pipeline is run not when it is


> ElasticsearchIO breaks for non locally accessible DB
> ----------------------------------------------------
>                 Key: BEAM-2541
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-extensions
>            Reporter: Rik Nauta
>            Assignee: Etienne Chauchot
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
> We have a ES (2.4) database running in production that is not accessible from dev-machines
because of VPS firewall rules. However, we do create the Beam graph locally on our machines
and submit it to Google Dataflow for execution. However because of this check (line 213)
> {code:java}
> private static void checkVersion(ConnectionConfiguration connectionConfiguration)
>         throws IOException {
>       RestClient restClient = connectionConfiguration.createClient();
>       Response response = restClient.performRequest("GET", "", new BasicHeader("", ""));
>       JsonNode jsonNode = parseResponse(response);
>       String version = jsonNode.path("version").path("number").asText();
>       boolean version2x = version.startsWith("2.");
>       restClient.close();
>       checkArgument(
>           version2x,
>           "ConnectionConfiguration.create(addresses, index, type): "
>               + "the Elasticsearch version to connect to is different of 2.x. "
>               + "This version of the ElasticsearchIO is only compatible with Elasticsearch
> }
> {code}
> the creation of the graph fails because it can't complete the restClient request from
our dev machines. This check should probably not happen this early since there is no guarantee
that the database is reachable when building the graph
> {code:java}
> exception in thread "main"
>         at org.apache.http.nio.pool.RouteSpecificPool.timeout(
>         at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(
>         at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(
>         at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(
>         at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(
>         at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(
>         at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(
>         at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(
>         at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$
>         at
> {code}

This message was sent by Atlassian JIRA

View raw message