beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rik Nauta (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (BEAM-2541) ElasticsearchIO breaks for non locally accessible DB
Date Thu, 29 Jun 2017 15:36:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rik Nauta updated BEAM-2541:
----------------------------
    Description: 
We have a ES (2.4) database running in production that is not accessible from dev-machines
because of VPS firewall rules. However, we do create the Beam graph locally on our machines
and submit it to Google Dataflow for execution. However because of this check (line 213)

{{private static void checkVersion(ConnectionConfiguration connectionConfiguration)
        throws IOException {
      RestClient restClient = connectionConfiguration.createClient();
      Response response = restClient.performRequest("GET", "", new BasicHeader("", ""));
      JsonNode jsonNode = parseResponse(response);
      String version = jsonNode.path("version").path("number").asText();
      boolean version2x = version.startsWith("2.");
      restClient.close();
      checkArgument(
          version2x,
          "ConnectionConfiguration.create(addresses, index, type): "
              + "the Elasticsearch version to connect to is different of 2.x. "
              + "This version of the ElasticsearchIO is only compatible with Elasticsearch
v2.x");
}}}

the creation of the graph fails because it can't complete the restClient request from our
dev machines. This check should probably not happen this early since there is no guarantee
that the database is reachable when building the graph

  was:
We have a ES (2.4) database running in production that is not accessible from dev-machines
because of VPS firewall rules. However, we do create the Beam graph locally on our machines
and submit it to Google Dataflow for execution. However because of this check (line 213)

{{
private static void checkVersion(ConnectionConfiguration connectionConfiguration)
        throws IOException {
      RestClient restClient = connectionConfiguration.createClient();
      Response response = restClient.performRequest("GET", "", new BasicHeader("", ""));
      JsonNode jsonNode = parseResponse(response);
      String version = jsonNode.path("version").path("number").asText();
      boolean version2x = version.startsWith("2.");
      restClient.close();
      checkArgument(
          version2x,
          "ConnectionConfiguration.create(addresses, index, type): "
              + "the Elasticsearch version to connect to is different of 2.x. "
              + "This version of the ElasticsearchIO is only compatible with Elasticsearch
v2.x");
}
}}

the creation of the graph fails because it can't complete the restClient request from our
dev machines. This check should probably not happen this early since there is no guarantee
that the database is reachable when building the graph


> ElasticsearchIO breaks for non locally accessible DB
> ----------------------------------------------------
>
>                 Key: BEAM-2541
>                 URL: https://issues.apache.org/jira/browse/BEAM-2541
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-extensions
>            Reporter: Rik Nauta
>            Assignee: Davor Bonaci
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> We have a ES (2.4) database running in production that is not accessible from dev-machines
because of VPS firewall rules. However, we do create the Beam graph locally on our machines
and submit it to Google Dataflow for execution. However because of this check (line 213)
> {{private static void checkVersion(ConnectionConfiguration connectionConfiguration)
>         throws IOException {
>       RestClient restClient = connectionConfiguration.createClient();
>       Response response = restClient.performRequest("GET", "", new BasicHeader("", ""));
>       JsonNode jsonNode = parseResponse(response);
>       String version = jsonNode.path("version").path("number").asText();
>       boolean version2x = version.startsWith("2.");
>       restClient.close();
>       checkArgument(
>           version2x,
>           "ConnectionConfiguration.create(addresses, index, type): "
>               + "the Elasticsearch version to connect to is different of 2.x. "
>               + "This version of the ElasticsearchIO is only compatible with Elasticsearch
v2.x");
> }}}
> the creation of the graph fails because it can't complete the restClient request from
our dev machines. This check should probably not happen this early since there is no guarantee
that the database is reachable when building the graph



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message