flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ant burton <apburto...@gmail.com>
Subject Re: Flink Elastic Sink AWS ES
Date Sun, 27 Aug 2017 22:12:09 GMT
Hi Arpit,

The response fromm _nodes doesn’t contain an ip address in my case. Is this something that
you experienced?

> curl -XGET 'http://<your aws es url>/_nodes'

Thanks,


> On 27 Aug 2017, at 14:32, ant burton <apburton84@gmail.com> wrote:
> 
> Thanks! I'll check later this evening.
> 
> On Sun, 27 Aug 2017 at 07:44, arpit srivastava <arpit8622@gmail.com <mailto:arpit8622@gmail.com>>
wrote:
> We also had same setup where ES cluster was behind a proxy server for which port 80 was
used which redirected it to ES cluster 9200 port.
> 
> For using Flink we got the actual ip address of the ES nodes and put that in ips below.
> 
> transportAddresses.add(new InetSocketAddress(InetAddress.getByName("127.0.0.1"), 9300))
> transportAddresses.add(new InetSocketAddress(InetAddress.getByName("10.2.3.1"), 9300))
> But this worked only because 9300 port was open on ES nodes in our setup and so accessible
from our Flink cluster.​
> 
> Get your node list on your ES Cluster using
> curl -XGET 'http://<your aws es url>/_nodes'
> 
> ​and then check whether you can telnet on that <es node ip> on port 9300 from
your flink cluster nodes
> 
> $ telnet <es node ip> 9300
> 
> If this works then you can use above solution.​
> 
> 
> On Sun, Aug 27, 2017 at 4:09 AM, ant burton <apburton84@gmail.com <mailto:apburton84@gmail.com>>
wrote:
> Hi Ted,
> 
> Changing the port from 9300 to 9200 in the example you provides causes the error in the
my original message
> 
> my apologies for not providing context in the form of code in my original message, to
confirm I am using the example you provided in my application and have it working using port
9300 in a docker environment locally. 
> 
> Thanks,
> 
>> On 26 Aug 2017, at 23:24, Ted Yu <yuzhihong@gmail.com <mailto:yuzhihong@gmail.com>>
wrote:
>> 
>> If port 9300 in the following example is replaced by 9200, would that work ?
>> 
>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html>
>> 
>> Please use Flink 1.3.1+
>> 
>> On Sat, Aug 26, 2017 at 3:00 PM, ant burton <apburton84@gmail.com <mailto:apburton84@gmail.com>>
wrote:
>> Hello,
>> 
>> Has anybody been able to use the Flink Elasticsearch connector to sink data to AWS
ES.
>> 
>> I don’t believe this is possible as AWS ES only allows access to port 9200 (via
port 80) on the master node of the ES cluster, and not port 9300 used by the the Flink Elasticsearch
connector.
>> 
>> The error message that occurs when attempting to connect to AWS ES via port 80 (9200)
with the Flink Elasticsearch connector is:
>> 
>>     Elasticsearch client is not connected to any Elasticsearch nodes!
>> 
>> Could anybody confirm the above? and if possible provide an alternative solution?
>> 
>> Thanks you,
>> 
> 
> 


Mime
View raw message