flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ant burton <apburto...@gmail.com>
Subject Re: Flink Elastic Sink AWS ES
Date Mon, 28 Aug 2017 22:26:00 GMT
Hey Arpit,

> _cat/nodes?v&h=ip,port


returns the following which I have not added the x’s they were returned on the response

ip            port
x.x.x.x 9300
Thanks your for you help

Anthony


> On 28 Aug 2017, at 10:34, arpit srivastava <arpit8622@gmail.com> wrote:
> 
> Hi Ant,
> 
> Can you try this.
> 
> curl -XGET 'http://<your aws es url>/_cat/nodes?v&h=ip,port'
> 
> This should give you ip and port
> 
> On Mon, Aug 28, 2017 at 3:42 AM, ant burton <apburton84@gmail.com <mailto:apburton84@gmail.com>>
wrote:
> Hi Arpit,
> 
> The response fromm _nodes doesn’t contain an ip address in my case. Is this something
that you experienced?
> 
>> curl -XGET 'http://<your aws es url>/_nodes'
> 
> Thanks,
> 
> 
>> On 27 Aug 2017, at 14:32, ant burton <apburton84@gmail.com <mailto:apburton84@gmail.com>>
wrote:
>> 
>> Thanks! I'll check later this evening.
>> 
>> On Sun, 27 Aug 2017 at 07:44, arpit srivastava <arpit8622@gmail.com <mailto:arpit8622@gmail.com>>
wrote:
>> We also had same setup where ES cluster was behind a proxy server for which port
80 was used which redirected it to ES cluster 9200 port.
>> 
>> For using Flink we got the actual ip address of the ES nodes and put that in ips
below.
>> 
>> transportAddresses.add(new InetSocketAddress(InetAddress.getByName("127.0.0.1"),
9300))
>> transportAddresses.add(new InetSocketAddress(InetAddress.getByName("10.2.3.1"), 9300))
>> But this worked only because 9300 port was open on ES nodes in our setup and so accessible
from our Flink cluster.​
>> 
>> Get your node list on your ES Cluster using
>> curl -XGET 'http://<your aws es url>/_nodes'
>> 
>> ​and then check whether you can telnet on that <es node ip> on port 9300
from your flink cluster nodes
>> 
>> $ telnet <es node ip> 9300
>> 
>> If this works then you can use above solution.​
>> 
>> 
>> On Sun, Aug 27, 2017 at 4:09 AM, ant burton <apburton84@gmail.com <mailto:apburton84@gmail.com>>
wrote:
>> Hi Ted,
>> 
>> Changing the port from 9300 to 9200 in the example you provides causes the error
in the my original message
>> 
>> my apologies for not providing context in the form of code in my original message,
to confirm I am using the example you provided in my application and have it working using
port 9300 in a docker environment locally. 
>> 
>> Thanks,
>> 
>>> On 26 Aug 2017, at 23:24, Ted Yu <yuzhihong@gmail.com <mailto:yuzhihong@gmail.com>>
wrote:
>>> 
>>> If port 9300 in the following example is replaced by 9200, would that work ?
>>> 
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html>
>>> 
>>> Please use Flink 1.3.1+
>>> 
>>> On Sat, Aug 26, 2017 at 3:00 PM, ant burton <apburton84@gmail.com <mailto:apburton84@gmail.com>>
wrote:
>>> Hello,
>>> 
>>> Has anybody been able to use the Flink Elasticsearch connector to sink data to
AWS ES.
>>> 
>>> I don’t believe this is possible as AWS ES only allows access to port 9200
(via port 80) on the master node of the ES cluster, and not port 9300 used by the the Flink
Elasticsearch connector.
>>> 
>>> The error message that occurs when attempting to connect to AWS ES via port 80
(9200) with the Flink Elasticsearch connector is:
>>> 
>>>     Elasticsearch client is not connected to any Elasticsearch nodes!
>>> 
>>> Could anybody confirm the above? and if possible provide an alternative solution?
>>> 
>>> Thanks you,
>>> 
>> 
>> 
> 
> 


Mime
View raw message