beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varsha Thanooj (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-5059) Access Restful elasticsearch on aws from beam
Date Thu, 02 Aug 2018 09:36:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566541#comment-16566541
] 

Varsha Thanooj commented on BEAM-5059:
--------------------------------------

Yes, I am referring to fully hosted Amazon elasticsearch service.

As per the documentation ElasticsearchIO is not providing any way to input the aws credentials.

I tried
{code:java}
final String[] es_host = new String[] {"https://XXXX"}

data.apply(ElasticsearchIO.write()
.withConnectionConfiguration(
      ElasticsearchIO.ConnectionConfiguration.create(es_host, "XXX", "XXX")));

{code}
 

The console shows
{code:java}
Exception in thread "main" org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: Cannot get
Elasticsearch version

at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)

at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache.get(LocalCache.java:4053)

at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)

at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)

at org.apache.beam.runners.direct.DoFnLifecycleManager.get(DoFnLifecycleManager.java:61)

at org.apache.beam.runners.direct.ParDoEvaluatorFactory.createEvaluator(ParDoEvaluatorFactory.java:124)

at org.apache.beam.runners.direct.ParDoEvaluatorFactory.forApplication(ParDoEvaluatorFactory.java:78)

at org.apache.beam.runners.direct.TransformEvaluatorRegistry.forApplication(TransformEvaluatorRegistry.java:178)

at org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:118)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

at java.base/java.lang.Thread.run(Thread.java:844)

Caused by: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException:
Cannot get Elasticsearch version

at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)

at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn$DoFnInvoker.invokeSetup(Unknown
Source)

at org.apache.beam.runners.direct.DoFnLifecycleManager$DeserializingCacheLoader.load(DoFnLifecycleManager.java:102)

at org.apache.beam.runners.direct.DoFnLifecycleManager$DeserializingCacheLoader.load(DoFnLifecycleManager.java:91)

at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)

at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)

at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)

at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)

... 13 more

Caused by: java.lang.IllegalArgumentException: Cannot get Elasticsearch version

at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.getBackendVersion(ElasticsearchIO.java:1045)

at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.setup(ElasticsearchIO.java:925)


{code}

> Access Restful elasticsearch on aws from beam
> ---------------------------------------------
>
>                 Key: BEAM-5059
>                 URL: https://issues.apache.org/jira/browse/BEAM-5059
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-elasticsearch
>    Affects Versions: 2.5.0
>            Reporter: Varsha Thanooj
>            Assignee: Ismaël Mejía
>            Priority: Blocker
>              Labels: newbie
>
> Gone through the documentation of ElasticsearchIO. But I haven't found any way to write
records to aws elasticsearch.
> Is there a way to access Restful Elasticsearch hosted on aws from beam ?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message