flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Götzinger <m...@simplydevelop.de>
Subject Re: Using S3 as state backend
Date Tue, 15 Dec 2015 04:39:06 GMT
Hi Brian

Can you give me short summary how to achieve this.
Am 14.12.2015 23:20 schrieb "Brian Chhun" <brian.chhun@getbraintree.com>:

> For anyone else looking, I was able to use the s3a filesystem which can
> use IAM role based authentication as provided by the underlying AWS client
> library.
>
> Thanks,
> Brian
>
> On Thu, Dec 10, 2015 at 4:28 PM, Brian Chhun <brian.chhun@getbraintree.com
> > wrote:
>
>> Thanks Ufuk, this did the trick.
>>
>> Thanks,
>> Brian
>>
>> On Wed, Dec 9, 2015 at 4:37 PM, Ufuk Celebi <uce@apache.org> wrote:
>>
>>> Hey Brian,
>>>
>>> did you follow the S3 setup guide?
>>> https://ci.apache.org/projects/flink/flink-docs-master/apis/example_connectors.html
>>>
>>> You have to set the fs.hdfs.hadoopconf property and add
>>>
>>> <property>
>>> <name>fs.s3.impl</name>
>>> <value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
>>> </property>
>>>
>>> to core-site.xml
>>>
>>> – Ufuk
>>>
>>> > On 09 Dec 2015, at 20:50, Brian Chhun <brian.chhun@getbraintree.com>
>>> wrote:
>>> >
>>> > Hello,
>>> >
>>> > I'm trying to setup an HA cluster and I'm running into issues using S3
>>> as the state backend. This is raised during startup:
>>> >
>>> > 2015-12-09T19:23:36.430724+00:00 i-1ec317c4
>>> docker/jobmanager01-d3174d6[1207]: java.io.IOException: No file system
>>> found with scheme s3, referenced in file URI 's3:///flink/recovery/blob'.
>>> >
>>> > 2015-12-09T19:23:36.430858+00:00 i-1ec317c4
>>> docker/jobmanager01-d3174d6[1207]: #011at
>>> org.apache.flink.core.fs.FileSystem.get(FileSystem.java:242)
>>> >
>>> > 2015-12-09T19:23:36.430989+00:00 i-1ec317c4
>>> docker/jobmanager01-d3174d6[1207]: #011at
>>> org.apache.flink.runtime.blob.FileSystemBlobStore.<init>(FileSystemBlobStore.java:67)
>>> >
>>> > 2015-12-09T19:23:36.431297+00:00 i-1ec317c4
>>> docker/jobmanager01-d3174d6[1207]: #011at
>>> org.apache.flink.runtime.blob.BlobServer.<init>(BlobServer.java:105)
>>> >
>>> > 2015-12-09T19:23:36.431435+00:00 i-1ec317c4
>>> docker/jobmanager01-d3174d6[1207]: #011at
>>> org.apache.flink.runtime.jobmanager.JobManager$.createJobManagerComponents(JobManager.scala:1814)
>>> >
>>> > 2015-12-09T19:23:36.431569+00:00 i-1ec317c4
>>> docker/jobmanager01-d3174d6[1207]: #011at
>>> org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:1944)
>>> >
>>> > 2015-12-09T19:23:36.431690+00:00 i-1ec317c4
>>> docker/jobmanager01-d3174d6[1207]: #011at
>>> org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:1898)
>>> >
>>> > 2015-12-09T19:23:36.431810+00:00 i-1ec317c4
>>> docker/jobmanager01-d3174d6[1207]: #011at
>>> org.apache.flink.runtime.jobmanager.JobManager$.startActorSystemAndJobManagerActors(JobManager.scala:1584)
>>> >
>>> > 2015-12-09T19:23:36.431933+00:00 i-1ec317c4
>>> docker/jobmanager01-d3174d6[1207]: #011at
>>> org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:1486)
>>> >
>>> > 2015-12-09T19:23:36.432414+00:00 i-1ec317c4
>>> docker/jobmanager01-d3174d6[1207]: #011at
>>> org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1447)
>>> >
>>> > 2015-12-09T19:23:36.432649+00:00 i-1ec317c4
>>> docker/jobmanager01-d3174d6[1207]: #011at
>>> org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
>>> >
>>> > Is it possible to use S3 as the backend store or is only hdfs/mapfs
>>> supported?
>>> >
>>> >
>>> > Thanks,
>>> > Brian
>>>
>>>
>>
>

Mime
View raw message