flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Using S3 as state backend
Date Wed, 09 Dec 2015 22:37:27 GMT
Hey Brian,

did you follow the S3 setup guide? https://ci.apache.org/projects/flink/flink-docs-master/apis/example_connectors.html

You have to set the fs.hdfs.hadoopconf property and add

<property>
<name>fs.s3.impl</name>
<value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
</property>

to core-site.xml

– Ufuk

> On 09 Dec 2015, at 20:50, Brian Chhun <brian.chhun@getbraintree.com> wrote:
> 
> Hello,
> 
> I'm trying to setup an HA cluster and I'm running into issues using S3 as the state backend.
This is raised during startup:
> 
> 2015-12-09T19:23:36.430724+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: java.io.IOException:
No file system found with scheme s3, referenced in file URI 's3:///flink/recovery/blob'.
> 
> 2015-12-09T19:23:36.430858+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at
org.apache.flink.core.fs.FileSystem.get(FileSystem.java:242)
> 
> 2015-12-09T19:23:36.430989+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at
org.apache.flink.runtime.blob.FileSystemBlobStore.<init>(FileSystemBlobStore.java:67)
> 
> 2015-12-09T19:23:36.431297+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at
org.apache.flink.runtime.blob.BlobServer.<init>(BlobServer.java:105)
> 
> 2015-12-09T19:23:36.431435+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at
org.apache.flink.runtime.jobmanager.JobManager$.createJobManagerComponents(JobManager.scala:1814)
> 
> 2015-12-09T19:23:36.431569+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at
org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:1944)
> 
> 2015-12-09T19:23:36.431690+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at
org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:1898)
> 
> 2015-12-09T19:23:36.431810+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at
org.apache.flink.runtime.jobmanager.JobManager$.startActorSystemAndJobManagerActors(JobManager.scala:1584)
> 
> 2015-12-09T19:23:36.431933+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at
org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:1486)
> 
> 2015-12-09T19:23:36.432414+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at
org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1447)
> 
> 2015-12-09T19:23:36.432649+00:00 i-1ec317c4 docker/jobmanager01-d3174d6[1207]: #011at
org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
> 
> Is it possible to use S3 as the backend store or is only hdfs/mapfs supported?
> 
> 
> Thanks,
> Brian


Mime
View raw message