flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joshua Griffith <JGriff...@CampusLabs.com>
Subject Using Azure Blob Storage with Flink
Date Tue, 29 Aug 2017 22:24:53 GMT
I’m attempting to write to Azure Blob Storage using Flink's FileOutputFormat. I’ve included
hadoop-azure<https://hadoop.apache.org/docs/current/hadoop-azure/index.html#Configuring_Credentials>
within the jar I submit to Flink and configured the paths to be prefixed with wasb://{CONTAINERNAME}@{ACCOUNTNAME}.blob.core.windows.net/<http://blob.core.windows.net/>.

When the file output format initializes, I get the following error: ERROR ROOT - Run 4bfb099a-8d07-11e7-8d3a-fb4d07562cc0
failed with error: 'org.apache.flink.client.program.ProgramInvocationException: The program
execution failed: Cannot initialize task 'DataSink (/out/data)': No file system found with
scheme wasb, referenced in file URI 'wasb://blob@{ACCOUNTNAME}.blob.core.windows.net/<http://blob.core.windows.net/>out/data’.

Can I register the format programmatically from within the job (without putting credentials
into a core-site.xml file on the task manager)? Can I still use Flink’s FileOutputFormat
or should I be using a Hadoop OutputFormat?

Thanks,

Joshua
Mime
View raw message