flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Using Azure Blob Storage with Flink
Date Tue, 29 Aug 2017 22:32:31 GMT
Was hadoop-azure jar on the classpath ?

Please also see the following from
https://hadoop.apache.org/docs/current/hadoop-azure/index.html :

The built jar file, named hadoop-azure.jar, also declares transitive
dependencies on the additional artifacts it requires, notably the Azure
Storage SDK for Java.

On Tue, Aug 29, 2017 at 3:24 PM, Joshua Griffith <JGriffith@campuslabs.com>
wrote:

> I’m attempting to write to Azure Blob Storage using Flink's
> FileOutputFormat. I’ve included hadoop-azure
> <https://hadoop.apache.org/docs/current/hadoop-azure/index.html#Configuring_Credentials>
within
> the jar I submit to Flink and configured the paths to be prefixed with
> wasb://{CONTAINERNAME}@{ACCOUNTNAME}.blob.core.windows.net/.
>
> When the file output format initializes, I get the following error: ERROR
> ROOT - Run 4bfb099a-8d07-11e7-8d3a-fb4d07562cc0 failed with error:
> 'org.apache.flink.client.program.ProgramInvocationException: The program
> execution failed: Cannot initialize task 'DataSink (/out/data)': No file
> system found with scheme wasb, referenced in file URI 'wasb://blob@
> {ACCOUNTNAME}.blob.core.windows.net/out/data’.
>
> Can I register the format programmatically from within the job (without
> putting credentials into a core-site.xml file on the task manager)? Can I
> still use Flink’s FileOutputFormat or should I be using a Hadoop
> OutputFormat?
>
> Thanks,
>
> Joshua
>

Mime
View raw message