beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akanksha Sharma B <akanksha.b.sha...@ericsson.com>
Subject writing to maprfs?
Date Thu, 24 May 2018 07:24:25 GMT
Hi All,


I have a pipeline; it writes to hdfs, using "org.apache.beam.sdk.io.hdfs" package.

I was hoping that it will work with maprfs as well. However I have been debugging for some
days, with no success.


I do not provide hdfsConfiguration from command line, and instead use the configuration file
/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml (please find it attached with the mail)

However I get the stack trace ("No FileSystem for scheme: maprfs"). Please find the detailed
trace attached. I made sure that I have /opt/mapr/lib/maprfs-5.2.2-mapr.jar on classpath.
I use spark submit to run the pipeline.


If you have already tried writing to maprfs from beam, ccould you please help?


Regards,

Akanksha

Mime
View raw message