ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haithem Turki <turki.hait...@gmail.com>
Subject Writing data to S3A
Date Thu, 02 Jun 2016 01:13:56 GMT
Hello, I'm running into the following issue when trying to write to IGFS
backed by s3a storage as of 1.6.0:

hadoop fs -put file igfs://igfs@<node>/folder/

Throws:

2016-06-02 00:58:05,137 ERROR [igfs-igfs-ipc-#97%null%]
igfs.IgfsMetaManager (Log4JLogger.java:error(495)) - File create in DUAL
mode failed [path=/folder/file._COPYING_, simpleCreate=false,
props={permission=0644, locWrite=false}, overwrite=true, bufferSize=131072,
replication=3, blockSize=33554432]
class org.apache.ignite.IgniteCheckedException: Failed to open output
stream to the file created in the secondary file system because it no
longer exists: /folder/file._COPYING_
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager.fsException(IgfsMetaManager.java:2897)
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager.onSuccessCreate(IgfsMetaManager.java:1859)
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager$4.onSuccess(IgfsMetaManager.java:1949)
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager$4.onSuccess(IgfsMetaManager.java:1943)
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronizeAndExecute(IgfsMetaManager.java:2793)
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronizeAndExecute(IgfsMetaManager.java:2608)
at
org.apache.ignite.internal.processors.igfs.IgfsMetaManager.createDual(IgfsMetaManager.java:1967)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1007)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:992)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl.safeOp(IgfsImpl.java:1942)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl.create0(IgfsImpl.java:992)
at
org.apache.ignite.internal.processors.igfs.IgfsImpl.create(IgfsImpl.java:965)
at
org.apache.ignite.internal.processors.igfs.IgfsIpcHandler$2.apply(IgfsIpcHandler.java:399)
at
org.apache.ignite.internal.processors.igfs.IgfsIpcHandler$2.apply(IgfsIpcHandler.java:319)
at org.apache.ignite.igfs.IgfsUserContext.doAs(IgfsUserContext.java:54)
at
org.apache.ignite.internal.processors.igfs.IgfsIpcHandler.processPathControlRequest(IgfsIpcHandler.java:319)
at
org.apache.ignite.internal.processors.igfs.IgfsIpcHandler.execute(IgfsIpcHandler.java:240)
at
org.apache.ignite.internal.processors.igfs.IgfsIpcHandler.access$000(IgfsIpcHandler.java:56)
at
org.apache.ignite.internal.processors.igfs.IgfsIpcHandler$1.run(IgfsIpcHandler.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.igfs.IgfsException: Failed to open
output stream to the file created in the secondary file system because it
no longer exists: /folder/file._COPYING_
... 22 more

My default-config has the following block:

    <property name="fileSystemConfiguration">
            <list>
                <bean
class="org.apache.ignite.configuration.FileSystemConfiguration">
                    <!-- IGFS name you will use to access IGFS through
Hadoop API. -->
                    <property name="name" value="igfs"/>
                    <!-- Caches with these names must be configured. -->
                    <property name="metaCacheName" value="igfs-meta"/>
                    <property name="dataCacheName" value="igfs-data"/>

                    <!-- Configure TCP endpoint for communication with the
file system instance. -->
                    <property name="ipcEndpointConfiguration">
                        <bean
class="org.apache.ignite.igfs.IgfsIpcEndpointConfiguration">
                            <property name="type" value="TCP" />
                            <property name="host" value="0.0.0.0" />
                            <property name="port" value="10500" />
                        </bean>
                    </property>

                    <property name="secondaryFileSystem">
                        <bean
class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
                            <property name="fileSystemFactory">
                                <bean
class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory">
                                    <property name="uri"
value="s3a://my-bucket"/>
                                    <property name="configPaths">
                                        <list>

<value>/opt/ignite/core-site.xml</value>
                                        </list>
                                    </property>
                               </bean>
                            </property>
                        </bean>
                    </property>
                </bean>
            </list>
        </property>

And interestingly I can create folders which get reflected in the
underlying s3a bucket fine - it's just writing files that doesn't work

Any debugging help would be much appreciated - thanks!

Mime
View raw message