flink-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From u..@apache.org
Subject flink git commit: [docs] Add note about S3AFileSystem 'buffer.dir' property
Date Fri, 06 May 2016 12:46:43 GMT
Repository: flink
Updated Branches:
  refs/heads/release-1.0 4d3dcb148 -> 115d12aa1


[docs] Add note about S3AFileSystem 'buffer.dir' property


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/115d12aa
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/115d12aa
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/115d12aa

Branch: refs/heads/release-1.0
Commit: 115d12aa1157977958d08d46f723ee2ec2adadf9
Parents: 4d3dcb1
Author: Ufuk Celebi <uce@apache.org>
Authored: Fri May 6 14:46:02 2016 +0200
Committer: Ufuk Celebi <uce@apache.org>
Committed: Fri May 6 14:46:32 2016 +0200

----------------------------------------------------------------------
 docs/setup/aws.md | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/flink/blob/115d12aa/docs/setup/aws.md
----------------------------------------------------------------------
diff --git a/docs/setup/aws.md b/docs/setup/aws.md
index 156b4f0..ce1dcfa 100644
--- a/docs/setup/aws.md
+++ b/docs/setup/aws.md
@@ -95,13 +95,20 @@ There are two popular S3 file system implementations available:
 
 This is the recommended S3 FileSystem implementation to use. It uses Amazon's SDK internally
and works with IAM roles (see [Configure Access Credential](#configure-access-credentials)).
 
-You need to point Flink to a valid Hadoop configuration, which contains the following property
in `core-site.xml`:
+You need to point Flink to a valid Hadoop configuration, which contains the following properties
in `core-site.xml`:
 
 ```xml
 <property>
   <name>fs.s3.impl</name>
   <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
 </property>
+
+<!-- Comma separated list of local directories used to buffer
+     large results prior to transmitting them to S3. -->
+<property>
+  <name>fs.s3.buffer.dir</name>
+  <value>/tmp</value>
+</property>
 ```
 
 This registers `S3AFileSystem` as the default FileSystem for URIs with the `s3://` scheme.
@@ -311,3 +318,24 @@ Caused by: org.jets3t.service.impl.rest.HttpException [...]
 ```
 
 {% top %}
+
+### NullPointerException at org.apache.hadoop.fs.LocalDirAllocator
+
+This Exception is usually caused by skipping the local buffer directory configuration `fs.s3.buffer.dir`
for the `S3AFileSystem`. Please refer to the [S3AFileSystem configuration](#s3afilesystem-recommended)
section to see how to configure the `S3AFileSystem` properly.
+
+```
+[...]
+Caused by: java.lang.NullPointerException at
+o.a.h.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:268) at
+o.a.h.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
at
+o.a.h.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:416)
at
+o.a.h.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:198) at
+o.a.h.fs.s3a.S3AOutputStream.<init>(S3AOutputStream.java:87) at
+o.a.h.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:410) at
+o.a.h.fs.FileSystem.create(FileSystem.java:907) at
+o.a.h.fs.FileSystem.create(FileSystem.java:888) at
+o.a.h.fs.FileSystem.create(FileSystem.java:785) at
+o.a.f.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:404) at
+o.a.f.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:48) at
+... 25 more
+```


Mime
View raw message