kylin-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From liy...@apache.org
Subject incubator-kylin git commit: doc, add new Development page 'About Temp Files'
Date Thu, 20 Aug 2015 09:18:10 GMT
Repository: incubator-kylin
Updated Branches:
  refs/heads/0.7-staging 19e4c2363 -> 9cbd0e10a


doc, add new Development page 'About Temp Files'


Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/9cbd0e10
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/9cbd0e10
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/9cbd0e10

Branch: refs/heads/0.7-staging
Commit: 9cbd0e10aa3dab59ed2d6c91bbd2bd5fac951ae2
Parents: 19e4c23
Author: Li, Yang <yangli9@ebay.com>
Authored: Thu Aug 20 17:17:59 2015 +0800
Committer: Li, Yang <yangli9@ebay.com>
Committed: Thu Aug 20 17:17:59 2015 +0800

----------------------------------------------------------------------
 website/_data/development.yml    |  2 +-
 website/_dev/about_temp_files.md | 29 +++++++++++++++++++++++++++++
 2 files changed, 30 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/9cbd0e10/website/_data/development.yml
----------------------------------------------------------------------
diff --git a/website/_data/development.yml b/website/_data/development.yml
index 50ac6a4..83d4298 100644
--- a/website/_data/development.yml
+++ b/website/_data/development.yml
@@ -27,4 +27,4 @@
   - new_metadata
   - web_tech
   - test_minicluster
-  
\ No newline at end of file
+  - about_temp_files

http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/9cbd0e10/website/_dev/about_temp_files.md
----------------------------------------------------------------------
diff --git a/website/_dev/about_temp_files.md b/website/_dev/about_temp_files.md
new file mode 100644
index 0000000..b3fba83
--- /dev/null
+++ b/website/_dev/about_temp_files.md
@@ -0,0 +1,29 @@
+---
+layout: dev
+title:  "About Temp Files"
+categories: development
+permalink: /development/about_temp_files.html
+---
+
+As we reviewed the code we found that Kylin left lots of garbage files in:
+1. Local file system of the CLI
+2. HDFS
+3. Local file system of the hadoop nodes.
+
+A ticked was opened to track this issue:
+https://issues.apache.org/jira/browse/KYLIN-926
+
+For future developments, please:
+1. Whenever you want to create temp files at Local, choose
+File.createTempFile or use the folder:
+BatchConstants.CFG_KYLIN_LOCAL_TEMP_DIR(/tmp/kylin), do not randomly use
+another folder in /tmp, it will end up a mess, and look unprofessional.
+
+2. Whenever you create temp files at Local, remember to delete it after
+using it. It's best to use FileUtils.forceDelete, as it also works for
+deleting folders. Try avoid deleteOnExit, in case Kylin exits abnormally.
+
+3. Whenever you want to create files in HDFS, try to create it under
+kylin.hdfs.working.dir or BatchConstants.CFG_KYLIN_HDFS_TEMP_DIR, and
+remember to delete it after it is no longer useful. Try avoid throwing
+everything into hdfs:///tmp and leave it as garbage.


Mime
View raw message