carbondata-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ravipes...@apache.org
Subject [1/2] incubator-carbondata git commit: In case while copying store to hdfs, if stream didn't closed properly, file doesn't get copied to HDFS So we need to propogate exception to executor so that task can be launched again.
Date Mon, 07 Nov 2016 06:03:04 GMT
Repository: incubator-carbondata
Updated Branches:
  refs/heads/master 2cbbe8ac7 -> f6cceaa08


In case while copying store to hdfs, if stream didn't closed properly, file doesn't get copied
to HDFS
So we need to propogate exception to executor so that task can be launched again.


Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/84c57ac4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/84c57ac4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/84c57ac4

Branch: refs/heads/master
Commit: 84c57ac4c060a4f41838922d3a337f6067cd4caa
Parents: 2cbbe8a
Author: Ashok <ashok.blend@gmail.com>
Authored: Thu Nov 3 07:41:07 2016 +0530
Committer: ravipesala <ravi.pesala@gmail.com>
Committed: Mon Nov 7 11:31:39 2016 +0530

----------------------------------------------------------------------
 .../apache/carbondata/core/util/CarbonUtil.java | 21 ++++++++++++++------
 .../store/writer/AbstractFactDataWriter.java    |  3 ++-
 2 files changed, 17 insertions(+), 7 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/84c57ac4/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java b/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
index e1a993a..3c976db 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
@@ -118,18 +118,27 @@ public final class CarbonUtil {
     // Added if to avoid NullPointerException in case one stream is being passed as null
     if (null != streams) {
       for (Closeable stream : streams) {
-        if (null != stream) {
-          try {
-            stream.close();
-          } catch (IOException e) {
-            LOGGER.error("Error while closing stream" + stream);
-          }
+        try {
+          closeStream(stream);
+        } catch (IOException e) {
+          LOGGER.error("Error while closing stream:" + e);
         }
       }
     }
   }
 
   /**
+   * This method closes stream
+   *
+   * @param stream
+   * @throws IOException
+   */
+  public static void closeStream(Closeable stream) throws IOException {
+    if (null != stream) {
+      stream.close();
+    }
+  }
+  /**
    * @param baseStorePath
    * @return
    */

http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/84c57ac4/processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
----------------------------------------------------------------------
diff --git a/processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
b/processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
index f56a5d2..16bd771 100644
--- a/processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
+++ b/processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
@@ -632,7 +632,8 @@ public abstract class AbstractFactDataWriter<T> implements CarbonFactDataWriter<
           .getDataInputStream(localFilePath, FileFactory.getFileType(localFilePath), bufferSize);
       IOUtils.copyBytes(dataInputStream, dataOutputStream, bufferSize);
     } finally {
-      CarbonUtil.closeStreams(dataInputStream, dataOutputStream);
+      CarbonUtil.closeStream(dataInputStream);
+      CarbonUtil.closeStream(dataOutputStream);
     }
   }
 


Mime
View raw message