parquet-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From w...@apache.org
Subject parquet-cpp git commit: PARQUET-775: Make TrackingAllocator thread-safe
Date Wed, 09 Nov 2016 23:51:06 GMT
Repository: parquet-cpp
Updated Branches:
  refs/heads/master 3fa46d121 -> 002466539


PARQUET-775: Make TrackingAllocator thread-safe

Mutex is only locked around the statistics as the other parts are
thread-safe and should run in parallel if needed/possible.

Author: Uwe L. Korn <uwelk@xhochy.com>

Closes #190 from xhochy/PARQUET-775 and squashes the following commits:

18fabb7 [Uwe L. Korn] PARQUET-775: Make TrackingAllocator thread-safe


Project: http://git-wip-us.apache.org/repos/asf/parquet-cpp/repo
Commit: http://git-wip-us.apache.org/repos/asf/parquet-cpp/commit/00246653
Tree: http://git-wip-us.apache.org/repos/asf/parquet-cpp/tree/00246653
Diff: http://git-wip-us.apache.org/repos/asf/parquet-cpp/diff/00246653

Branch: refs/heads/master
Commit: 002466539f6aba7bf1f885b66f61f302ed88fa6b
Parents: 3fa46d1
Author: Uwe L. Korn <uwelk@xhochy.com>
Authored: Wed Nov 9 18:50:59 2016 -0500
Committer: Wes McKinney <wes.mckinney@twosigma.com>
Committed: Wed Nov 9 18:50:59 2016 -0500

----------------------------------------------------------------------
 src/parquet/util/mem-allocator.cc | 16 +++++++++++-----
 src/parquet/util/mem-allocator.h  |  2 ++
 2 files changed, 13 insertions(+), 5 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/00246653/src/parquet/util/mem-allocator.cc
----------------------------------------------------------------------
diff --git a/src/parquet/util/mem-allocator.cc b/src/parquet/util/mem-allocator.cc
index 4d42bc4..2b6592d 100644
--- a/src/parquet/util/mem-allocator.cc
+++ b/src/parquet/util/mem-allocator.cc
@@ -30,17 +30,23 @@ uint8_t* TrackingAllocator::Malloc(int64_t size) {
 
   uint8_t* p = static_cast<uint8_t*>(std::malloc(size));
   if (!p) { throw ParquetException("OOM: memory allocation failed"); }
-  total_memory_ += size;
-  if (total_memory_ > max_memory_) { max_memory_ = total_memory_; }
+  {
+    std::lock_guard<std::mutex> lock(stats_mutex_);
+    total_memory_ += size;
+    if (total_memory_ > max_memory_) { max_memory_ = total_memory_; }
+  }
   return p;
 }
 
 void TrackingAllocator::Free(uint8_t* p, int64_t size) {
   if (nullptr != p && size > 0) {
-    if (total_memory_ < size) {
-      throw ParquetException("Attempting to free too much memory");
+    {
+      std::lock_guard<std::mutex> lock(stats_mutex_);
+      if (total_memory_ < size) {
+        throw ParquetException("Attempting to free too much memory");
+      }
+      total_memory_ -= size;
     }
-    total_memory_ -= size;
     std::free(p);
   }
 }

http://git-wip-us.apache.org/repos/asf/parquet-cpp/blob/00246653/src/parquet/util/mem-allocator.h
----------------------------------------------------------------------
diff --git a/src/parquet/util/mem-allocator.h b/src/parquet/util/mem-allocator.h
index b20d9f9..a0f3693 100644
--- a/src/parquet/util/mem-allocator.h
+++ b/src/parquet/util/mem-allocator.h
@@ -19,6 +19,7 @@
 #define PARQUET_UTIL_MEMORY_POOL_H
 
 #include <cstdint>
+#include <mutex>
 
 #include "parquet/util/visibility.h"
 
@@ -48,6 +49,7 @@ class PARQUET_EXPORT TrackingAllocator : public MemoryAllocator {
   int64_t MaxMemory() { return max_memory_; }
 
  private:
+  std::mutex stats_mutex_;
   int64_t total_memory_;
   int64_t max_memory_;
 };


Mime
View raw message