impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Behm (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] [PREVIEW] Use native allocation while building catalog updates
Date Tue, 05 Sep 2017 17:29:40 GMT
Alex Behm has posted comments on this change.

Change subject: [PREVIEW] Use native allocation while building catalog updates
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/7955/1/be/src/catalog/catalog.cc
File be/src/catalog/catalog.cc:

PS1, Line 113: free(buf);
> Switching the JVM to allocate huge chunks of memory via malloc is going to 
1. This code path is catalogd only.
2. Roughly the same amount of untracked memory is already allocated today when creating the
thrift topic update to the statestore (data lives in std::string). Imo, this patch makes the
untracked memory problem no worse than it already is - fair point that it does not make it
better either.
3. The 2GB JVM array limit means the catalogd goes down for large catalogs, and is very hard
to recover, so we really need to solve this issue. I can imagine alternative solutions that
involve copying large amounts of data back-and-forth through JNI (e.g. implementing a chunked
ByteArrayOutputStream)

Do you have an alternative proposal?


-- 
To view, visit http://gerrit.cloudera.org:8080/7955
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I383684effa9524734ce3c6c0fb7ed37de0e15782
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bharathv@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bharathv@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message