calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Feng Zhu <wellfeng...@gmail.com>
Subject [DISCUSSION] Cache Optimization in JdbcSchema
Date Tue, 29 Oct 2019 02:28:57 GMT
Hi all,
We made some optimizations in practice. But I'm not sure whether this kind
of change is necessary to the community, because it will make the code
complex.

Current now, JdbcSchema caches all JdbcTables in tableMap (i.e.,*
ImmutableMap<String, JdbcTable> tableMap*)

In our production environment, there are about 3000+ datasources and
correspondingly creating 3000+ JdbcSchemas, while each JdbcSchema may
contain up to 10000+ tables.Consequently, the table map occupies nearly
10GB memory, bringing great pressure on the server.

We encode <*catalogName, schemaName, tableTypeName*> tuple as unique
Integer, and simplify the table map as <*String, Integer*>. According to
the Integer, we can find tuple and construct JdbcTable dynamically. Benefit
from this, the cached table map costs only about 800MB memory.

Best,
DonnyZone

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message