calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danny Chan <yuzhao....@gmail.com>
Subject Re: [DISCUSSION] Cache Optimization in JdbcSchema
Date Tue, 29 Oct 2019 04:10:43 GMT
Sounds very attractive, could you give an intuitive design doc to illustrate how it works ?
And we may review the design then ;)

Best,
Danny Chan
在 2019年10月29日 +0800 AM10:36,Feng Zhu <wellfengzhu@gmail.com>,写道:
> Hi all,
> We made some optimizations in practice. But I'm not sure whether this kind
> of change is necessary to the community, because it will make the code
> complex.
>
> Current now, JdbcSchema caches all JdbcTables in tableMap (i.e.,*
> ImmutableMap<String, JdbcTable> tableMap*)
>
> In our production environment, there are about 3000+ datasources and
> correspondingly creating 3000+ JdbcSchemas, while each JdbcSchema may
> contain up to 10000+ tables.Consequently, the table map occupies nearly
> 10GB memory, bringing great pressure on the server.
>
> We encode <*catalogName, schemaName, tableTypeName*> tuple as unique
> Integer, and simplify the table map as <*String, Integer*>. According to
> the Integer, we can find tuple and construct JdbcTable dynamically. Benefit
> from this, the cached table map costs only about 800MB memory.
>
> Best,
> DonnyZone

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message