ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: SQL Engine
Date Wed, 17 Oct 2018 16:47:00 GMT
Hi Igor,

We never deserialize values during query execution. Instead, we copy the
row to heap and extract fields as needed. In general case it is impossible
to avoid reading the whole row because we do not know whether this is
COUNT(*) or COUNT(*) WHERE <something> or COUNT(<something>) WHERE
<something>.
We do handle plain COUNT(*) as speical case and iterate over index only,
but event this simple query cannot avoid row reads in general case when new
snapshot mode is enabled, because an entry in the index may be not visible
to current transaction.

On Wed, Oct 17, 2018 at 7:19 PM igor.tanackovic <igor.tanackovic@gmail.com>
wrote:

> Hello,
>
> Seems that SQL engine always deserialize whole objects instead of using
> just
> SQL enabled fields (annotated with @QuerySqlField). This may have a huge
> impact on Ignite heap usage and GC overhead as well.
>
> For example, we have a cache holding big objects but with only two sql
> query
> fields which for each query execution (SELECT COUNT(*) FROM 'cache')
> consumes large amount on heap memory (~300MB). As a proof of concept, we
> divided the same cache to *index* cache with only sql query field and a
> *data* holding whole object for materialization. The same query (SELECT
> COUNT(*) FROM 'index-cache') consumes ~25 time less memory! The same is
> true
> for all other queries.
>
> The obvious workaround would be to always have separated regions for
> indexes
> (sql query enabled region) and a data/value region for materialization, but
> it might be a good idea to fix this in a systematic way during off heap
> deserialization.
>
> Regards,
> Igor
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message