avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thiruvalluvan M. G. (JIRA)" <j...@apache.org>
Subject [jira] Commented: (AVRO-210) Memory leak with recursive schemas when constructed by hand
Date Wed, 18 Nov 2009 14:02:39 GMT

    [ https://issues.apache.org/jira/browse/AVRO-210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779458#action_12779458
] 

Thiruvalluvan M. G. commented on AVRO-210:
------------------------------------------

{quote}
The leak should not happen when we build the schema by parsing a JSON schema file. This is
because the current implementation does not use pointers for symbolic links; it uses symbols
and there is a symbol table that resolves the symbols at runtime.
{quote}

With the recent patch due to JIRA-204, the problem will occur even when nested schemas are
built from JSON files.

> Memory leak with recursive schemas when constructed by hand
> -----------------------------------------------------------
>
>                 Key: AVRO-210
>                 URL: https://issues.apache.org/jira/browse/AVRO-210
>             Project: Avro
>          Issue Type: Bug
>          Components: c++
>            Reporter: Thiruvalluvan M. G.
>
> Schema consists of a node or bunch of nodes. These nodes are represented as intrusive
pointers of nodes (NodPtr). Since the intrusive pointers use reference counts, recursive schemas
which result in cycles of intrusive pointers lead to memory leak. The following code, when
compiled and run, causes the memory to grow steadily:
> {code:title=test.cc|borderStyle=solid}
> #include <unistd.h>
> #include "Schema.hh"
> int main(int argc, char** argv)
> {
>     const int count1 = 10;
>     const int count2 = 1000;
>     for (int i = 0; i < count1; i++) {
>         for (int j = 0; j < count2; j++) {
>             avro::RecordSchema rec("LongList");
>             rec.addField("value", avro::LongSchema());
>             avro::UnionSchema next;
>             next.addType(avro::NullSchema());
>             next.addType(rec);
>             rec.addField("next", next);
>             rec.addField("end", avro::BoolSchema());
>         }
>         sleep(1);
>     }
> }
> {code}
> The leak should not happen when we build the schema by parsing a JSON schema file. This
is because the current implementation does not use pointers for symbolic links; it uses symbols
and there is a symbol table that resolves the symbols at runtime. But unfortunately the nested
schema file generates an error. I'll file a separate JIRA for that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message