cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (Reopened) (JIRA)" <j...@apache.org>
Subject [jira] [Reopened] (CASSANDRA-2189) json2sstable fails due to OutOfMemory
Date Wed, 07 Dec 2011 23:46:39 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Ellis reopened CASSANDRA-2189:
---------------------------------------


This didn't actually fix the problem.  Tatu again:

"This calls configure on parser; but at that point the symbol table has already been created.
So configure must be called on the factory first (and only need to be called once, really),
and then settings will be passed as expected."
                
> json2sstable fails due to OutOfMemory
> -------------------------------------
>
>                 Key: CASSANDRA-2189
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2189
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>    Affects Versions: 0.7.0
>         Environment: linux
>            Reporter: Shotaro Kamio
>            Assignee: Jonathan Ellis
>            Priority: Minor
>             Fix For: 0.7.3
>
>         Attachments: 2189.txt
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> I have a json file created with sstable2json for a column family of super column type.
Its size is about 1.9GB. (It's a dump of all keys because I cannot find out how to specify
keys to dump in sstable2json.)
> When I tried to create sstable from the json file, it failed with OutOfMemoryError as
follows.
>  WARN 00:31:58,595 Schema definitions were defined both locally and in cassandra.yaml.
Definitions in cassandra.yaml were ignored.
> Exception in thread "main" java.lang.OutOfMemoryError: PermGen space
>         at java.lang.String.intern(Native Method)
>         at org.codehaus.jackson.util.InternCache.intern(InternCache.java:40)
>         at org.codehaus.jackson.sym.BytesToNameCanonicalizer.addName(BytesToNameCanonicalizer.java:471)
>         at org.codehaus.jackson.impl.Utf8StreamParser.addName(Utf8StreamParser.java:893)
>         at org.codehaus.jackson.impl.Utf8StreamParser.findName(Utf8StreamParser.java:773)
>         at org.codehaus.jackson.impl.Utf8StreamParser.parseLongFieldName(Utf8StreamParser.java:379)
>         at org.codehaus.jackson.impl.Utf8StreamParser.parseMediumFieldName(Utf8StreamParser.java:347)
>         at org.codehaus.jackson.impl.Utf8StreamParser._parseFieldName(Utf8StreamParser.java:304)
>         at org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:140)
>         at org.codehaus.jackson.map.deser.UntypedObjectDeserializer.mapObject(UntypedObjectDeserializer.java:93)
>         at org.codehaus.jackson.map.deser.UntypedObjectDeserializer.deserialize(UntypedObjectDeserializer.java:65)
>         at org.codehaus.jackson.map.deser.MapDeserializer._readAndBind(MapDeserializer.java:197)
>         at org.codehaus.jackson.map.deser.MapDeserializer.deserialize(MapDeserializer.java:145)
>         at org.codehaus.jackson.map.deser.MapDeserializer.deserialize(MapDeserializer.java:23)
>         at org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:1261)
>         at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:517)
>         at org.codehaus.jackson.JsonParser.readValueAs(JsonParser.java:897)
>         at org.apache.cassandra.tools.SSTableImport.importUnsorted(SSTableImport.java:208)
>         at org.apache.cassandra.tools.SSTableImport.importJson(SSTableImport.java:197)
>         at org.apache.cassandra.tools.SSTableImport.main(SSTableImport.java:421)
> So, what I had to is that split the json file with "split" command and modify them to
be correct json file. Create sstable for each small json files.
> Could you change json2sstable to avoid OutOfMemory?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message