directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eric tse <hfe...@gmail.com>
Subject Re: parseLdifFile Run out of memory
Date Wed, 29 Jan 2014 17:38:55 GMT
Thanks Kiran,

Can you point me out any simple sample java code that demonstrate the most
basic concept of using "iterate on the reader using reader.next()"?

Thanks and regards,
Eric



On Wed, Jan 29, 2014 at 9:12 AM, Kiran Ayyagari <kayyagari@apache.org>wrote:

>
>
>
> On Wed, Jan 29, 2014 at 9:23 PM, eric tse <hfetse@gmail.com> wrote:
>
>> Hello
>>
>> I was using parseLdifFile to parse my LDIF file.
>> When the LDIF size is small, then it is okay.
>> But if the LDIF file size is 500M, with 1 million entries.. then it runs
>> out of memory
>> even I assign 6G memory to java heap
>>
>> Can you please tell me if I just don't give them enough memory (how much)
>> or
>> this library does not support large LDIF size (can you please point me
>> out what library I should use) ?
>>
>> LdifReader reader = new LdifReader();
>>  List<LdifEntry> entries = reader.parseLdifFile(
>> "/home/myname/export.ldif");
>>
>> you are parsing the entire file and keeping the entries in memory, do not
> use this method,
> instead iterate on the reader using reader.next(), this will fix your
> issue.
>
>> It is the hash map throwing the out of memory exception.
>>
>>
>> *Attached the exception*
>>
>> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>>
>>         at java.util.HashMap.<init>(Unknown Source)
>>
>>         at java.util.LinkedHashMap.<init>(Unknown Source)
>>
>>         at java.util.HashSet.<init>(Unknown Source)
>>
>>         at java.util.LinkedHashSet.<init>(Unknown Source)
>>
>>         at
>> org.apache.directory.api.ldap.model.entry.DefaultAttribute.<init>(Def
>>
>> aultAttribute.java:57)
>>
>>         at
>> org.apache.directory.api.ldap.model.entry.DefaultEntry.add(DefaultEnt
>>
>> ry.java:921)
>>
>>         at
>> org.apache.directory.api.ldap.model.ldif.LdifEntry.addAttribute(LdifE
>>
>> ntry.java:444)
>>
>>         at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.parseAttributeVal
>>
>> ue(LdifReader.java:942)
>>
>>         at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.parseEntry(LdifRe
>>
>> ader.java:1403)
>>
>>         at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.nextInternal(Ldif
>>
>> Reader.java:1755)
>>
>>         at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.access$100(LdifRe
>>
>> ader.java:168)
>>
>>         at
>> org.apache.directory.api.ldap.model.ldif.LdifReader$1.next(LdifReader
>>
>> .java:1859)
>>
>>         at
>> org.apache.directory.api.ldap.model.ldif.LdifReader$1.next(LdifReader
>>
>> .java:1850)
>>
>>         at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.parseLdif(LdifRea
>>
>> der.java:1911)
>>
>>         at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.parseLdifFile(Ldi
>>
>> fReader.java:1652)
>>
>>         at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.parseLdifFile(Ldi
>>
>> fReader.java:1616)
>>
>>         at
>> public_register_not_enrolled.main(public_register_not_enrolled.java:5
>>
>> 0)
>>
>>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>

Mime
View raw message