xerces-c-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Harm" <Martin.H...@cimt-ag.de>
Subject AW: Limitation of file size with DOM
Date Sun, 14 Jan 2007 12:06:47 GMT
Hi,
well it didnt work just rigth now:
 - xml-file: 661M 14. Jan 12:29 accountStatement.400000.xml
 - call: xerces-c-src_2_7_0/bin/DOMCount accountStatement.400000.xml
 - on:
 uname: Linux tolero 2.6.18-1.2257.fc5 #1 Fri Dec 15 16:06:24 EST 2006 i686 athlon i386 GNU/Linux
 gcc:  gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
 free
             total       used       free     shared    buffers     cached
Mem:       1555628     389748    1165880          0      23496     260140
-/+ buffers/cache:     106112    1449516
Swap:      1060208          0    1060208
 
because of abolute no memory left .. the system freezes, and I have to reset it.

The symtoms of having strange errors occured on older versions of xerces and linux
and are not reproducable on my current system.

You should test it on your own.. mayby this litte perl script helps on generating
large XML-Files.

use strict;
my @cnt = ( 1,3, 100, 200, 5000, 10000, 20000, 40000, 50000, 60000,80000, 100000,200000,400000,800000);

my $baseName=shift;

open FF,"$baseName.header"; my @headerLines=<FF>; close FF;
open FF,"$baseName.entry"; my @contentLines=<FF>;close FF;
open FF,"$baseName.footer";my @footerLines=<FF>;close FF;

foreach my $entryCnt (@cnt) {
    my $fname = sprintf("$baseName.%d.xml",$entryCnt);
    print "Building: $fname\n";
    my $fout;
    open($fout, ">$fname");
    print $fout  @headerLines;
    for(my $i=0; $i<$entryCnt;$i++) {
        print $fout @contentLines;
    }
    print $fout  @footerLines;
    close($fout);

}
1;

ciao
martin



-----Urspr√ľngliche Nachricht-----
Von: Kosaraju Karuna-C9013Z [mailto:C9013Z@motorola.com]
Gesendet: Fr 2007-01-12 18:04
An: c-users@xerces.apache.org
Cc: jason.e.stewart@gmail.com; Martin Harm
Betreff: RE: Limitation of file size with DOM
 
Thanks Jason and Martin for the replies,
I am now confused with the contradictory experiences of both of you.
Is it possible to let me know the xerces version, OS, kernel version
etc., on which you have seen this.
I just wanted to understand the scenario where it worked (for Jason) and
it dint work (for Martin)

Thanks again,
Karuna

-----Original Message-----
From: Jason Stewart [mailto:jason.e.stewart@gmail.com] 
Sent: Thursday, January 11, 2007 11:21 PM
To: c-users@xerces.apache.org
Subject: Re: Limitation of file size with DOM

Hi,

On 1/11/07, Kosaraju Karuna-C9013Z <C9013Z@motorola.com> wrote:
> Hi,
>
> We are using Xserces-c DOM APIS to parse our applications XML file. 
> Our XML file size may go to MB.
> Wanted to know whether there is any restriction on xerces-c DOM on the

> size of the file it can handle.
> The version we are using is xercesc-c.2.6.0. Any replies would be 
> appreciated
>

I've used Xerces to parse file sizes in the 500Mb range...

The only limitation I had was on the memory of the machine...

Cheers, jas.


Hi,
 
I tested parsing large files. The memory consumption was aprox. 7 to 10
times of the XML-File-Size (pretty printed). But there was at least a
limit on the size of the XML-File (aprox. 250Mb) where the xerces (2.6)
bounces with strange errors.. and there *was* still some RAM left :-(
(tested on linux). So I was forced to switch to SAX.
 
cu
martin


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message