ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Buck, Robert" <rb...@verisign.com>
Subject RE: ivy:retrieve performance
Date Thu, 14 Jun 2007 16:32:18 GMT
What the performance test did is:

1. increase by powers of 2 a file size variable

2. for each file size, create/open a new file

3. write specified number of bytes to file

4. close file

Tests were performed on two systems:

1. IBM ThinkPad T42 (slow) laptop, stock hardware, 1.5GB RAM

2. Dell Dual CPU, 5150, 64-bit Red Hat 4; (2) SAS 15k 146GB disks in
RAID 1 configuration; 16 GB RAM; this is production grade hardware.

Both sytems produced similar profiles (shape), however, on the
production hardware with Linux (ext2, I believe) the numbers were
substantially greater (makes sense when you consider the fast disks I
have; at 3GB/sec rates, these SAS drives scream).

In both cases buffers were configured for both Old IO and New IO. With
NIO direct buffers were used. In both cases, I determined the ideal
buffer sizes based upon experimentation to be around 32kb - 64 kb. Going
to buffer sizes less than, or greater than, this range reduced overall
throughput.

-Bob

> -----Original Message-----
> From: Xavier Hanin [mailto:xavier.hanin@gmail.com] 
> Sent: Thursday, June 14, 2007 8:38 AM
> To: ivy-user@incubator.apache.org
> Subject: Re: ivy:retrieve performance
> 
> On 6/14/07, Buck, Robert <rbuck@verisign.com> wrote:
> >
> > Folks,
> >
> > We benchmarked a number of JDK IO API's for an internal project. To 
> > neutralize any questions regarding NIO vs Old IO, please 
> take a look 
> > at the attached diagram. These rates will be largely 
> identical on both 
> > Linux and Windows. Blue line NIO, red line old io.
> 
> 
> I've uploaded the image here:
> http://www.flickr.com/photos/41363014@N00/548163404/
> 
> Could you give a little bit more details about how you did 
> the tests: jvm used, size of buffer used for old IO, ...
> 
> According to your tests it seems that NIO should be preferred 
> in any case, it wasn't what some users in the javalobby 
> thread seemed to say. So I wonder what makes the difference.
> 
> Xavier
> 
> If you do not get the attached JPEG file, let me know. I can send it
> > directly to you if you so request.
> >
> > Cheers,
> >
> > Bob
> >
> > > -----Original Message-----
> > > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> > > Sent: Thursday, June 14, 2007 6:22 AM
> > > To: ivy-user@incubator.apache.org
> > > Subject: Re: ivy:retrieve performance
> > >
> > > On 6/14/07, Gilles Scokart <gscokart@gmail.com> wrote:
> > > >
> > > > > 2. The buffer in FileUtils.java is too small. It's set at
> > > 8192. It
> > > > > seems to
> > > > > > be much better for me to set it much larger. This is due to

> > > > > > the fact
> > > > > that
> > > > > > it
> > > > > > needs to read and write simultaneously. The bigger the
> > > buffer is,
> > > > > > the smaller number of time, HD header has to move. For
> > > me, 65536
> > > > > > seems to perform much better but I haven't tried 
> other numbers.
> > > > >
> > > > >
> > > > > I'd like to get more feedback on this. One use case is
> > > not the other.
> > > > This
> > > > > size has been borrowed from Ant copy mechanism. Maybe
> > > what we could
> > > > > do
> > > > is
> > > > > make this configurable, so that one could adapt to its
> > > needs. Or try
> > > > > to guess a good size depending on the size (when it's 
> possible 
> > > > > to get an
> > > > idea
> > > > > of the size before copying).
> > > > >
> > > > > Xavier
> > > > >
> > > >
> > > >
> > > > Couldn't we use the nio for that?  (See
> > > > http://www.javalobby.org/java/forums/t17036.html)
> > >
> > >
> > > According to comments 10 and 11 NIO have bad performance 
> for large 
> > > files on linux, and input stream with byte buffer is 
> pretty close to 
> > > NIO for small files (see comment 13 conclusion). So I'm not sure 
> > > switching to NIO would indeed help a lot. According to 
> the tests in 
> > > the thread you pointed using a 64kB buffer seems to be a 
> good choice 
> > > (which confirms  testn tests), at least for large files. OTOH the 
> > > last conclusion (comment 17) is different.
> > > So I don't really know what to think about that. We 
> should make some 
> > > tests on several platforms and jvms to draw conclusion 
> ourself, but 
> > > it takes time.
> > >
> > >
> > > Xavier
> > >
> > > Gilles
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Xavier Hanin - Independent Java Consultant Manage your 
> dependencies 
> > > with Ivy!
> > > http://incubator.apache.org/ivy/
> > >
> >
> 
> 
> 
> --
> Xavier Hanin - Independent Java Consultant Manage your 
> dependencies with Ivy!
> http://incubator.apache.org/ivy/
> 

Mime
View raw message