ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From testn <te...@doramail.com>
Subject Re: ivy:retrieve performance
Date Mon, 02 Jul 2007 01:48:15 GMT

IVY-551 has been opened for this.

Thanks,


Xavier Hanin wrote:
> 
> On 6/21/07, testn <test1@doramail.com> wrote:
>>
>>
>> Hi Xavier,
> 
> 
> Hi, but this question is not only for me, Ivy is a community project where
> the community takes decisions.
> 
> Any plan to put it in alpha2? It doesn't look like it requires a lot of
> work
>> if we just want to change the buffer size. nio may be a long shot.
> 
> 
> I'm +1 to increase the buffer to 64kB. I'm +1 to open an issue for this,
> and
> another one to later further improve this with NIO. What do others think?
> 
> Xavier
> 
> Thanks
>>
>>
>> Xavier Hanin wrote:
>> >
>> > Great, thanks a lot for these data!
>> >
>> > Xavier
>> >
>> > On 6/14/07, Buck, Robert <rbuck@verisign.com> wrote:
>> >>
>> >> What the performance test did is:
>> >>
>> >> 1. increase by powers of 2 a file size variable
>> >>
>> >> 2. for each file size, create/open a new file
>> >>
>> >> 3. write specified number of bytes to file
>> >>
>> >> 4. close file
>> >>
>> >> Tests were performed on two systems:
>> >>
>> >> 1. IBM ThinkPad T42 (slow) laptop, stock hardware, 1.5GB RAM
>> >>
>> >> 2. Dell Dual CPU, 5150, 64-bit Red Hat 4; (2) SAS 15k 146GB disks in
>> >> RAID 1 configuration; 16 GB RAM; this is production grade hardware.
>> >>
>> >> Both sytems produced similar profiles (shape), however, on the
>> >> production hardware with Linux (ext2, I believe) the numbers were
>> >> substantially greater (makes sense when you consider the fast disks I
>> >> have; at 3GB/sec rates, these SAS drives scream).
>> >>
>> >> In both cases buffers were configured for both Old IO and New IO. With
>> >> NIO direct buffers were used. In both cases, I determined the ideal
>> >> buffer sizes based upon experimentation to be around 32kb - 64 kb.
>> Going
>> >> to buffer sizes less than, or greater than, this range reduced overall
>> >> throughput.
>> >>
>> >> -Bob
>> >>
>> >> > -----Original Message-----
>> >> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
>> >> > Sent: Thursday, June 14, 2007 8:38 AM
>> >> > To: ivy-user@incubator.apache.org
>> >> > Subject: Re: ivy:retrieve performance
>> >> >
>> >> > On 6/14/07, Buck, Robert <rbuck@verisign.com> wrote:
>> >> > >
>> >> > > Folks,
>> >> > >
>> >> > > We benchmarked a number of JDK IO API's for an internal project.
>> To
>> >> > > neutralize any questions regarding NIO vs Old IO, please
>> >> > take a look
>> >> > > at the attached diagram. These rates will be largely
>> >> > identical on both
>> >> > > Linux and Windows. Blue line NIO, red line old io.
>> >> >
>> >> >
>> >> > I've uploaded the image here:
>> >> > http://www.flickr.com/photos/41363014@N00/548163404/
>> >> >
>> >> > Could you give a little bit more details about how you did
>> >> > the tests: jvm used, size of buffer used for old IO, ...
>> >> >
>> >> > According to your tests it seems that NIO should be preferred
>> >> > in any case, it wasn't what some users in the javalobby
>> >> > thread seemed to say. So I wonder what makes the difference.
>> >> >
>> >> > Xavier
>> >> >
>> >> > If you do not get the attached JPEG file, let me know. I can send it
>> >> > > directly to you if you so request.
>> >> > >
>> >> > > Cheers,
>> >> > >
>> >> > > Bob
>> >> > >
>> >> > > > -----Original Message-----
>> >> > > > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
>> >> > > > Sent: Thursday, June 14, 2007 6:22 AM
>> >> > > > To: ivy-user@incubator.apache.org
>> >> > > > Subject: Re: ivy:retrieve performance
>> >> > > >
>> >> > > > On 6/14/07, Gilles Scokart <gscokart@gmail.com> wrote:
>> >> > > > >
>> >> > > > > > 2. The buffer in FileUtils.java is too small. It's
set at
>> >> > > > 8192. It
>> >> > > > > > seems to
>> >> > > > > > > be much better for me to set it much larger.
This is due
>> to
>> >> > > > > > > the fact
>> >> > > > > > that
>> >> > > > > > > it
>> >> > > > > > > needs to read and write simultaneously. The
bigger the
>> >> > > > buffer is,
>> >> > > > > > > the smaller number of time, HD header has
to move. For
>> >> > > > me, 65536
>> >> > > > > > > seems to perform much better but I haven't
tried
>> >> > other numbers.
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > I'd like to get more feedback on this. One use
case is
>> >> > > > not the other.
>> >> > > > > This
>> >> > > > > > size has been borrowed from Ant copy mechanism.
Maybe
>> >> > > > what we could
>> >> > > > > > do
>> >> > > > > is
>> >> > > > > > make this configurable, so that one could adapt
to its
>> >> > > > needs. Or try
>> >> > > > > > to guess a good size depending on the size (when
it's
>> >> > possible
>> >> > > > > > to get an
>> >> > > > > idea
>> >> > > > > > of the size before copying).
>> >> > > > > >
>> >> > > > > > Xavier
>> >> > > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > Couldn't we use the nio for that?  (See
>> >> > > > > http://www.javalobby.org/java/forums/t17036.html)
>> >> > > >
>> >> > > >
>> >> > > > According to comments 10 and 11 NIO have bad performance
>> >> > for large
>> >> > > > files on linux, and input stream with byte buffer is
>> >> > pretty close to
>> >> > > > NIO for small files (see comment 13 conclusion). So I'm not
sure
>> >> > > > switching to NIO would indeed help a lot. According to
>> >> > the tests in
>> >> > > > the thread you pointed using a 64kB buffer seems to be a
>> >> > good choice
>> >> > > > (which confirms  testn tests), at least for large files.
OTOH
>> the
>> >> > > > last conclusion (comment 17) is different.
>> >> > > > So I don't really know what to think about that. We
>> >> > should make some
>> >> > > > tests on several platforms and jvms to draw conclusion
>> >> > ourself, but
>> >> > > > it takes time.
>> >> > > >
>> >> > > >
>> >> > > > Xavier
>> >> > > >
>> >> > > > Gilles
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > > >
>> >> > > > --
>> >> > > > Xavier Hanin - Independent Java Consultant Manage your
>> >> > dependencies
>> >> > > > with Ivy!
>> >> > > > http://incubator.apache.org/ivy/
>> >> > > >
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Xavier Hanin - Independent Java Consultant Manage your
>> >> > dependencies with Ivy!
>> >> > http://incubator.apache.org/ivy/
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > Xavier Hanin - Independent Java Consultant
>> > Manage your dependencies with Ivy!
>> > http://incubator.apache.org/ivy/
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11230918
>> Sent from the ivy-user mailing list archive at Nabble.com.
>>
>>
> 
> 
> -- 
> Xavier Hanin - Independent Java Consultant
> Manage your dependencies with Ivy!
> http://incubator.apache.org/ivy/
> 
> 

-- 
View this message in context: http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html#a11387196
Sent from the ivy-user mailing list archive at Nabble.com.


Mime
View raw message