ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xavier Hanin" <xavier.ha...@gmail.com>
Subject Re: ivy:retrieve performance
Date Mon, 25 Jun 2007 07:10:34 GMT
On 6/25/07, Jan.Materne@rzf.fin-nrw.de <Jan.Materne@rzf.fin-nrw.de> wrote:
>
> Could the performance increase when downloading artefacts parallel?


I think we already have an issue suggesting the download in parallel, which
makes sense especially when artifacts are downloaded from different
locations.

The problem is that this issue is not easy to solve, because Ivy has not
been very well designed for thread safety. Another problem is download
progress reporting which is not easy when artifacts are downloaded
concurrently. So this issue is not scheduled for 2.0 ATM.

Xavier

Jan
>
> >-----Urspr√ľngliche Nachricht-----
> >Von: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> >Gesendet: Freitag, 22. Juni 2007 11:28
> >An: ivy-user@incubator.apache.org
> >Betreff: Re: ivy:retrieve performance
> >
> >On 6/21/07, testn <test1@doramail.com> wrote:
> >>
> >>
> >> Hi Xavier,
> >
> >
> >Hi, but this question is not only for me, Ivy is a community
> >project where
> >the community takes decisions.
> >
> >Any plan to put it in alpha2? It doesn't look like it requires
> >a lot of work
> >> if we just want to change the buffer size. nio may be a long shot.
> >
> >
> >I'm +1 to increase the buffer to 64kB. I'm +1 to open an issue
> >for this, and
> >another one to later further improve this with NIO. What do
> >others think?
> >
> >Xavier
> >
> >Thanks
> >>
> >>
> >> Xavier Hanin wrote:
> >> >
> >> > Great, thanks a lot for these data!
> >> >
> >> > Xavier
> >> >
> >> > On 6/14/07, Buck, Robert <rbuck@verisign.com> wrote:
> >> >>
> >> >> What the performance test did is:
> >> >>
> >> >> 1. increase by powers of 2 a file size variable
> >> >>
> >> >> 2. for each file size, create/open a new file
> >> >>
> >> >> 3. write specified number of bytes to file
> >> >>
> >> >> 4. close file
> >> >>
> >> >> Tests were performed on two systems:
> >> >>
> >> >> 1. IBM ThinkPad T42 (slow) laptop, stock hardware, 1.5GB RAM
> >> >>
> >> >> 2. Dell Dual CPU, 5150, 64-bit Red Hat 4; (2) SAS 15k
> >146GB disks in
> >> >> RAID 1 configuration; 16 GB RAM; this is production grade
> >hardware.
> >> >>
> >> >> Both sytems produced similar profiles (shape), however, on the
> >> >> production hardware with Linux (ext2, I believe) the numbers were
> >> >> substantially greater (makes sense when you consider the
> >fast disks I
> >> >> have; at 3GB/sec rates, these SAS drives scream).
> >> >>
> >> >> In both cases buffers were configured for both Old IO and
> >New IO. With
> >> >> NIO direct buffers were used. In both cases, I determined
> >the ideal
> >> >> buffer sizes based upon experimentation to be around 32kb - 64 kb.
> >> Going
> >> >> to buffer sizes less than, or greater than, this range
> >reduced overall
> >> >> throughput.
> >> >>
> >> >> -Bob
> >> >>
> >> >> > -----Original Message-----
> >> >> > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> >> >> > Sent: Thursday, June 14, 2007 8:38 AM
> >> >> > To: ivy-user@incubator.apache.org
> >> >> > Subject: Re: ivy:retrieve performance
> >> >> >
> >> >> > On 6/14/07, Buck, Robert <rbuck@verisign.com> wrote:
> >> >> > >
> >> >> > > Folks,
> >> >> > >
> >> >> > > We benchmarked a number of JDK IO API's for an
> >internal project. To
> >> >> > > neutralize any questions regarding NIO vs Old IO, please
> >> >> > take a look
> >> >> > > at the attached diagram. These rates will be largely
> >> >> > identical on both
> >> >> > > Linux and Windows. Blue line NIO, red line old io.
> >> >> >
> >> >> >
> >> >> > I've uploaded the image here:
> >> >> > http://www.flickr.com/photos/41363014@N00/548163404/
> >> >> >
> >> >> > Could you give a little bit more details about how you did
> >> >> > the tests: jvm used, size of buffer used for old IO, ...
> >> >> >
> >> >> > According to your tests it seems that NIO should be preferred
> >> >> > in any case, it wasn't what some users in the javalobby
> >> >> > thread seemed to say. So I wonder what makes the difference.
> >> >> >
> >> >> > Xavier
> >> >> >
> >> >> > If you do not get the attached JPEG file, let me know.
> >I can send it
> >> >> > > directly to you if you so request.
> >> >> > >
> >> >> > > Cheers,
> >> >> > >
> >> >> > > Bob
> >> >> > >
> >> >> > > > -----Original Message-----
> >> >> > > > From: Xavier Hanin [mailto:xavier.hanin@gmail.com]
> >> >> > > > Sent: Thursday, June 14, 2007 6:22 AM
> >> >> > > > To: ivy-user@incubator.apache.org
> >> >> > > > Subject: Re: ivy:retrieve performance
> >> >> > > >
> >> >> > > > On 6/14/07, Gilles Scokart <gscokart@gmail.com>
wrote:
> >> >> > > > >
> >> >> > > > > > 2. The buffer in FileUtils.java is too small.
> >It's set at
> >> >> > > > 8192. It
> >> >> > > > > > seems to
> >> >> > > > > > > be much better for me to set it much
larger.
> >This is due to
> >> >> > > > > > > the fact
> >> >> > > > > > that
> >> >> > > > > > > it
> >> >> > > > > > > needs to read and write simultaneously.
The bigger the
> >> >> > > > buffer is,
> >> >> > > > > > > the smaller number of time, HD header
has to move. For
> >> >> > > > me, 65536
> >> >> > > > > > > seems to perform much better but I haven't
tried
> >> >> > other numbers.
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > > > I'd like to get more feedback on this. One
use case is
> >> >> > > > not the other.
> >> >> > > > > This
> >> >> > > > > > size has been borrowed from Ant copy mechanism.
Maybe
> >> >> > > > what we could
> >> >> > > > > > do
> >> >> > > > > is
> >> >> > > > > > make this configurable, so that one could
adapt to its
> >> >> > > > needs. Or try
> >> >> > > > > > to guess a good size depending on the size
(when it's
> >> >> > possible
> >> >> > > > > > to get an
> >> >> > > > > idea
> >> >> > > > > > of the size before copying).
> >> >> > > > > >
> >> >> > > > > > Xavier
> >> >> > > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > Couldn't we use the nio for that?  (See
> >> >> > > > > http://www.javalobby.org/java/forums/t17036.html)
> >> >> > > >
> >> >> > > >
> >> >> > > > According to comments 10 and 11 NIO have bad performance
> >> >> > for large
> >> >> > > > files on linux, and input stream with byte buffer is
> >> >> > pretty close to
> >> >> > > > NIO for small files (see comment 13 conclusion). So
> >I'm not sure
> >> >> > > > switching to NIO would indeed help a lot. According
to
> >> >> > the tests in
> >> >> > > > the thread you pointed using a 64kB buffer seems to
be a
> >> >> > good choice
> >> >> > > > (which confirms  testn tests), at least for large
> >files. OTOH the
> >> >> > > > last conclusion (comment 17) is different.
> >> >> > > > So I don't really know what to think about that. We
> >> >> > should make some
> >> >> > > > tests on several platforms and jvms to draw conclusion
> >> >> > ourself, but
> >> >> > > > it takes time.
> >> >> > > >
> >> >> > > >
> >> >> > > > Xavier
> >> >> > > >
> >> >> > > > Gilles
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > > >
> >> >> > > > --
> >> >> > > > Xavier Hanin - Independent Java Consultant Manage your
> >> >> > dependencies
> >> >> > > > with Ivy!
> >> >> > > > http://incubator.apache.org/ivy/
> >> >> > > >
> >> >> > >
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Xavier Hanin - Independent Java Consultant Manage your
> >> >> > dependencies with Ivy!
> >> >> > http://incubator.apache.org/ivy/
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Xavier Hanin - Independent Java Consultant
> >> > Manage your dependencies with Ivy!
> >> > http://incubator.apache.org/ivy/
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> >http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html
> >#a11230918
> >> Sent from the ivy-user mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> >--
> >Xavier Hanin - Independent Java Consultant
> >Manage your dependencies with Ivy!
> >http://incubator.apache.org/ivy/
> >
>



-- 
Xavier Hanin - Independent Java Consultant
Manage your dependencies with Ivy!
http://incubator.apache.org/ivy/

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message