maven-wagon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Maczka <mmac...@interia.pl>
Subject Re: New interfaces in Wagon
Date Thu, 01 Apr 2004 06:05:05 GMT
Jason van Zyl wrote:

[...]

>>Other example:
>>When we are searching for the latest version of the snapshot artifact,
>>we need to fetch from all visible repositories xx.artifact-version file
>>and find that repository whic contains the latest version of the given snapshot and
compare it with the local version.
>>This file is quite small but still we don't want to save dozen of such files
>>to disk (finding unique name for each) and then read from them and delete them right
after we are done with each of them.
>>    
>>
>
>I don't see what the big deal is, so we save a bunch of files to disk
>first. But the API can be expanded out again and the first client Wagon
>will be fully integrated into will be Maven. Let me wire Wagon in
>completely to Maven and see what happens.
>
>  
>

I am right now using quite a lot of large artifacts (mainly wars) - each 
of them has - 20 MB+.
In the solution I have proposed it was possible to do the multiplexing 
of streams and for example in single pass transfer a file to
the repository and compute in memory it's md5 and sha1 checksum and then 
transfer thow two strings as artifiates directly from memory
If you have to read 20 MB three time from disk performance starts to matter.
For example in place of 7 seconds you might have 18 (I don't know real  
numbers!)

>>My implementation of Wagon introduced 2 key interfaces: Wagon and Repository.
>>
>>I imagined theat simplification of usage of the API will be implemented one level
higher than that.
>>
>>I was actually thinking about something like:
>>
>>class ArtifactRepository
>>{
>>     
>>     WagonFactory wagonFactory;
>>
>>     Repository  repository
>>     
>>     File  getToFile( Artifact artifat ) throws ....;
>>
>>     String  getToString( Artifact artifat ); 
>>     
>>     /**
>>      * Controls if md5 checksum are computed and verified
>>      * during "get" operation and md5 checkusum are automatically
>>      * computed and transfered
>>      * to the repository during "put" operation (this means 2 calls to  
>>      * wagon.put are made - one for artifact second for md5sum
>>      */
>>     void useMd5s( boolean value);
>>
>>     void useSha1s( boolean value );
>>     
>>     ...
>>
>>}
>>    
>>
>
>I don't think mixing verification into the actual artifact is a good
>idea and there is no notion of searching in Wagon but we could add that.
>
>  
>
It is not mixing of concern. Any "artifact verifier" can be plugged in 
dynamically into the process. Even such verifiers which validate if 
given file is valid XML file
or if given POM is valid.  I just don't think that if you want to use 
100 artifact validators to validate given artificat
you should read the same file 100 times ( I am exagerating here) . This 
just don't scale well. You are validating the content.
So you can plug content validatorsas early as that content arrives.
I know that in reality we will have 2 maybe 3 validators...

>>etc. So users will even don't have to know that wagon exists. 
>>
>>
>>I think that last changes in the API over simplified it and excluded some nice possibilties
while the simplicity level for the end user is still not optimal. 
>>    
>>
>
>I'm going to go from end-to-end today with integrating Wagon into Maven
>so I'll be the first to tell you. But again, there were barely any tests
>for anything so I went to town.
>

I don't agree that everything was that badly tested! Wagon API  module 
was tested in 98% (clover)  and that's where all those things were 
implemented.
You can hardly do any better!!!
In there I  defined interfaces like WagonSource  WagonResult  and 
provided fully tested implementation of them like for example 
MemoryResult which I wanted to use for transfering "to memory". Only 
thing which was abolutly not tested were Wagon Providers.
But I completly agree that some intention which I had weren't that 
obvious and clear and still that many thing could be simplified. E.g 
instead of WagonResult probably just OutputStream might be used. That's 
why  I am trying to explain some things now and I am glad that finally I 
have to defend my vison.
For example the exmplanaion why such "artificial" things like event in 
Wagon exists is simple. 
I wanted to use events for informing transfer observers like those which 
can compute message digests of input stream. Other use case I wanted to 
support  is to display progress bars once Maven with Wagon will be 
integrated with GUI/IDE.  I hoped that this would make it even more 
attractive.
All other liblaries I know of simply do not support such feature.

> I also believe the Wagon interface is a
>balance between simplicity and functionality. It's easy to make simple
>stream based wagons and the interface in the WagonManager/Conducter will
>eventually be. Let me do the round trip with integrating Wagon into
>Maven first.
>
>  
>

OK. I  still belive that too much of useful functinality was removed. I 
also belive that we could build on top of  that what it was
the  layer  which could be even simpler to use then what we have now. I 
think that I know well what you want to achive as i feel that you are 
repeting
my early ideas. I spent couple weeks implementing and during those few weeks
wagon was oscilting between something very easy and something horribly 
complicated. I change my mind couple of times reagrding the API but I 
tried to implement something simple and extendible.

Just to make thing even more clear: My idea was that we should have 
three layers:

[ maven artifact ] (3)
[ wagon core API] (2)
[ wagon providers] (1)

and only the third layer should be normally used in application. You 
should never use Wagon API directly in application unless you are doing 
something extremly bizzare.  Other importand design  goal was to make 
wagon providers very simple to write

that's why you don't see many methods in wagons - you won't find there 
things like:

interface Wagon
{
    void get( Artifact artifact, File destination )
        throws TransferFailedException, ResourceDoesNotExistException, 
AuthorizationException;

    void put( File source, Artifact artifact )
        throws TransferFailedException, ResourceDoesNotExistException, 
AuthorizationException;

   // not yet exists
  void get( Artifact artifact,  OutputStream destination )
        throws TransferFailedException, ResourceDoesNotExistException, 
AuthorizationException;

   void put( InputStream inputStream, Artifact artifact )
        throws TransferFailedException, ResourceDoesNotExistException, 
AuthorizationException;

   void getAsString( Artifact );

}


each Wagon (if I rember correctly ) had only

interfce Wagon
{
       transfer( PutRequest )
       transfer( GetRequest )
}

and this first layer were free from the from the entities like Artifact. 
In exchange in this layer just "paths" relative to  repositor root  were 
used. 
So with any wagon provider you can  put/get any file, even such which is 
completly not obeing the layout which we are using in maven world.

So in my solution layers (1) and (3) were supposed to be super simple.  
Layer 2 was rather finished and tested in ~100%.
Layer 1 was implemented (some providers exists) but  not tetsted at all. 
And layer 3 was not yet started.

Michal


---------------------------------------------------------------------
To unsubscribe, e-mail: wagon-dev-unsubscribe@maven.apache.org
For additional commands, e-mail: wagon-dev-help@maven.apache.org


Mime
View raw message