river-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Firmstone <j...@zeus.net.au>
Subject Re: Space/outrigger suggestions (remote iterator vs. collection)
Date Wed, 19 Jan 2011 10:49:02 GMT
Dan Creswell wrote:
>> When access large disk records and pulling these into memory, you typically
>> only take a small chunk, process it and move on, so the behaviour is more
>> stream like, rather than iterator, I created an interface called
>> ResultStream (yes it supports Generics, but beware the compilation
>> boundaries that haven't been checked by the compiler), result stream is
>> terminated by a null value.
> That's generally similar to how MatchSet works under the covers....
>  <snip>
>> package org.apache.river.api.util;
>> /**
>> * This interface is similar to an Enumerator, it is designed to return
>> * results incrementally in loops, however unlike an Enumerator, there is no
>> * check first operation as implementors must return a null value after
>> * the backing data source has been exhausted. So this terminates like a
>> stream
>> * by returning a null value.
>> *
>> * @author Peter Firmstone
>> */
>> public interface ResultStream<T> {
>>   /**
>>    * Get next T, call from a loop until T is null;
>>    * @return T unless end of stream in which case null is returned.
>>    */
>>   public T get();
> So this overall, isn't far from where MatchSet goes. For those following
> along, remoteness issues are such that we can't reliably return null to
> indicate "end of stream". One may never reach the end of the stream if
> there's a failure at the back-end. One could counter that with a  "wait for
> the failure to go away" scenario but that assumes the failure does go away
> and it might not (and whilst you're figuring all that out your thread is
> blocked and can't say much to your upstream users).
> In essence, then returning null means "we successfully reached the end of
> the stream". And we need some other mechanism to say "we didn't reach the
> end of the stream for some reason". Typically the mechanism is an
> exception....

Yeah, you'd need to throw an IOException, I've pondered adding it, so it 
can be used over remote connections, that way another interface can 
implement both Remote and ResultStream.

Thanks, I think that was all the convincing I needed.

ResultStream was originally created for Parallel iteration, multiple 
threads calling get, you can't check first, you have to check after ;)



View raw message