cayenne-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Musall, Maik" <m...@selbstdenker.ag>
Subject Re: Fetching lots of objects
Date Tue, 07 Mar 2017 07:45:58 GMT
Hi Marcel,

I know how to do the actual computation in parallel. My question is how to fetch and instantiate
the DataObjects in parallel before I can start the computations. An iterator would only slow
down the fetch because of the added roundtrips. Iterators are about reducing memory footprint,
while I am not memory-constrained here.

Maik

> Am 07.03.2017 um 08:30 schrieb Markus Reich <markus.reich@markusreich.at>:
> 
> Hi Maik,
> 
> maybe you can use the new iterator and split the iterator for parallel
> computation?
> 
> public static <T> Stream<T> asStream(Iterator<T> sourceIterator, boolean
> parallel) {
>   Iterable<T> iterable = () -> sourceIterator;
>   return StreamSupport.stream(iterable.spliterator(), parallel);
> }
> 
> found at
> http://stackoverflow.com/questions/24511052/how-to-convert-an-iterator-to-a-stream
> 
> br
> Meex
> 
> Musall, Maik <maik@selbstdenker.ag> schrieb am Mo., 6. März 2017 um
> 22:25 Uhr:
> 
>> Hi all,
>> 
>> I have a number of statistics functions which need to fetch large amounts
>> of objects. I need the actual DataObjects because that's where the business
>> logic is that I need for the computations.
>> 
>> Let's say I need to fetch 300.000 objects. Let's also assume the database
>> sits on a fast SSD array and can serve multiple connections easily. I'm
>> assuming in this case the CPU time needed for DataObject instantiation is
>> the main performance constraint. Is that correct?
>> 
>> If so, how can I speed this up? Could I partition my fetch, and fetch in
>> several threads in parallel into the same ObjectContext? Or is there an
>> easier way to make use of multiple CPU cores for this?
>> 
>> Thanks
>> Maik
>> 
>> 


Mime
View raw message