ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kevin <kevin.c...@leonardo.com>
Subject DataStreamer performance with removing data
Date Tue, 10 Nov 2015 21:10:34 GMT
I am removing about 100 thousands records, using a data streamer as follows:


>         try (
>                 QueryCursor
> &lt;
> List&lt;?
> &gt;
>> cursor = cache.query(new SqlFieldsQuery("SELECT _key FROM MyClass WHERE x
>> = ?").setArgs(y));
>                 IgniteDataStreamer
> &lt;
> MyKey, MyClass
> &gt;
>  streamer = ignite.dataStreamer(cache.getName());
>         ) {
>             streamer.allowOverwrite(true);
>             for (List<?> fieldsResult : cursor) {
>                 MyKey key = (MyKey) fieldsResult.get(0);
>                 streamer.removeData(key);
>             }
>         }

I noticed this seems to be taking a while (~ 1 min) so I tried a different
approach just to see how it does.


>         try (
>                 QueryCursor
> &lt;
> List&lt;?
> &gt;
>> cursor = cache.query(new SqlFieldsQuery("SELECT _key FROM MyClass WHERE x
>> = ?").setArgs(y));
>         ) {
>             Set
> <MyKey>
>  keys = new HashSet<>();
> 
>             for (List<?> fieldsResult : cursor) {
>                 keys.add((MyKey) fieldsResult.get(0));
>             }
> 
>             cache.clearAll(keys);
>         }

Using clearAll instead took about 1s. This was surprising since I thought
data streamer is supposed to be faster? Could someone explain the big
difference in speed here? Am I missing something?
I tried it with both 1 and 2 nodes and the results were about the same.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/DataStreamer-performance-with-removing-data-tp1917.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message