asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Carey <dtab...@gmail.com>
Subject Re: Feed Capabilities
Date Wed, 17 May 2017 16:23:43 GMT
NICE!!!


On 5/17/17 8:05 AM, Steven Jacobs wrote:
> The two being compared had everything the same (including logging) other
> than one was doing insert (with key check) and the other was upsert
> (without). I had corrected the logging before doing the comparison.
> Even more cool, I realized that the 143 million was actually the speed at
> which my python script was sending data over the socket, meaning that
> Asterix wasn't hitting any bottlenecks. So I tried using two feeds to the
> same dataset with two different ports, and I was able to double this speed
> again without hitting a bottleneck. I still haven't done any experiments
> for more than an hour, but this is really promising.
> Steven
>
> On Tue, May 16, 2017 at 11:14 PM Mike Carey <dtabass@gmail.com> wrote:
>
>> Wow!  @Steven, did you change the logging level simultaneously?
>>
>> On May 16, 2017 8:03 PM, "abdullah alamoudi" <bamousaa@gmail.com> wrote:
>>
>>> That is really cool.
>>>
>>>> On May 16, 2017, at 1:29 PM, Steven Jacobs <sjaco002@ucr.edu> wrote:
>>>>
>>>> Hi all,
>>>> I Just wanted to leave a comment continuing the feed speed capabilities
>>>> discussions that we've had recently. Using Abdullah's change to allow
>> an
>>>> upsert feed with no secondary index to skip the duplicate key check, I
>>> saw
>>>> a huge increase in speed capabilities. In the first hour of the
>>> experiment,
>>>> I jumped from inserting 17 million records without the change to 143
>>>> million records with the change (that's around 40 thousand records per
>>>> second). I haven't done more than hour-long experiments, but I
>> recommend
>>>> others with speed problems to try using this change as well (it's in
>>> master
>>>> already).
>>>> Steven
>>>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message