apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vrozov <...@git.apache.org>
Subject [GitHub] incubator-apex-core pull request: APEX-254 & APEX-269
Date Tue, 24 Nov 2015 23:28:18 GMT
Github user vrozov commented on the pull request:

    https://github.com/apache/incubator-apex-core/pull/173#issuecomment-159438298
  
    I don't agree that it is not possible to use 512K queue size. Depending on tuple size,
memory availability and processing requirements queue size of 512K or larger may be used by
applications. Whether or not default queue size should be changed to larger value is discussable.
Based on my experience and what I've heard, there is no good default value for the queue size
and applications need to choose between slightly larger memory consumption or a performance
boost. Note that 4x performance improvement of SpscArrayQueueReservoir over CircularBufferReservoir
is for CircularBufferReservoir configured also for 512K queue size. With the default queue
size, CircularBufferReservoir scales only to 3 million events per second.
    
    It is not clear why it is necessary to have exact size() and where do we pay price for
loosing accuracy of the size(). Performance benchmark does not show any degradation for the
CircularBufferReservoir that does not use size() and uses BlockingQueue interface.
    
    While benchmark is done under ideal conditions, it measures scalability of the Apex framework.
SpscArrayQueueReservoir provides the infrastructure that is more scalable compared to CircularBufferReservoir
and ArrayBlockingQueueReservoir. IMO, this should be sufficient to justify pulling the changes
in.
    
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message