edgent-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christofer Dutz <christofer.d...@c-ware.de>
Subject Re: Thread Usage in Edgent / Runtime
Date Sat, 22 Sep 2018 12:47:52 GMT
Hi Julian,

well as I told you, I got on board of the project shortly before IBM shifted their priorities.
Unfortunately I didn't have the chance to dig into all the details before that happened and
other dropped out.
I wouldn't be expecting much input from the rest. So I guess we would have to dig through
stuff like this on our own. 

But I am really willing to do this as I don't know of a similar project and I really want
to help revive Edgent again.

Chris


Am 22.09.18, 08:13 schrieb "Julian Feinauer" <j.feinauer@pragmaticminds.de>:

    Hey all,
    
    just a short appendum.
    I profiled through the code some more and found out the following:
    
    * Edgent (ETIAO Runtime) creates (and deletes) one Thread for each scheduled invocation
of a Polling Source
    * This thread is not only used to poll but also to do all subsequent processing of the
next pipeline steps (Oplets)
    * This leads to creations of hundres of Threads / min which imposes significant pressure
on GC (I looked at the SensorAnalyticsApplication where GC was triggered on my laptop each
30s and about 300MB in Thread Objets were collected (about 1000 instances)
    
    Furthermore this has the following consequences:
    * No backpressure can be used with the current setup slow sinks have to throw away messages(?)
    * No guarantee of order as sheduleAtFixedRate is used and the runnable that’s scheduled
processes the complete pipeline for the fetched element, so if, e.g., some intense operations
is only performed on some objects (or simple filtering) this could lead to races where elements
that were pulled later enter the sink earlier
    
    I will try to provide a concrete code example where these situations are demonstrated.
    
    It would be interesting for me to get an idea why this implementation was chosen or what
you think are the advantages of this approach. I think Edgent has some very nice features
and can solve some of the common problems for "edge" analytics in an elegant way but I'm not
that sure with the ETIAO Runtime for the reasons stated above.
    
    But perhaps I am also wrong with some of my conclusions as they are solely from profiling
and debugging through the code so please correct me if I am.
    
    Best
    Julian
    
    
    Am 22.09.18, 10:07 schrieb "Julian Feinauer" <j.feinauer@pragmaticminds.de>:
    
        Hi Edgenteers,
        
        
        
        after chris does not stop to convince me that Edgent is THE THING to use for IoT I
started once again a deeper look into its internals.
        
        I do not yet fully understand the core of Edgent and its runtime or how it physically
runs the users operation at the end.
        
        But what I observed is that it uses a pretty high number of threads and what’s way
worse it consumes many threads, i.e., it starts and stops threads all the time (about 1 per
second in several Demo Applications (Water / Sensor Analytics Application).
        
        And this is also not dependent on the Provider (Direct vs Development).
        
        
        
        Can anyone here give me some explanations on how the Threading Model of Edgents runtime
is and why it is done that way as for me, it seems a bit wasteful with regards to resources
and we want to be as low footprint in our applications as possible.
        
        
        
        Thank you!
        
        Julian
        
    
    

Mime
View raw message