flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wang, Yongkun | Yongkun | BDD" <yongkun.w...@mail.rakuten.com>
Subject sleep() in script doesn't work when called by exec Source
Date Mon, 19 Aug 2013 02:29:48 GMT

I am testing with apache-flume-1.4.0-bin.
I made a naive python script for exec source to do throttling by calling sleep() function.
But the sleep() doesn't work when called by exec source.
Any ideas about this or do you have some simply solution for throttling instead of a custom

Flume config:

agent.sources = src1
agent.sources.src1.type = exec
agent.sources.src1.command = read-file-throttle.py



import time

with open("apache.log") as infile:
    for line in infile:
        line = line.strip()
        print line
        count += 1
        if count % 50000 == 0:
            now_time = time.time()
            diff = now_time - pre_time
            if diff < 10:
                #print "sleeping %s seconds ..." % (diff)
                pre_time = now_time

Thank you very much.

Best Regards,
Yongkun Wang

View raw message