zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vivek Goel <goelvivek2...@gmail.com>
Subject How to schedule a event in future time
Date Tue, 18 Sep 2012 11:02:43 GMT
My problem description is follows:
I have n state based database *infinite* crawlers:
Currently how it is happening:

   - We are using single machine for crawling.
   - We have three level of priority queue. High, Medium and LOW.
   - At starting all Database job are put into lower level queue.
   - Worker reads a job from queue and do operation.
   - After finishing job it reschedule it with a delay of 5 minutes.

Solution I found

For Priority Queue I can use:


Problem solution I am still searching are:

   1. How to reschedule a job in queue with future schedule time. Is there
   a way to do that in zookeeper ?
   2. Canceling a already started job. Suppose user change his database
   authentication details. I want to stop already running job for that
   database and restart with new details.
     What I thought is while starting a worker It will subscribe for that
   it's znode changes and if something happen, It will stop that job and
   reschedule it.
   3. Infinite Queue
   What I thought is that after finishing it will remove it from queue and
   readd it with future schdule time. (It implementation depend on point 1)

Is it correct way of doing this task infinite task?
Vivek Goel

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message