curator-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vadim <va...@ant.ee>
Subject Re: Curator queue usage problems
Date Wed, 16 Nov 2016 21:18:07 GMT
Dear Jordan, 

Technical details are briefly: 

 	* Application is build with Spring Boot and Zookeeper 3.4.9

	* Curator is used to elect leader (LeaderLatch) and to provide
distributed queue that is lock-aware. Here is queue initialization code:

    private void distributedQueueInit() throws Exception {
        FileConsumer consumer = new FileConsumer(receiver);
        FileReferenceSerializer serializer = new
FileReferenceSerializer();
        QueueBuilder<FileReference> builder =
QueueBuilder.builder(client, consumer, serializer, queuePath);
        builder.lockPath(lockPath);
        fileRefs = builder.buildQueue();
        fileRefs.start();    
    } 
	* FileConsumer implements QueueConsumer and its processMessage() is
very short:

   @Override
    public void consumeMessage(FileReference reference) throws Exception
{
        log.info("Processing file: " + reference.getReference() + "|" +
reference.getRefType());
        receiver.processFile(reference);
    } 	* The problem is at receiver.processFile(reference) function.  It
has HTTP call to external resource that retrieves file by reference and
save it on disk. When problem arise (it happened once for 3 days) I can
see at my logs that "Processing file:" is fired. Next log message is
after file is saved to disk. When error happened - it did not appear. At
the same time I can see lock and message in the queue that are not
processed. I thing there are only 2 reasons for consumer to die :

 	* HTTP call takes too much time and consumer dies
 	* HTTP connection dies without exception and consumer does not receive
any error and hangs on

   If you need more details I can provide information about HTTP call,
but I think for such kind of problem it is not relevant. I am curious
about what to do next because there are may be more than 2 reasons for
such a behavior. What do you think? 

Regards, 

Vadim

On 2016-11-16 17:31, Jordan Zimmerman wrote:

> Can you send a code snippet or test that shows the issue? 
> 
> -Jordan 
> 
>> On Nov 16, 2016, at 4:35 AM, Vadim <vadim@ant.ee> wrote: 
>> 
>> Hello all, 
>> 
>> My name is Vadim and I am new curator user. I am very happy with curator, but think
that may be using it in a wrong way a bit. Particularly  I am using Distributed Queue receipt
and code runs well until Consumer silently dies. Messages for queue are small and does not
exceed 50 bytes. I use queue to distribute tasks between Workers. 
>> 
>> My QueueConsumer method consumeMessate() calls part of the code that may fail silently
(without exception). Since message delivery at my scenario is "durable" I can see locks that
are never freed. Curator does not "cure" such a stale consumer as well. Am I doing something
wrong? 
>> 
>> I have an idea to call code inside "consumeMessage"  at separate blocking thread
that has a timeout. Thus when my code fails silently -- consumer will get timeout exception.
But this is not an elegant solution I think. What do you think? 
>> 
>> Thank you in advance, 
>> 
>> Vadim.
 
Mime
View raw message