beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Robert <daniel.rob...@acm.org>
Subject RabbitMqIO issues and open PRs
Date Thu, 31 Oct 2019 20:38:14 GMT
I'm pretty new to the Beam ecosystem, so apologies if this is not the 
right forum for this.

My team has been learning and starting to use Beam for the past few 
months and have run into myriad problems with the RabbitIO connector for 
java, aspects of which seem perhaps fundamentally broken or incorrect in 
the released implementation. I've tracked our significant issues down 
and opened tickets and PRs for them. I'm not certain what the typical 
response time is, but given the severity of the issues (as I perceive 
them), I'd like to escalate some of these PRs and try to get some fixes 
into the next Beam release.

I originally opened BEAM-8390 (https://github.com/apache/beam/pull/9782) 
as it was impossible to set the 'useCorrelationId' property (implying 
this functionality was also untested). Since then, I've found and PR'd 
the following, which are awaiting feedback/approval:

1. Watermarks not advancing

Ticket/PR: BEAM-8347 - https://github.com/apache/beam/pull/9820

Impact: under low message volumes, the watermark never advances and 
windows can never 'on time' fire.

Notes: The RabbitMq UnboundedSource uses 'oldest known time' as a 
watermark when other similar sources (and documentation on watermarking) 
state for monotonically increasing timestamps (the case with a queue) it 
should be the most recent time. I have a few open questions about 
testing and implementation details in the PR but it should work as-is.

2. Exchanges are always declared, which fail if a pre-existing exchange 
differs

Ticket/PR: BEAM-8513 - https://github.com/apache/beam/pull/9937

Impact: It is impossible to utilize an existing, durable exchange.

Notes: I'm hooking Beam up to an existing topic exchange (an 'event 
bus') that is 'durable'. RabbitMqIO current implementation will always 
attempt to declare the exchange, and does so as non-durable, which 
causes rabbit to fail the declaration. (Interestingly qpid does not fail 
in this scenario.) The PR allows the caller to disable declaring the 
exchange, similar to `withQueueDeclare` for declaring a queue.

This PR also calls out a lot of the documentation that seems misleading; 
implying that one either interacts with queues *or* exchanges when that 
is not how AMQP fundamentally operates. The implementation of the 
RabbitMqIO connector before this PR seems like it probably works with 
the default exchange and maybe a fanout exchange, but not a topic exchange.

3. Library versions

Tickets/PR: BEAM-7434, BEAM-5895, and BEAM-5894 - 
https://github.com/apache/beam/pull/9900

Impact: The rabbitmq amqp client for java released the 5.x line in 
September of 2017. Some automated tickets were open to upgrade, plus a 
manual ticket to drop the use of the deprecated QueueConsumer API.

Notes: The upgrade was relatively simple, but I implemented it using a 
pull-based API rather than push-based (Consumer) which may warrant some 
discussion. I'm used to discussing this type of thing over PRs but I'm 
happy to do whatever the community prefers.

---

Numbers 1 and 2 above are 'dealbreaker' issues for my team. They 
effectively make rabbitmq unusable as an unbounded source, forcing 
developers to fork and modify the code. Number 3 is much less 
significant and I've put it here more for 'good, clean living' than an 
urgent issue.

Aside from the open issues, given the low response rate so far, I'd be 
more than happy to take on a more active role in maintaining/reviewing 
the rabbitmq io for java. For now, however, is this list the best way to 
'bump' these open issues and move forward? Further, is the general 
approach before opening a PR to ask some preliminary questions in this 
email list?

Thank you,
-Daniel Robert


Mime
View raw message