camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claus Ibsen (JIRA)" <j...@apache.org>
Subject [jira] Updated: (CAMEL-3124) polling of feeds in FeedEntryPollingConsumer is broken.
Date Wed, 15 Sep 2010 12:13:40 GMT

     [ https://issues.apache.org/activemq/browse/CAMEL-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Claus Ibsen updated CAMEL-3124:
-------------------------------

     Original Estimate:     (was: 5 minutes)
    Remaining Estimate:     (was: 5 minutes)
            Issue Type: Improvement  (was: Bug)
              Priority: Minor  (was: Major)
           Description: 
The FeedEntryPollingConsumer class implements the poll() method for the 'splitEntries' mode
of the RssEndpoint is broken.

You can think of two ways that polling feeds could work:
1) A feed is created, then one item is processed, then the delay, then process another item.
This way the feed is kept between calls to poll().
2) A feed is created, then all the items are processed, the feed is cleared, and then the
delay.

But the way it presently works:
A feed is created, one items is processed, and the feed is cleared, then the delay, and again
the feed is created and the next item is cleared.

This is clearly wrong. Feed entries can be missed, because the index of the next item to process
is stored over polls but the list isn't. Also this creates a big network overhead when polling
very active feeds such as twitter search...

This is easy to fix. In the below code:
{code}
public void poll() throws Exception {
        Object feed = createFeed();
        populateList(feed);   

        while (hasNextEntry()) {
            Object entry = list.get(entryIndex--);

            boolean valid = true;
            if (entryFilter != null) {
                valid = entryFilter.isValidEntry(endpoint, feed, entry);
            }
            if (valid) {
                Exchange exchange = endpoint.createExchange(feed, entry);
                getProcessor().process(exchange);
                // return and wait for the next poll to continue from last time (this consumer
is stateful)
                return;
            }
        }
{code}

The return (at line 56 of org.apache.camel.component.feed.FeedEntryPollingConsumer) should
be deleted.

  was:
The FeedEntryPollingConsumer class implements the poll() method for the 'splitEntries' mode
of the RssEndpoint is broken. You can think of two ways that polling feeds could work:
1) A feed is created, then one item is processed, then the delay, then process another item.
This way the feed is kept between calls to poll().
2) A feed is created, then all the items are processed, the feed is cleared, and then the
delay.

But the way it presently works:
- A feed is created, one items is processed, and the feed is cleared, then the delay, and
again the feed is created and the next item is cleared.
This is clearly wrong. Feed entries can be missed, because the index of the next item to process
is stored over polls but the list isn't. Also this creates a big network overhead when polling
very active feeds such as twitter search...

This is easy to fix. In the below code:
public void poll() throws Exception {
        Object feed = createFeed();
        populateList(feed);   

        while (hasNextEntry()) {
            Object entry = list.get(entryIndex--);

            boolean valid = true;
            if (entryFilter != null) {
                valid = entryFilter.isValidEntry(endpoint, feed, entry);
            }
            if (valid) {
                Exchange exchange = endpoint.createExchange(feed, entry);
                getProcessor().process(exchange);
                // return and wait for the next poll to continue from last time (this consumer
is stateful)
                return;
            }
        }

The return (at line 56 of org.apache.camel.component.feed.FeedEntryPollingConsumer) should
be deleted.


This is how its designed to work form the very start, its not a bug.

> polling of feeds in FeedEntryPollingConsumer is broken.
> -------------------------------------------------------
>
>                 Key: CAMEL-3124
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-3124
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-rss
>    Affects Versions: 2.4.0
>         Environment: ubuntu 10.4
> java 6
>            Reporter: Ernst Bunders
>            Priority: Minor
>             Fix For: Future
>
>
> The FeedEntryPollingConsumer class implements the poll() method for the 'splitEntries'
mode of the RssEndpoint is broken.
> You can think of two ways that polling feeds could work:
> 1) A feed is created, then one item is processed, then the delay, then process another
item. This way the feed is kept between calls to poll().
> 2) A feed is created, then all the items are processed, the feed is cleared, and then
the delay.
> But the way it presently works:
> A feed is created, one items is processed, and the feed is cleared, then the delay, and
again the feed is created and the next item is cleared.
> This is clearly wrong. Feed entries can be missed, because the index of the next item
to process is stored over polls but the list isn't. Also this creates a big network overhead
when polling very active feeds such as twitter search...
> This is easy to fix. In the below code:
> {code}
> public void poll() throws Exception {
>         Object feed = createFeed();
>         populateList(feed);   
>         while (hasNextEntry()) {
>             Object entry = list.get(entryIndex--);
>             boolean valid = true;
>             if (entryFilter != null) {
>                 valid = entryFilter.isValidEntry(endpoint, feed, entry);
>             }
>             if (valid) {
>                 Exchange exchange = endpoint.createExchange(feed, entry);
>                 getProcessor().process(exchange);
>                 // return and wait for the next poll to continue from last time (this
consumer is stateful)
>                 return;
>             }
>         }
> {code}
> The return (at line 56 of org.apache.camel.component.feed.FeedEntryPollingConsumer) should
be deleted.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message