commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brad Neuberg <b...@columbia.edu>
Subject [feedparser] Fixes and unit tests for Flickr and Yahoo Groups
Date Fri, 22 Oct 2004 18:21:42 GMT


This patch does the following:
         * Adds unit tests for Flickr and Yahoo Groups
         * Modifies the Feed Parser to correctly handle Flickr feeds.  It 
turns out there are five different kinds of Flickr feeds; some of these 
feeds support autodiscovery and some don't:
                 * Feeds linked to a single user - supports autodiscovery
                 * Feeds linked to an aggregation of all of a user's 
contacts - does not support autodiscovery
                 * Feeds linked to a Flickr Tag - does not support 
autodiscovery
                 * Feeds linked to a Flickr Group - supports autodiscovery
                 * Feeds linked to a Flickr Group's Comments - supports 
autodiscovery
         * None of the locators were setting FeedReference.method.  This 
attribute holds the way in which a feed was discovered, whether through 
autodiscovery, extraction from the HTML content, or through aggresive 
probing.  I fixed this bug and added unit tests for it.
         * ProbeLocator was incorrectly building up its path to test.  It 
used to assume that all of the feed links it was given were relative, which 
isn't always true; sometimes they can be absolute.  I added the method 
FeedReference.isAbsoluteResource() which ProbeLocator now checks when 
building up its path to test.

One final thing; all of the Flickr feeds are now support except the one 
that is an aggregation of all of a user's contacts.  It turns out this kind 
of feed requires us to have done HTTP authentication to the Flickr site 
before we can retrieve it.  The Feed Parser currently does not do HTTP 
basic or digest authentication, or even support setting cookies, so we can 
not access any "private" feeds.  This also affects the Yahoo Groups 
implementation; there are two kinds of Yahoo Groups, public and 
private.  We can currently find and read the public ones, but the private 
ones again require authentication.

We need to come up with a design to support HTTP authentication through the 
Feed Parser.

The patch:


? blogservice.patch
? build.patch
? compile
? lib
? locate.patch
? log4j.xml
? test.patch
? unit.patch
? src/java/org/apache/commons/feedparser/locate/blogservice
Index: src/java/org/apache/commons/feedparser/locate/DiscoveryLocator.java
===================================================================
RCS file: 
/home/cvspublic/jakarta-commons-sandbox/feedparser/src/java/org/apache/commons/feedparser/locate/DiscoveryLocator.java,v
retrieving revision 1.13
diff -r1.13 DiscoveryLocator.java
112a113
 >                 feedReference.method = FeedReference.METHOD_AUTO_DISCOVERY;
134d134
<      *
Index: src/java/org/apache/commons/feedparser/locate/FeedReference.java
===================================================================
RCS file: 
/home/cvspublic/jakarta-commons-sandbox/feedparser/src/java/org/apache/commons/feedparser/locate/FeedReference.java,v
retrieving revision 1.9
diff -r1.9 FeedReference.java
79c79,87
<
---
 >
 >     /** If true, then the resource value for this FeedReference is
 >      *  absolute (i.e. it starts with a scheme, such as "http://",
 >      *  "file://", etc.)  If false then it is not absolute and is relative.
 >      */
 >     public boolean isAbsoluteResource() {
 >         return ResourceExpander.hasScheme(resource);
 >     }
 >
Index: src/java/org/apache/commons/feedparser/locate/LinkLocator.java
===================================================================
RCS file: 
/home/cvspublic/jakarta-commons-sandbox/feedparser/src/java/org/apache/commons/feedparser/locate/LinkLocator.java,v
retrieving revision 1.9
diff -r1.9 LinkLocator.java
129a130
 >                         ref.method = FeedReference.METHOD_LINK_DISCOVERY;
145c146,147
<
---
 >                         ref.method = FeedReference.METHOD_LINK_DISCOVERY;
 >
169a172
 >                         ref.method = FeedReference.METHOD_LINK_DISCOVERY;
Index: src/java/org/apache/commons/feedparser/locate/ProbeLocator.java
===================================================================
RCS file: 
/home/cvspublic/jakarta-commons-sandbox/feedparser/src/java/org/apache/commons/feedparser/locate/ProbeLocator.java,v
retrieving revision 1.16
diff -r1.16 ProbeLocator.java
106c106,110
<                     String pathToTest = baseFeedPath + mapping[i].resource;
---
 >                     String pathToTest;
 >                     if (mapping[i].isAbsoluteResource())
 >                         pathToTest = mapping[i].resource;
 >                     else
 >                         pathToTest = baseFeedPath + mapping[i].resource;
Index: src/java/org/apache/commons/feedparser/locate/ResourceExpander.java
===================================================================
RCS file: 
/home/cvspublic/jakarta-commons-sandbox/feedparser/src/java/org/apache/commons/feedparser/locate/ResourceExpander.java,v
retrieving revision 1.6
diff -r1.6 ResourceExpander.java
27c27
<  * @version $Id: ResourceExpander.java,v 1.6 2004/10/22 00:37:08 burton Exp $
---
 >  * @version $Id: ResourceExpander.java,v 1.5 2004/09/03 19:46:47 burton 
Exp $
169a170,172
 >         if (resource == null || resource.trim().equals(""))
 >             return false;
 >
Index: src/java/org/apache/commons/feedparser/test/TestFeedFilter.java
===================================================================
RCS file: 
/home/cvspublic/jakarta-commons-sandbox/feedparser/src/java/org/apache/commons/feedparser/test/TestFeedFilter.java,v
retrieving revision 1.5
diff -r1.5 TestFeedFilter.java
44c44
<  * @version $Id: TestFeedFilter.java,v 1.5 2004/10/22 00:37:08 burton Exp $
---
 >  * @version $Id: TestFeedFilter.java,v 1.4 2004/09/14 01:32:04 burton Exp $
Index: src/java/org/apache/commons/feedparser/test/TestFeedLocator.java
===================================================================
RCS file: 
/home/cvspublic/jakarta-commons-sandbox/feedparser/src/java/org/apache/commons/feedparser/test/TestFeedLocator.java,v
retrieving revision 1.6
diff -r1.6 TestFeedLocator.java
44c44
<  * @version $Id: TestFeedLocator.java,v 1.6 2004/10/22 00:37:08 burton Exp $
---
 >  * @version $Id: TestFeedLocator.java,v 1.5 2004/08/31 21:00:32 burton Exp $
Index: src/java/org/apache/commons/feedparser/test/TestProbeLocator.java
===================================================================
RCS file: 
/home/cvspublic/jakarta-commons-sandbox/feedparser/src/java/org/apache/commons/feedparser/test/TestProbeLocator.java,v
retrieving revision 1.6
diff -r1.6 TestProbeLocator.java
31c31
<  * @version $Id: TestProbeLocator.java,v 1.6 2004/10/22 00:37:08 burton Exp $
---
 >  * @version $Id: TestProbeLocator.java,v 1.5 2004/09/23 01:51:26 burton 
Exp $
62a63,64
 >         test.testFlickr();
 >         test.testYahooGroups();
197,201c199,203
<         // We need to firm up our autodiscovery regular expressions before
<         // this will pass; what happens is it has autodiscovery, but those
<         // point to a different location than the usual ones aggresive
<         // discovery would find (which are also there).  We get back 
different
<         // results at different times
---
 >         /* FIXME: We need to firm up our autodiscovery regular 
expressions before
 >            this will pass; what happens is it has autodiscovery, but those
 >            point to a different location than the usual ones aggresive
 >            discovery would find (which are also there).  We get back 
different
 >            results at different times */
253c255
<                  HAS_RSS_FEED, 
"http://journals.aol.com/redhdka/brandnewday/rss.xml");
---
 >                  HAS_RSS_FEED, 
"http://journals.aol.com/redhdka/BrandNewDay/rss.xml");
512a515,734
 >     }
 >
 >     public void testFlickr() throws Exception {
 >         System.out.println("\nTesting Flickr...");
 >         // Flickr has several different kinds of feeds; test them all
 >         // Feeds linked to a single user - has autodiscovery
 >             FeedList list;
 >             FeedReference atomFeed, rssFeed;
 >             String resource;
 >
 >             resource = "http://www.flickr.com/photos/35468147429@N01/";
 >             System.out.println("Testing " + resource + "...");
 >             list = testFeedLocator(
 >                  resource,
 >                  new Flickr(), 2,
 >                  new String[] {
 >                                 FeedReference.ATOM_MEDIA_TYPE,
 >                                 FeedReference.RSS_MEDIA_TYPE
 >                               },
 >                  new String[] {
 > 
"http://www.flickr.com/services/feeds/photos_public.gne?id=35468147429@N01&format=atom_03",
 > 
"http://www.flickr.com/services/feeds/photos_public.gne?id=35468147429@N01&format=rss_200"


 >                               },
 >                  HAS_ATOM_FEED, 
"http://www.flickr.com/services/feeds/photos_public.gne?id=35468147429@n01&format=atom_03",
 >                  HAS_RSS_FEED, 
"http://www.flickr.com/services/feeds/photos_public.gne?id=35468147429@n01&format=rss_200");
 >             // make sure we found this through discovery and not aggresive
 >             // probing
 >             assertNotNull(list);
 >             atomFeed = list.getAdAtomFeed();
 >             rssFeed = list.getAdRSSFeed();
 >             assertNotNull(atomFeed);
 >             assertNotNull(rssFeed);
 >             assertEquals(FeedReference.METHOD_AUTO_DISCOVERY, 
atomFeed.method);
 >             assertEquals(FeedReference.METHOD_AUTO_DISCOVERY, 
rssFeed.method);
 >
 >             resource = "http://www.flickr.com/photos/28753124@N00/";
 >             System.out.println("Testing " + resource + "...");
 >             list = testFeedLocator(
 >                  resource,
 >                  new Flickr(), 2,
 >                  new String[] {
 >                                 FeedReference.ATOM_MEDIA_TYPE,
 >                                 FeedReference.RSS_MEDIA_TYPE
 >                               },
 >                  new String[] {
 > 
"http://www.flickr.com/services/feeds/photos_public.gne?id=28753124@N00&format=atom_03",
 > 
"http://www.flickr.com/services/feeds/photos_public.gne?id=28753124@N00&format=rss_200"


 >                               },
 >                  HAS_ATOM_FEED, 
"http://www.flickr.com/services/feeds/photos_public.gne?id=28753124@n00&format=atom_03",
 >                  HAS_RSS_FEED, 
"http://www.flickr.com/services/feeds/photos_public.gne?id=28753124@n00&format=rss_200");
 >             // make sure we found this through discovery and not aggresive
 >             // probing
 >             assertNotNull(list);
 >             atomFeed = list.getAdAtomFeed();
 >             rssFeed = list.getAdRSSFeed();
 >             assertNotNull(atomFeed);
 >             assertNotNull(rssFeed);
 >             assertEquals(FeedReference.METHOD_AUTO_DISCOVERY, 
atomFeed.method);
 >             assertEquals(FeedReference.METHOD_AUTO_DISCOVERY, 
rssFeed.method);
 >
 >         // Feeds linked to a single group - has autodiscovery
 >             resource = "http://www.flickr.com/groups/bluehue/pool/";
 >             System.out.println("Testing " + resource + "...");
 >             list = testFeedLocator(
 >                  resource,
 >                  new Flickr(), 2,
 >                  new String[] {
 >                                 FeedReference.ATOM_MEDIA_TYPE,
 >                                 FeedReference.RSS_MEDIA_TYPE
 >                               },
 >                  new String[] {
 > 
"http://www.flickr.com/groups/bluehue/pool/feed/?format=atom_03",
 > 
"http://www.flickr.com/groups/bluehue/pool/feed/?format=rss_200"
 >                               },
 >                  HAS_ATOM_FEED, 
"http://www.flickr.com/groups/bluehue/pool/feed/?format=atom_03",
 >                  HAS_RSS_FEED, 
"http://www.flickr.com/groups/bluehue/pool/feed/?format=rss_200");
 >
 >               resource = "http://www.flickr.com/groups/blurvision/pool";
 >               System.out.println("Testing " + resource + "...");
 >               list = testFeedLocator(
 >                  resource,
 >                  new Flickr(), 2,
 >                  new String[] {
 >                                 FeedReference.ATOM_MEDIA_TYPE,
 >                                 FeedReference.RSS_MEDIA_TYPE
 >                               },
 >                  new String[] {
 > 
"http://www.flickr.com/groups/blurvision/pool/feed/?format=atom_03",
 > 
"http://www.flickr.com/groups/blurvision/pool/feed/?format=rss_200"
 >                               },
 >                  HAS_ATOM_FEED, 
"http://www.flickr.com/groups/blurvision/pool/feed/?format=atom_03",
 >                  HAS_RSS_FEED, 
"http://www.flickr.com/groups/blurvision/pool/feed/?format=rss_200");
 >
 >             // make sure we found this through discovery and not aggresive
 >             // probing
 >             assertNotNull(list);
 >             atomFeed = list.getAdAtomFeed();
 >             rssFeed = list.getAdRSSFeed();
 >             assertNotNull(atomFeed);
 >             assertNotNull(rssFeed);
 >             assertEquals(FeedReference.METHOD_AUTO_DISCOVERY, 
atomFeed.method);
 >             assertEquals(FeedReference.METHOD_AUTO_DISCOVERY, 
rssFeed.method);
 >
 >         // Feeds linked to the discussion forum of a group - has 
autodiscovery
 >             resource = "http://www.flickr.com/groups/travellers/";
 >             System.out.println("Testing " + resource + "...");
 >             list = testFeedLocator(
 >                  resource,
 >                  new Flickr(), 2,
 >                  new String[] {
 >                                 FeedReference.ATOM_MEDIA_TYPE,
 >                                 FeedReference.RSS_MEDIA_TYPE
 >                               },
 >                  new String[] {
 > 
"http://www.flickr.com/groups_feed.gne?id=52241960364@N01&format=atom_03",
 > 
"http://www.flickr.com/groups_feed.gne?id=52241960364@N01&format=rss_200"
 >                               },
 >                  HAS_ATOM_FEED, 
"http://www.flickr.com/groups_feed.gne?id=52241960364@n01&format=atom_03",
 >                  HAS_RSS_FEED, 
"http://www.flickr.com/groups_feed.gne?id=52241960364@n01&format=rss_200");
 >             // make sure we found this through discovery and not aggresive
 >             // probing
 >             assertNotNull(list);
 >             atomFeed = list.getAdAtomFeed();
 >             rssFeed = list.getAdRSSFeed();
 >             assertNotNull(atomFeed);
 >             assertNotNull(rssFeed);
 >             assertEquals(FeedReference.METHOD_AUTO_DISCOVERY, 
atomFeed.method);
 >             assertEquals(FeedReference.METHOD_AUTO_DISCOVERY, 
rssFeed.method);
 >
 >             resource = "http://www.flickr.com/groups/blurvision/";
 >             System.out.println("Testing " + resource + "...");
 >             list = testFeedLocator(
 >                  resource,
 >                  new Flickr(), 2,
 >                  new String[] {
 >                                 FeedReference.ATOM_MEDIA_TYPE,
 >                                 FeedReference.RSS_MEDIA_TYPE
 >                               },
 >                  new String[] {
 > 
"http://www.flickr.com/groups_feed.gne?id=52241701415@N01&format=atom_03",
 > 
"http://www.flickr.com/groups_feed.gne?id=52241701415@N01&format=rss_200"
 >                               },
 >                  HAS_ATOM_FEED, 
"http://www.flickr.com/groups_feed.gne?id=52241701415@n01&format=atom_03",
 >                  HAS_RSS_FEED, 
"http://www.flickr.com/groups_feed.gne?id=52241701415@n01&format=rss_200");
 >             // make sure we found this through discovery and not aggresive
 >             // probing
 >             assertNotNull(list);
 >             atomFeed = list.getAdAtomFeed();
 >             rssFeed = list.getAdRSSFeed();
 >             assertNotNull(atomFeed);
 >             assertNotNull(rssFeed);
 >             assertEquals(FeedReference.METHOD_AUTO_DISCOVERY, 
atomFeed.method);
 >             assertEquals(FeedReference.METHOD_AUTO_DISCOVERY, 
rssFeed.method);
 >
 >         // Feeds that aggregate all of your contacts photos together -
 >             // does not have autodiscovery
 >             // no way to even do aggresive probing here, because
 >             // the URI is always at http://www.flickr.com/photos/friends/
 >             // and it uses a cookie to see who is signed in
 >             // FIXME: When we get authentication working we can support
 >             // this kind of feed
 >
 >         // Feeds linked to a single tag - does not have autodiscovery
 >             testSite(
 >                  "http://www.flickr.com/photos/tags/2004/",
 >                  new Flickr(), 1,
 >                  new String[] {
 >                                 FeedReference.ATOM_MEDIA_TYPE
 >                               },
 >                  new String[] {
 > 
"http://flickr.com/services/feeds/photos_public.gne?tags=2004&format=atom_03"
 >                               },
 >                  HAS_ATOM_FEED, 
"http://flickr.com/services/feeds/photos_public.gne?tags=2004&format=atom_03",
 >                  NO_RSS_FEED, null);
 >
 >             testSite(
 >                  "http://www.flickr.com/photos/tags/blue",
 >                  new Flickr(), 1,
 >                  new String[] {
 >                                 FeedReference.ATOM_MEDIA_TYPE
 >                               },
 >                  new String[] {
 > 
"http://flickr.com/services/feeds/photos_public.gne?tags=blue&format=atom_03"
 >                               },
 >                  HAS_ATOM_FEED, 
"http://flickr.com/services/feeds/photos_public.gne?tags=blue&format=atom_03",
 >                  NO_RSS_FEED, null);
 >     }
 >
 >     public void testYahooGroups() throws Exception {
 >         System.out.println("\nTesting Yahoo Groups...");
 >         // there are two kinds of groups in Yahoo, public and private groups
 >         // Public Groups
 >         testSite("http://groups.yahoo.com/group/talkinaboutarchitecture/",
 >                  new YahooGroups(), 1,
 >                  new String[] {
 >                                 FeedReference.RSS_MEDIA_TYPE
 >                               },
 >                  new String[] {
 > 
"http://rss.groups.yahoo.com/group/talkinaboutarchitecture/rss"
 >                               },
 >                  NO_ATOM_FEED, null,
 >                  HAS_RSS_FEED, 
"http://rss.groups.yahoo.com/group/talkinaboutarchitecture/rss");
 >
 >         testSite("http://groups.yahoo.com/group/BPWFL_PUBLIC_POLICY/",
 >                  new YahooGroups(), 1,
 >                  new String[] {
 >                                 FeedReference.RSS_MEDIA_TYPE
 >                               },
 >                  new String[] {
 > 
"http://rss.groups.yahoo.com/group/BPWFL_PUBLIC_POLICY/rss"
 >                               },
 >                  NO_ATOM_FEED, null,
 >                  HAS_RSS_FEED, 
"http://rss.groups.yahoo.com/group/BPWFL_PUBLIC_POLICY/rss");
 >
 >         // Private Groups
 >         // FIXME: These will currently fail, because we don't support 
private
 >         // groups
 >         // FIXME: Enumerate some private groups here; since we don't have
 >         // a way to programmatically sign in and do HTTP authentication 
through
 >         // the feedparser we can't create tests here


---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org


Mime
View raw message