streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robdouglas <...@git.apache.org>
Subject [GitHub] incubator-streams pull request: Streams 115
Date Thu, 07 Aug 2014 18:27:47 GMT
Github user robdouglas commented on a diff in the pull request:

    https://github.com/apache/incubator-streams/pull/51#discussion_r15954062
  
    --- Diff: streams-contrib/streams-provider-twitter/src/main/java/org/apache/streams/twitter/provider/TwitterTimelineProvider.java
---
    @@ -272,12 +210,11 @@ void shutdownAndAwaitTermination(ExecutorService pool) {
             }
         }
     
    -
         @Override
         public void prepare(Object o) {
    --- End diff --
    
    I'm not sure I agree that using the numericIdsOnly and screenNamesOnly are the best call
in the prepare method. Is the goal here to ensure that our "id" list contains nothing but
numeric IDs, given the possibility that the list of identifiers that we start with can have
both screenNames as well as numeric IDs?
    
    If that's the case then I would suggest getting rid of the numericIdsOnly and screenNamesOnly
methods and replace them with something like this:
    
    ''''
    @Override
        public void prepare(Object o) {
    
            executor = getExecutor();
    
            try {
                lock.writeLock().lock();
                providerQueue = constructQueue();
            } finally {
                lock.writeLock().unlock();
            }
    
            Preconditions.checkNotNull(providerQueue);
            Preconditions.checkNotNull(this.klass);
            Preconditions.checkNotNull(config.getOauth().getConsumerKey());
            Preconditions.checkNotNull(config.getOauth().getConsumerSecret());
            Preconditions.checkNotNull(config.getOauth().getAccessToken());
            Preconditions.checkNotNull(config.getOauth().getAccessTokenSecret());
            Preconditions.checkNotNull(config.getInfo());
    
            consolidateToIDs();
        }
    
        /**
         * Using the "info" list that is contained in the configuration, ensure that all
         * account identifiers are converted to IDs (Longs) instead of screenNames (Strings)
         */
        private void consolidateToIDs() {
            List<String> screenNames = Lists.newArrayList();
            ids = Lists.newArrayList();
    
            for(Object account : config.getInfo()) {
                if(account instanceof String) {
                    screenNames.add((String)account);
                } else if (account instanceof Long) {
                    ids.add(Long.parseLong(Objects.toString(account, null)));
                }
            }
    
            // Twitter allows for batches up to 100 per request, but you cannot mix types
            screenNameBatches = new ArrayList<String[]>();
            while(screenNames.size() >= 100) {
                screenNameBatches.add(screenNames.subList(0, 100).toArray(new String[0]));
                screenNames = screenNames.subList(100, screenNames.size());
            }
    
            if(screenNames.size() > 0)
                screenNameBatches.add(screenNames.toArray(new String[ids.size()]));
    
            Iterator<String[]> screenNameBatchIterator = screenNameBatches.iterator();
    
            while(screenNameBatchIterator.hasNext()) {
                Collection<Long> batchIds = retrieveIds(screenNameBatchIterator.next());
                ids.addAll(batchIds);
            }
        }
    ''''


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message