Mailing-List: contact dev-help@streams.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@streams.incubator.apache.org
From: robdouglas <git@git.apache.org>
To: dev@streams.incubator.apache.org
Reply-To: dev@streams.incubator.apache.org
References: <git-pr-51-incubator-streams@git.apache.org>
In-Reply-To: <git-pr-51-incubator-streams@git.apache.org>
Subject: [GitHub] incubator-streams pull request: Streams 115
Content-Type: text/plain
Message-Id: <20140807182747.66DCB94A212@tyr.zones.apache.org>
Date: Thu,  7 Aug 2014 18:27:47 +0000 (UTC)

Github user robdouglas commented on a diff in the pull request:

    https://github.com/apache/incubator-streams/pull/51#discussion_r15954062
  
    --- Diff: streams-contrib/streams-provider-twitter/src/main/java/org/apache/streams/twitter/provider/TwitterTimelineProvider.java ---
    @@ -272,12 +210,11 @@ void shutdownAndAwaitTermination(ExecutorService pool) {
             }
         }
     
    -
         @Override
         public void prepare(Object o) {
    --- End diff --
    
    I'm not sure I agree that using the numericIdsOnly and screenNamesOnly are the best call in the prepare method. Is the goal here to ensure that our "id" list contains nothing but numeric IDs, given the possibility that the list of identifiers that we start with can have both screenNames as well as numeric IDs?
    
    If that's the case then I would suggest getting rid of the numericIdsOnly and screenNamesOnly methods and replace them with something like this:
    
    ''''
    @Override
        public void prepare(Object o) {
    
            executor = getExecutor();
    
            try {
                lock.writeLock().lock();
                providerQueue = constructQueue();
            } finally {
                lock.writeLock().unlock();
            }
    
            Preconditions.checkNotNull(providerQueue);
            Preconditions.checkNotNull(this.klass);
            Preconditions.checkNotNull(config.getOauth().getConsumerKey());
            Preconditions.checkNotNull(config.getOauth().getConsumerSecret());
            Preconditions.checkNotNull(config.getOauth().getAccessToken());
            Preconditions.checkNotNull(config.getOauth().getAccessTokenSecret());
            Preconditions.checkNotNull(config.getInfo());
    
            consolidateToIDs();
        }
    
        /**
         * Using the "info" list that is contained in the configuration, ensure that all
         * account identifiers are converted to IDs (Longs) instead of screenNames (Strings)
         */
        private void consolidateToIDs() {
            List<String> screenNames = Lists.newArrayList();
            ids = Lists.newArrayList();
    
            for(Object account : config.getInfo()) {
                if(account instanceof String) {
                    screenNames.add((String)account);
                } else if (account instanceof Long) {
                    ids.add(Long.parseLong(Objects.toString(account, null)));
                }
            }
    
            // Twitter allows for batches up to 100 per request, but you cannot mix types
            screenNameBatches = new ArrayList<String[]>();
            while(screenNames.size() >= 100) {
                screenNameBatches.add(screenNames.subList(0, 100).toArray(new String[0]));
                screenNames = screenNames.subList(100, screenNames.size());
            }
    
            if(screenNames.size() > 0)
                screenNameBatches.add(screenNames.toArray(new String[ids.size()]));
    
            Iterator<String[]> screenNameBatchIterator = screenNameBatches.iterator();
    
            while(screenNameBatchIterator.hasNext()) {
                Collection<Long> batchIds = retrieveIds(screenNameBatchIterator.next());
                ids.addAll(batchIds);
            }
        }
    ''''


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---