Return-Path: X-Original-To: apmail-crunch-dev-archive@www.apache.org Delivered-To: apmail-crunch-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 161A5C24F for ; Tue, 18 Jun 2013 10:58:10 +0000 (UTC) Received: (qmail 73635 invoked by uid 500); 18 Jun 2013 10:31:30 -0000 Delivered-To: apmail-crunch-dev-archive@crunch.apache.org Received: (qmail 73555 invoked by uid 500); 18 Jun 2013 10:31:23 -0000 Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@crunch.apache.org Delivered-To: mailing list dev@crunch.apache.org Received: (qmail 73538 invoked by uid 500); 18 Jun 2013 10:31:21 -0000 Delivered-To: apmail-incubator-crunch-dev@incubator.apache.org Received: (qmail 73523 invoked by uid 99); 18 Jun 2013 10:31:20 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jun 2013 10:31:20 +0000 Date: Tue, 18 Jun 2013 10:31:20 +0000 (UTC) From: "Tom White (JIRA)" To: crunch-dev@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CRUNCH-219) Support multiple paths in Avro source MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CRUNCH-219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated CRUNCH-219: ----------------------------- Attachment: CRUNCH-219.patch Here's a new patch that adds multi-path support to all file-based inputs. I haven't changed MaterializableIterable, but then I'm not sure it's needed, since only Sources can have multiple paths. Targets and SourceTargets are still single paths, and for each of MapsideJoinStrategy, BloomFilterJoinStrategy, and Sort the PCollection being materialized is not an input collection, so it's a SourceTarget (I think), and hence a single path. (I'm not sure it's even possible to change MaterializableIterable to have a getPaths() method since FilterKeysWithBloomFilterFn calls PType.getPath() with a single path to get a SourceTarget.) Does this sound right to you Josh, or am I missing something? > Support multiple paths in Avro source > ------------------------------------- > > Key: CRUNCH-219 > URL: https://issues.apache.org/jira/browse/CRUNCH-219 > Project: Crunch > Issue Type: Improvement > Components: Core > Reporter: Tom White > Assignee: Josh Wills > Attachments: CRUNCH-219.patch, CRUNCH-219.patch > > > It would be useful to be able to specify multiple paths (which may be files, or directories, or a combination of both) to read from in a source. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira