Return-Path: X-Original-To: apmail-crunch-dev-archive@www.apache.org Delivered-To: apmail-crunch-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B0B2F18105 for ; Thu, 16 Jul 2015 20:08:04 +0000 (UTC) Received: (qmail 66142 invoked by uid 500); 16 Jul 2015 20:08:04 -0000 Delivered-To: apmail-crunch-dev-archive@crunch.apache.org Received: (qmail 66106 invoked by uid 500); 16 Jul 2015 20:08:04 -0000 Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@crunch.apache.org Delivered-To: mailing list dev@crunch.apache.org Received: (qmail 66091 invoked by uid 500); 16 Jul 2015 20:08:04 -0000 Delivered-To: apmail-incubator-crunch-dev@incubator.apache.org Received: (qmail 66088 invoked by uid 99); 16 Jul 2015 20:08:04 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2015 20:08:04 +0000 Date: Thu, 16 Jul 2015 20:08:04 +0000 (UTC) From: "Adric Eckstein (JIRA)" To: crunch-dev@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (CRUNCH-543) AvroPathPerKeyTarget copy nested subdirectories MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Adric Eckstein created CRUNCH-543: ------------------------------------- Summary: AvroPathPerKeyTarget copy nested subdirectories Key: CRUNCH-543 URL: https://issues.apache.org/jira/browse/CRUNCH-543 Project: Crunch Issue Type: Improvement Components: IO Reporter: Adric Eckstein When using AvroPathPerKeyTarget to write out a subpath in the output directory using a String key, the key might indicate multiple subfolders: Pair kv = new Pair("foo/bar", "value"); PTable kvs = pipeline.create(Arrays.asList(kv),Avros.tableOf(Avros.strings(), Avros.strings())); PTables.asPTable(kvs).write(new AvroPathPerKeyTarget("output")); This throws the error: java.io.IOException: java.lang.IllegalArgumentException: Reducer output name 'bar' cannot be parsed at org.apache.crunch.impl.mr.exec.CrunchJobHooks$CompletionHook.handleMultiPaths(CrunchJobHooks.java:92) ... In AvroPathPerKeyTarget the handleOutputs method would need to recursively copy subfolders (currently only checks first level in output directory) to enable keys that define multiple sub folders. -- This message was sent by Atlassian JIRA (v6.3.4#6332)