Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 79DC011FB5 for ; Fri, 12 Sep 2014 17:03:17 +0000 (UTC) Received: (qmail 20838 invoked by uid 500); 12 Sep 2014 17:03:12 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 20721 invoked by uid 500); 12 Sep 2014 17:03:12 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 20711 invoked by uid 99); 12 Sep 2014 17:03:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Sep 2014 17:03:12 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of kawa.adam@gmail.com designates 209.85.223.170 as permitted sender) Received: from [209.85.223.170] (HELO mail-ie0-f170.google.com) (209.85.223.170) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Sep 2014 17:03:08 +0000 Received: by mail-ie0-f170.google.com with SMTP id tp5so1298469ieb.29 for ; Fri, 12 Sep 2014 10:02:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=AONFIH8AfimtmePA1k1vPIGPg5F9wdWIdGhkYIXG42E=; b=O8tndDiAVBy2+DNHeEG6o19v6EgFPAkHP99oaC0ca0rK2Hvu4wywCVmVc/pHKdz9fl TnOxOJJhrgVpe5as/sPj/fp2Wd+7AQR0dHGdmDS2nG96p41rNC5YAOzu7HtD8mdzRyOI Sl6XxNWDSjFVDTWfGn3HT2c3/QYaM++J8bNUtybRmQLNn1lnx4SSyk8sNqJZx2Lv38kd WcFbe/bafVq0U7wRdaOy8/m1n2wPo64GtUiNaa8Ce7S3LDtJQJn8Arez1n3flM3QcTtK jWd/cKGNckMi1fuq+ckJJmBuRAoexVegdGDdh8ITrthp49lCMdGuAxGFUXSdraDqrOi9 JyNQ== MIME-Version: 1.0 X-Received: by 10.43.111.6 with SMTP id em6mr11410037icc.21.1410541367540; Fri, 12 Sep 2014 10:02:47 -0700 (PDT) Received: by 10.42.24.130 with HTTP; Fri, 12 Sep 2014 10:02:47 -0700 (PDT) In-Reply-To: References: Date: Fri, 12 Sep 2014 19:02:47 +0200 Message-ID: Subject: Re: MultipleTextOutputFormat in new api of 1.2.1? From: Adam Kawa To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=bcaec51718652744b60502e1410b X-Virus-Checked: Checked by ClamAV on apache.org --bcaec51718652744b60502e1410b Content-Type: text/plain; charset=UTF-8 Afaik, dynamic partitions in the new mapreduce API are actually not supported (please read http://grepalex.com/2013/07/16/multipleoutputs-part2/ and http://stackoverflow.com/questions/25503034/dynamic-key-based-names-of-output-files-in-new-hadoop-api ). If you don't want to use old mapred API, then dynamic partitioning in Hive might an alternative. 2014-09-12 5:05 GMT+02:00 Li Li : > I want to output different key ranges to different directory. > As of old api, there is a MultipleTextOutputFormat. I just need > rewrite generateFileNameForKeyValue. > But I can't find it in new api. > There is a MultipleOutputs. But it's not that good because it need > predefine keys by > MultipleOutputs.addNamedOutput > But before I run it, I don't know how many keys. > --bcaec51718652744b60502e1410b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Afaik, dynamic partitions in the new mapreduce API ar= e actually not supported (please read=C2=A0http://grepalex.com/2013/07/16/multipleou= tputs-part2/ and=C2=A0http://sta= ckoverflow.com/questions/25503034/dynamic-key-based-names-of-output-files-i= n-new-hadoop-api).

If you don't want t= o use old mapred API, then dynamic partitioning in Hive might an alternativ= e.

201= 4-09-12 5:05 GMT+02:00 Li Li <fancyerii@gmail.com>:
I want to output different key ranges to differ= ent directory.
As of old api, there is a MultipleTextOutputFormat. I just need
rewrite generateFileNameForKeyValue.
But I can't find it in new api.
There is a MultipleOutputs. But it's not that good because it need
predefine keys by
MultipleOutputs.addNamedOutput
But before I run it, I don't know how many keys.

--bcaec51718652744b60502e1410b--