hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Kawa <kawa.a...@gmail.com>
Subject Re: MultipleTextOutputFormat in new api of 1.2.1?
Date Fri, 12 Sep 2014 17:02:47 GMT
Afaik, dynamic partitions in the new mapreduce API are actually not
supported (please read http://grepalex.com/2013/07/16/multipleoutputs-part2/
and
http://stackoverflow.com/questions/25503034/dynamic-key-based-names-of-output-files-in-new-hadoop-api
).

If you don't want to use old mapred API, then dynamic partitioning in Hive
might an alternative.

2014-09-12 5:05 GMT+02:00 Li Li <fancyerii@gmail.com>:

> I want to output different key ranges to different directory.
> As of old api, there is a MultipleTextOutputFormat. I just need
> rewrite generateFileNameForKeyValue.
> But I can't find it in new api.
> There is a MultipleOutputs. But it's not that good because it need
> predefine keys by
> MultipleOutputs.addNamedOutput
> But before I run it, I don't know how many keys.
>

Mime
View raw message