hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 梁景明 <futur...@gmail.com>
Subject Customized OutputFormat doesnt work
Date Thu, 03 Sep 2009 03:25:27 GMT
Hi, I have a problem with Customized OutputFormat, my OutputFormat splits
kinds of my seeds and outputs three paths of results,and it works when
running in one node or eclipse plugin but it doesnt work right in 5 nodes
as  a jar running, the file will be smaller than the correct sizes. here is
my code ,any help will be appreciated .
    public static class SepOutputFormat implements OutputFormat {
        public void checkOutputSpecs(FileSystem arg0, JobConf arg1)
                throws IOException {
        public RecordWriter getRecordWriter(final FileSystem fs,
                final JobConf job, final String name, Progressable progress)
                throws IOException {
            return new RecordWriter() {
                String date = DateUtil.getString("yyyy-MM-dd-HH", new
                Path newPath = new Path("data/seednew/" + date);
                Path lostPath = new Path("data/seedlost/" + date);
                Path basePath = new Path("data/seedbase/" + date);
                SequenceFile.Writer newwriter = new SequenceFile.Writer(fs,
                        job, newPath, Text.class, Seed.class);
                SequenceFile.Writer lostwriter = new SequenceFile.Writer(fs,
                        job, lostPath, Text.class, Seed.class);
                SequenceFile.Writer basewriter = new SequenceFile.Writer(fs,
                        job, basePath, Text.class, Seed.class);

                public void close(Reporter reporter) throws IOException {
                public void write(Object key, Object value) throws
IOException {
                    String word = key.toString();
                    if (word.startsWith("new_")) {
                        String skey = word.replace("new_", "");
                        newwriter.append(new Text(skey), (Seed) value);
                    } else if (word.startsWith("lost_")) {
                        String skey = word.replace("lost_", "");
                        lostwriter.append(new Text(skey), (Seed) value);
                    } else {
                        basewriter.append(new Text(word), (Seed) value);

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message