hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "VITALIY SAVCHENKO (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13725) Open MapFile for append
Date Thu, 20 Oct 2016 03:58:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590692#comment-15590692
] 

VITALIY SAVCHENKO commented on HADOOP-13725:
--------------------------------------------

{code}
1.
        MapFile.Writer w = null;
        try {
            w = new MapFile.Writer(
                new Configuration(),
                new Path("hdfs://192.168.56.101:9000/20161020/data"),
                MapFile.Writer.keyClass(IntWritable.class),
                MapFile.Writer.valueClass(IntWritable.class)
            );
            w.append(new IntWritable(0), new IntWritable(100));
            w.append(new IntWritable(10), new IntWritable(200));
            w.append(new IntWritable(5), new IntWritable(400)); //java.io.IOException: key
out of order: 5 after 10
        } catch (Exception ex) {
            ex.printStackTrace();
        } finally {
            w.close();
        }
        MapFile.Reader reader = new MapFile.Reader(
            new Path("hdfs://192.168.56.101:9000/20161020/data"),
            new Configuration()
        );
        System.out.println(reader.get(new IntWritable(0), new IntWritable())); //print 100
        System.out.println(reader.get(new IntWritable(10), new IntWritable())); // print 200
MapFile corrent
        reader.close();
2. Open MapFile for apped
        MapFile.Writer w = null;
        try {
            w = new MapFile.Writer(
                new Configuration(),
                new Path("hdfs://192.168.56.101:9000/20161020/data"),
                MapFile.Writer.keyClass(IntWritable.class),
                MapFile.Writer.valueClass(IntWritable.class)
            );
            w.append(new IntWritable(0), new IntWritable(100));
            w.append(new IntWritable(10), new IntWritable(200));
            w.close();

            w = new MapFile.Writer(
                new Configuration(),
                new Path("hdfs://192.168.56.101:9000/20161020/data"),
                SequenceFile.Writer.appendIfExists(true), // append to exist MapFile
                SequenceFile.Writer.replication((short)2),
                MapFile.Writer.keyClass(IntWritable.class),
                MapFile.Writer.valueClass(IntWritable.class)
            );
            w.append(new IntWritable(20), new IntWritable(300));
            w.append(new IntWritable(30), new IntWritable(400));
        } catch (Exception ex) {
            ex.printStackTrace();
        } finally {
            w.close();
        }
        MapFile.Reader reader = new MapFile.Reader(
            new Path("hdfs://192.168.56.101:9000/20161020/data"),
            new Configuration()
        );
        System.out.println(reader.get(new IntWritable(10), new IntWritable())); //print 200
        System.out.println(reader.get(new IntWritable(20), new IntWritable())); //print 300
MapFile correct
        reader.close();

3. Append to exist MapFile, but incorrect range
        MapFile.Writer w = null;
        try {
            w = new MapFile.Writer(
                new Configuration(),
                new Path("hdfs://192.168.56.101:9000/20161020/data"),
                MapFile.Writer.keyClass(IntWritable.class),
                MapFile.Writer.valueClass(IntWritable.class)
            );
            w.append(new IntWritable(10), new IntWritable(100));
            w.append(new IntWritable(20), new IntWritable(200));
            w.close();

            w = new MapFile.Writer(
                new Configuration(),
                new Path("hdfs://192.168.56.101:9000/20161020/data"),
                SequenceFile.Writer.appendIfExists(true), //append to MapFile
                MapFile.Writer.keyClass(IntWritable.class),
                MapFile.Writer.valueClass(IntWritable.class)
            );
            w.append(new IntWritable(5), new IntWritable(300)); //No exception here
            w.append(new IntWritable(10), new IntWritable(400));
        } catch (Exception ex) {
            ex.printStackTrace();
        } finally {
            w.close();
        }

        MapFile.Reader reader = new MapFile.Reader(
            new Path("hdfs://192.168.56.101:9000/20161020/data"),
            new Configuration()
        );
        System.out.println(reader.get(new IntWritable(5), new IntWritable())); // java.io.IOException:
key out of order: 5 after 10 - MapFile corrupted
        System.out.println(reader.get(new IntWritable(10), new IntWritable()));
        System.out.println(reader.get(new IntWritable(20), new IntWritable()));
        reader.close();
{code}

Reason: When open MapFile with option SequenceFile.Writer.appendIfExists(true) - writer not
read last key from exist MapFile.


> Open MapFile for append
> -----------------------
>
>                 Key: HADOOP-13725
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13725
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: VITALIY SAVCHENKO
>
> I think it possible to open MapFile for appending.
> SequenceFile support it (Option SequenceFile.Writer.appendIfExists(true) HADOOP-7139)
> Now it almost working. But if use SequenceFile.Writer.appendIfExists(true) MapFile.Writer
- it not read last key and does not check new keys. That's why MapFile can be corrupted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message