hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "robert Cook (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-6513) SequenceFile.Sorter design issue and class-check bug
Date Thu, 28 Jan 2010 21:13:34 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-6513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

robert Cook updated HADOOP-6513:
--------------------------------

    Attachment: TestSequenceFile.java

Here's an update to TestSequenceFile with a new Unit test as you requested.
the test suceeds; it should fail.

PS Re 6512
By your argument, there should be no synchronized prefixes in the SequenceFile user I/O methods,
but there are.
I respectfully disagree with your conclusion.

> SequenceFile.Sorter  design issue and class-check bug
> -----------------------------------------------------
>
>                 Key: HADOOP-6513
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6513
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1
>         Environment: hadoop 20.1, java 1.6.0_17,fedora
>            Reporter: robert Cook
>         Attachments: TestSequenceFile.java
>
>
> SequenceFile.Writer takes key/value classes as creation arguments and checks for validity
on every append.
> Reader does not take class arguments on creation because they are derived from the input
file.
> Sorter takes key/value classes as creation arguments??  no point.  should be derived
from input.
> In any case, SortPass does not compare Sorter key/value classes with input file classes.
> No error is given for the following:
> 	  private static void writeTest4(FileSystem fs, int count, int seed, Path file, 
> 			  SequenceFile.CompressionType compressionType, CompressionCodec codec, Configuration
conf)
> 	    throws IOException {
> 	    fs.delete(file, true);
> 	    LOG.info("creating " + count + " records with " + compressionType +
> 	             " compression");
> 	    SequenceFile.Writer writer = 
> 	      SequenceFile.createWriter(fs, conf, file, 
> 	    		  StringWritable.class, FloatWritable.class, compressionType, codec);
> 	    FloatWritable x=new FloatWritable();
> 	    StringWritable y=new StringWritable();
> 	    for (int i = count-1; i >= 0; i--) {
> 	      x.set(i);  y.set(""+i);
> 	      writer.append(y, x);
> 	    }
> 	    writer.close();
> 	  }
> 	  private static void sortTest(FileSystem fs, int count, int megabytes, 
> 			  int factor, boolean fast, Path file, Configuration conf)
> 	  throws IOException {
> 		  fs.delete(new Path(file+".sorted"), true);
> 		  SequenceFile.Sorter sorter = newSorter(fs, fast, megabytes, factor, conf);
> 		  LOG.debug("sorting " + count + " records");
> 		  sorter.sort(file, file.suffix(".sorted"));
> 		  LOG.info("done sorting " + count + " debug");
> 	  }
> 	  
> 	  private static SequenceFile.Sorter newSorter(FileSystem fs, 
> 			  boolean fast,
> 			  int megabytes, int factor, Configuration conf) {
> 		  SequenceFile.Sorter sorter = 
> 			  fast
> 			  ? new SequenceFile.Sorter(fs, new IntWritable.Comparator(),
> 					  FloatWritable.class, IntWritable.class, conf)
> 		  : new SequenceFile.Sorter(fs, FloatWritable.class, IntWritable.class, conf);
> 			  sorter.setMemory(megabytes * 1024*1024);
> 			  sorter.setFactor(factor);
> 			  return sorter;
> 	  }
> ---------------------Note String/Float  does not match Float/Int
> Macintosh-2:datanode bobcook$ od -c file          
> 0000000    S   E   Q 006 016   S   t   r   i   n   g   W   r   i   t   a
> 0000020    b   l   e  \r   F   l   o   a   t   W   r   i   t   a   b   l
> 0000040    e  \0  \0  \0  \0  \0  \0 203   `   n   E   J   z 272   d 352
> 0000060    w 177 373  \n 364   M 276  \0  \0  \0  \n  \0  \0  \0 006  \0
> 0000100   \0  \0 001  \0   4   @ 200  \0  \0  \0  \0  \0  \n  \0  \0  \0
> 0000120  006  \0  \0  \0 001  \0   3   @   @  \0  \0  \0  \0  \0  \n  \0
> 0000140   \0  \0 006  \0  \0  \0 001  \0   2   @  \0  \0  \0  \0  \0  \0
> 0000160   \n  \0  \0  \0 006  \0  \0  \0 001  \0   1   ? 200  \0  \0  \0
> 0000200   \0  \0  \n  \0  \0  \0 006  \0  \0  \0 001  \0   0  \0  \0  \0
> *
> 0000220
> Macintosh-2:datanode bobcook$ od -c file.sorted
> 0000000    S   E   Q 006  \r   F   l   o   a   t   W   r   i   t   a   b
> 0000020    l   e  \v   I   n   t   W   r   i   t   a   b   l   e  \0  \0
> 0000040   \0  \0  \0  \0   6 364 343  \r 256   h   U 222 365   T   7   l
> 0000060  357   i   ~   }  \0  \0  \0  \n  \0  \0  \0 006  \0  \0  \0 001
> 0000100   \0   4   @ 200  \0  \0  \0  \0  \0  \n  \0  \0  \0 006  \0  \0
> 0000120   \0 001  \0   3   @   @  \0  \0  \0  \0  \0  \n  \0  \0  \0 006
> 0000140   \0  \0  \0 001  \0   2   @  \0  \0  \0  \0  \0  \0  \n  \0  \0
> 0000160   \0 006  \0  \0  \0 001  \0   1   ? 200  \0  \0  \0  \0  \0  \n
> 0000200   \0  \0  \0 006  \0  \0  \0 001  \0   0  \0  \0  \0  \0        
> 0000216
> NOTE OUTPUT FILE IS TOTALLY TOASTED, but no error was generated!
> PS: Your evaluation of my previous bug reports was enlightening.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message