hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suraj Nayak M (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-10704) Corrupt Files are not moved to bad_files in LocalFileSystem
Date Mon, 16 Jun 2014 14:44:01 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Suraj Nayak M updated HADOOP-10704:
-----------------------------------

    Description: 
A file is created using LocalFileSystem. When the file is corrupted/changed and tried to read,
it throws ChecksumException as the CRC does not match. But this process does not move the
affected file to "bad_files" directory.

Code : 
public void testReportChecksumFailureOnCorruption() throws IOException {
		String sampleData = "My Test Data. This will be corrupted soon";
		FileSystem fs = LocalFileSystem.get(new Configuration());
		Path rootPath = new Path(TEST_ROOT_DIR);
		FSDataOutputStream fsOutStream = fs.create(new Path(rootPath
				+ "/testFile.txt"));
		ByteArrayInputStream inStream = new ByteArrayInputStream(
				sampleData.getBytes());
		IOUtils.copyBytes(inStream, fsOutStream, 4096, false);
		fsOutStream.close();
		inStream.close();
		// Corrupt the file
		File corruptFile = new File(TEST_ROOT_DIR + "/testFile.txt");
		FileOutputStream fOutCorruptingStream = new FileOutputStream(
				corruptFile, true);
		fOutCorruptingStream.write(". This is bad data!".getBytes());
		fOutCorruptingStream.close();
		// Read the corrupted file using LocalFS
		FileSystem readLFS = LocalFileSystem.get(new Configuration());
		FSDataInputStream fsInStream = readLFS.open(new Path(TEST_ROOT_DIR
				+ "/testFile.txt"));
		try {
			byte[] b = new byte[10];
			fsInStream.read(b, 0, 10);
		} catch (Exception e) {
			if (corruptFile.exists()) {
				// Should not exit before moving the corrupt file to bad_files
				assertFalse(true);
			} else {
				// Should exit after moving the corrupt file to bad_files
				assertTrue(true);
			}
		}
		fsInStream.close();
	}

  was:A file is created using LocalFileSystem. When the file is corrupted/changed and tried
to read, it throws ChecksumException as the CRC does not match. But this process does not
move the affected file to "bad_files" directory.


> Corrupt Files are not moved to bad_files in LocalFileSystem
> -----------------------------------------------------------
>
>                 Key: HADOOP-10704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10704
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Suraj Nayak M
>            Priority: Minor
>
> A file is created using LocalFileSystem. When the file is corrupted/changed and tried
to read, it throws ChecksumException as the CRC does not match. But this process does not
move the affected file to "bad_files" directory.
> Code : 
> public void testReportChecksumFailureOnCorruption() throws IOException {
> 		String sampleData = "My Test Data. This will be corrupted soon";
> 		FileSystem fs = LocalFileSystem.get(new Configuration());
> 		Path rootPath = new Path(TEST_ROOT_DIR);
> 		FSDataOutputStream fsOutStream = fs.create(new Path(rootPath
> 				+ "/testFile.txt"));
> 		ByteArrayInputStream inStream = new ByteArrayInputStream(
> 				sampleData.getBytes());
> 		IOUtils.copyBytes(inStream, fsOutStream, 4096, false);
> 		fsOutStream.close();
> 		inStream.close();
> 		// Corrupt the file
> 		File corruptFile = new File(TEST_ROOT_DIR + "/testFile.txt");
> 		FileOutputStream fOutCorruptingStream = new FileOutputStream(
> 				corruptFile, true);
> 		fOutCorruptingStream.write(". This is bad data!".getBytes());
> 		fOutCorruptingStream.close();
> 		// Read the corrupted file using LocalFS
> 		FileSystem readLFS = LocalFileSystem.get(new Configuration());
> 		FSDataInputStream fsInStream = readLFS.open(new Path(TEST_ROOT_DIR
> 				+ "/testFile.txt"));
> 		try {
> 			byte[] b = new byte[10];
> 			fsInStream.read(b, 0, 10);
> 		} catch (Exception e) {
> 			if (corruptFile.exists()) {
> 				// Should not exit before moving the corrupt file to bad_files
> 				assertFalse(true);
> 			} else {
> 				// Should exit after moving the corrupt file to bad_files
> 				assertTrue(true);
> 			}
> 		}
> 		fsInStream.close();
> 	}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message