hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-10892) Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'
Date Tue, 27 Sep 2016 22:13:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15527560#comment-15527560
] 

Mingliang Liu edited comment on HDFS-10892 at 9/27/16 10:12 PM:
----------------------------------------------------------------

The {{hadoop.hdfs.TestDFSShell#testUtf8Encoding}} was not able to pass in Jenkins. The stack
trace is like:
{quote}
Error Message

Malformed input or input contains unmappable characters: /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/1/RFrIWvn1nt/TestDFSShell/哈杜普.txt
Stacktrace

java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters:
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/1/RFrIWvn1nt/TestDFSShell/哈杜普.txt
	at sun.nio.fs.UnixPath.encode(UnixPath.java:147)
	at sun.nio.fs.UnixPath.<init>(UnixPath.java:71)
	at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281)
	at java.io.File.toPath(File.java:2234)
	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getLastAccessTime(RawLocalFileSystem.java:662)
	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.<init>(RawLocalFileSystem.java:673)
	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:643)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:871)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:635)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:435)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:360)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2093)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2061)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2026)
	at org.apache.hadoop.hdfs.TestDFSShell.testUtf8Encoding(TestDFSShell.java:3885)
{quote}

I was able to pass the test locally without any problem. Perhaps it has something to with
the {{LANG}} environment variable according to discussions on the Internet. I can confirm
that my local test machine is using following settings.
{code}
$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
{code}

I also think Yetus is setting the local correctly. [~aw] Do you have any idea about this?
Thanks.

As this test case is not tightly related to the commands "-tail" or "-stat", I remove this
test case in the latest patch. If anyone suggests a working approach, I'd like to submit separate
JIRA for tracking this test. Otherwise, we test it elsewhere (say, nightly system tests).
Just for what it's worth, the test code is:
{code}
  /**
   * Test that the file name and content can have UTF-8 chars.
   */
  @Test (timeout = 30000)
  public void testUtf8Encoding() throws Exception {
    final int blockSize = 1024;
    final Configuration conf = new HdfsConfiguration();
    conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, blockSize);

    try (MiniDFSCluster cluster =
             new MiniDFSCluster.Builder(conf).numDataNodes(3).build()) {
      cluster.waitActive();
      final DistributedFileSystem dfs = cluster.getFileSystem();
      final Path workDir= new Path("/testUtf8Encoding");
      dfs.mkdirs(workDir);

      System.setProperty("sun.jnu.encoding", "UTF-8");
      System.setProperty("file.encoding", "UTF-8");
      final String chineseStr = "哈杜普.txt";
      final File testFile = new File(TEST_ROOT_DIR, chineseStr);
      // create a local file; its content contains the Chinese file name
      createLocalFile(testFile);
      dfs.copyFromLocalFile(new Path(testFile.getPath()), workDir);
      assertTrue(dfs.exists(new Path(workDir, testFile.getName())));

      final ByteArrayOutputStream out = new ByteArrayOutputStream();
      System.setOut(new PrintStream(out));

      final String argv[] = new String[]{
          "-cat", workDir + "/" + testFile.getName()};
      final int ret = ToolRunner.run(new FsShell(conf), argv);
      assertEquals(Arrays.toString(argv) + " returned non-zero status " + ret,
          0, ret);
      assertTrue("Unexpected -cat output: " + out,
          out.toString().contains(chineseStr));
    }
  }
{code}


was (Author: liuml07):
The {{hadoop.hdfs.TestDFSShell#testUtf8Encoding}} was not able to run in Jenkins. I was able
to pass the test locally without any problem. Perhaps it has something to with the {{LANG}}
environment variable according to discussions on the Internet. I can confirm that my local
test machine is using following settings. I also think Yetus is setting the local correctly.
[~aw] Do you have any idea about this? Thanks.
{code}
$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
{code}

As this test case is not tightly related to the commands "-tail" or "-stat", I remove this
test case in the latest patch. Just for what it's worth, the test code is:
{code}

  /**
   * Test that the file name and content can have UTF-8 chars.
   */
  @Test (timeout = 30000)
  public void testUtf8Encoding() throws Exception {
    final int blockSize = 1024;
    final Configuration conf = new HdfsConfiguration();
    conf.setInt(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, blockSize);

    try (MiniDFSCluster cluster =
             new MiniDFSCluster.Builder(conf).numDataNodes(3).build()) {
      cluster.waitActive();
      final DistributedFileSystem dfs = cluster.getFileSystem();
      final Path workDir= new Path("/testUtf8Encoding");
      dfs.mkdirs(workDir);

      System.setProperty("sun.jnu.encoding", "UTF-8");
      System.setProperty("file.encoding", "UTF-8");
      final String chineseStr = "哈杜普.txt";
      final File testFile = new File(TEST_ROOT_DIR, chineseStr);
      // create a local file; its content contains the Chinese file name
      createLocalFile(testFile);
      dfs.copyFromLocalFile(new Path(testFile.getPath()), workDir);
      assertTrue(dfs.exists(new Path(workDir, testFile.getName())));

      final ByteArrayOutputStream out = new ByteArrayOutputStream();
      System.setOut(new PrintStream(out));

      final String argv[] = new String[]{
          "-cat", workDir + "/" + testFile.getName()};
      final int ret = ToolRunner.run(new FsShell(conf), argv);
      assertEquals(Arrays.toString(argv) + " returned non-zero status " + ret,
          0, ret);
      assertTrue("Unexpected -cat output: " + out,
          out.toString().contains(chineseStr));
    }
  }
{code}
If anyone suggests a working approach, I'd like to submit separate JIRA for tracking this
test. Otherwise, we test it elsewhere (say, nightly system tests).

> Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'
> -----------------------------------------------------------
>
>                 Key: HDFS-10892
>                 URL: https://issues.apache.org/jira/browse/HDFS-10892
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: fs, shell, test
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>         Attachments: HDFS-10892.000.patch, HDFS-10892.001.patch, HDFS-10892.002.patch,
HDFS-10892.003.patch, HDFS-10892.004.patch
>
>
> I did not find unit test in {{trunk}} code for following cases:
> - HDFS command {{dfs -tail}}
> - HDFS command {{dfs -stat}}
> I think it still merits to have one though the commands have served us for years.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message