hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siyao Meng (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-12345) Scale testing HDFS NameNode with real metadata and workloads (Dynamometer)
Date Thu, 28 Feb 2019 22:16:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16780957#comment-16780957
] 

Siyao Meng edited comment on HDFS-12345 at 2/28/19 10:15 PM:
-------------------------------------------------------------

We might have a little dependency issue here on JUnit.
I noticed that in the trunk (3.3.0), junit jar has been removed from the Hadoop tarball at
some point (it was under ./share/hadoop/mapreduce/lib/junit-4.11.jar in Hadoop 3.2.1 and before).
Since dynamometer-infra uses MiniDFSCluster that depends on JUnit, we won't be able to run
dynamometer-infra jar in trunk. It can still be compiled, just the JUnit jar is not included
in the distribution tarball.
I've tried to add junit jar back under somewhere in the tarball like ./share/hadoop/tools/lib/
but no luck so far. Question: is maven "copy-dependencies" the right way to do it? Need some
help here.


was (Author: smeng):
We might have a little dependency issue here on JUnit.
I noticed that in the trunk (3.3.0), junit jar has been removed from the Hadoop tarball at
some point (it was under ./share/hadoop/mapreduce/lib/junit-4.11.jar in Hadoop 3.2.1 and before).
Since dynamometer-infra uses MiniDFSCluster that depends on JUnit, we won't be able to run
dynamometer-infra jar in trunk. It can still be compiled, just not included in the distribution
tarball.
I've tried to add junit jar back under somewhere in the tarball like ./share/hadoop/tools/lib/
but no luck so far. Question: is maven "copy-dependencies" the right way to do it? Need some
help here.

> Scale testing HDFS NameNode with real metadata and workloads (Dynamometer)
> --------------------------------------------------------------------------
>
>                 Key: HDFS-12345
>                 URL: https://issues.apache.org/jira/browse/HDFS-12345
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: namenode, test
>            Reporter: Zhe Zhang
>            Assignee: Erik Krogen
>            Priority: Major
>         Attachments: HDFS-12345.000.patch, HDFS-12345.001.patch
>
>
> Dynamometer has now been open sourced on our [GitHub page|https://github.com/linkedin/dynamometer].
Read more at our [recent blog post|https://engineering.linkedin.com/blog/2018/02/dynamometer--scale-testing-hdfs-on-minimal-hardware-with-maximum].
> To encourage getting the tool into the open for others to use as quickly as possible,
we went through our standard open sourcing process of releasing on GitHub. However we are
interested in the possibility of donating this to Apache as part of Hadoop itself and would
appreciate feedback on whether or not this is something that would be supported by the community.
> Also of note, previous [discussions on the dev mail lists|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201707.mbox/%3c98FCEFFA-FAFF-4CF1-A14D-4FAAB6567F31@gmail.com%3e]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message