hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kostya Golikov <johnysilv...@gmail.com>
Subject Minimal and complete set of dependencies required to setup Miniclusters
Date Tue, 13 Jan 2015 18:41:16 GMT
Hey, guys!

>From "How to develop Hadoop tests"
<https://wiki.apache.org/hadoop/HowToDevelopUnitTests> I discovered that
there is such thing as Miniclusters, which provide easy-to-setup in-process
version of Hadoop. I was eager to use both MiniDFSCluster and
MiniYARNCluster for a functional test of my pig script (I am aware of
PigUnit and embraced it for unit tests), but had quite hard times,
resolving all transitive dependencies -- apparently below set of
dependencies is not enough to get up and running:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>${hadoop.version}</version>
    <classifier>tests</classifier>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-yarn-server-tests</artifactId>
    <version>${hadoop.version}</version>
    <classifier>tests</classifier>
    <scope>test</scope>
</dependency>

What is the canonical way to introduce miniclusters to a maven project,
given that I don't have any other hadoop components already listed in pom?

Mime
View raw message