apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Farkas <...@datatorrent.com>
Subject Re: Running Local Mode Test Application Connected to HDFS Cluster
Date Sun, 03 Apr 2016 19:33:49 GMT
I suspect since the hdfs jar is marked provided it is not in the test
classpath. So the hdfs files stem implementation is not found. If you mark
it with the test scope it will be included. There may be other issues with
the path as chandni
On Apr 3, 2016 11:58 AM, "Chandni Singh" <chandni@datatorrent.com> wrote:

> Since this is happening with the FileOutputOperator can you please let me
> know what is the value of property 'filePath' in the configuration?
>
> Thanks,
> Chandni
>
> On Sun, Apr 3, 2016 at 11:26 AM, Chandni Singh <chandni@datatorrent.com>
> wrote:
>
> > Hi Alex,
> >
> > What is the input directory path that you have specified in the
> > configuration?
> > On Apr 3, 2016 7:57 AM, "McCullough, Alex" <
> Alex.McCullough@capitalone.com>
> > wrote:
> >
> >> I have the dependencies defined in the POM but get the same error when
> >> running in the IDE.
> >>
> >> <dependency>
> >>     <groupId>org.apache.hadoop</groupId>
> >>     <artifactId>hadoop-hdfs</artifactId>
> >>     <version>2.7.2</version>
> >>     <scope>provided</scope>
> >> </dependency>
> >>
> >> <dependency>
> >>     <groupId>org.apache.hadoop</groupId>
> >>     <artifactId>hadoop-client</artifactId>
> >>     <version>2.7.2</version>
> >>     <scope>provided</scope>
> >> </dependency>
> >>
> >> <dependency>
> >>     <groupId>org.apache.hadoop</groupId>
> >>     <artifactId>hadoop-common</artifactId>
> >>     <version>2.7.2</version>
> >>     <scope>provided</scope>
> >> </dependency>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On 4/3/16, 10:33 AM, "Thomas Weise" <thomas@datatorrent.com> wrote:
> >>
> >> >Alex,
> >> >
> >> >Local mode in the IDE is fully embedded. It does not use any external
> >> >Hadoop install. Please try launching the app through the CLI (in local
> >> >mode) or you need to include the HDFS dependencies in your application
> >> pom
> >> >(with provided scope).
> >> >
> >> >Thanks
> >> >
> >> >--
> >> >sent from mobile
> >> >On Apr 3, 2016 7:29 AM, "McCullough, Alex" <
> >> Alex.McCullough@capitalone.com>
> >> >wrote:
> >> >
> >> >> Hello All,
> >> >>
> >> >> I am trying to get my local test application to run but when I try
I
> >> get
> >> >> an error. This test application is running on my local machine and
> >> >> attempting to connect to a remote HDFS cluster.
> >> >>
> >> >> I have made sure I have the same version of hadoop that is on the
> >> cluster
> >> >> installed locally and with the same config. I can run local HDFS
> >> commands
> >> >> that connect just fine to cluster to read the same files  my
> >> application
> >> >> should be reading/writing.
> >> >>
> >> >> Thanks,
> >> >> Alex
> >> >>
> >> >>
> >> >>
> >> >> Error:
> >> >>
> >> >> java.lang.RuntimeException: java.io.IOException: No FileSystem for
> >> scheme:
> >> >> hdfs
> >> >> at
> >> >>
> >>
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:334)
> >> >> at
> >> >>
> >>
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:71)
> >> >> at
> >> >>
> >>
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.setup(HdfsFileOutputOperator.java:22)
> >> >> at com.datatorrent.stram.engine.Node.setup(Node.java:182)
> >> >> at
> >> >>
> >>
> com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1290)
> >> >> at
> >> >>
> >>
> com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:129)
> >> >> at
> >> >>
> >>
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1363)
> >> >> Caused by: java.io.IOException: No FileSystem for scheme: hdfs
> >> >> at
> >> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
> >> >> at
> >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
> >> >> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
> >> >> at
> >> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
> >> >> at
> >> org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2455)
> >> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:414)
> >> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:422)
> >> >> at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:404)
> >> >> at
> >> >>
> >>
> com.capitalone.vault8.citadel.operators.impl.HdfsFileOutputOperator.getFSInstance(HdfsFileOutputOperator.java:90)
> >> >> at
> >> >>
> >>
> com.datatorrent.lib.io.fs.AbstractFileOutputOperator.setup(AbstractFileOutputOperator.java:332)
> >> >> ... 6 more
> >> >>
> >> >>
> >> >> Application Test Class:
> >> >>
> >> >>
> >> >> @Category(IntegrationTest.class)
> >> >> public class ApplicationTest {
> >> >>
> >> >>   @Test
> >> >>   public void testApplication() throws IOException, Exception {
> >> >>     try {
> >> >>       LocalMode lma = LocalMode.newInstance();
> >> >>       Configuration conf = new Configuration(false);
> >> >>
> >> >>
> >>
> conf.addResource(this.getClass().getResourceAsStream("/META-INF/properties.xml"));
> >> >>       lma.prepareDAG(new Application(), conf);
> >> >>       LocalMode.Controller lc = lma.getController();
> >> >>       lc.run(200000); // runs for 10 seconds and quits
> >> >>       lc.shutdown();
> >> >>     } catch (ConstraintViolationException e) {
> >> >>       Assert.fail("constraint violations: " +
> >> e.getConstraintViolations());
> >> >>     }
> >> >>   }
> >> >> }
> >> >> ________________________________________________________
> >> >>
> >> >> The information contained in this e-mail is confidential and/or
> >> >> proprietary to Capital One and/or its affiliates and may only be used
> >> >> solely in performance of work or services for Capital One. The
> >> information
> >> >> transmitted herewith is intended only for use by the individual or
> >> entity
> >> >> to which it is addressed. If the reader of this message is not the
> >> intended
> >> >> recipient, you are hereby notified that any review, retransmission,
> >> >> dissemination, distribution, copying or other use of, or taking of
> any
> >> >> action in reliance upon this information is strictly prohibited. If
> you
> >> >> have received this communication in error, please contact the sender
> >> and
> >> >> delete the material from your computer.
> >> >>
> >> ________________________________________________________
> >>
> >> The information contained in this e-mail is confidential and/or
> >> proprietary to Capital One and/or its affiliates and may only be used
> >> solely in performance of work or services for Capital One. The
> information
> >> transmitted herewith is intended only for use by the individual or
> entity
> >> to which it is addressed. If the reader of this message is not the
> intended
> >> recipient, you are hereby notified that any review, retransmission,
> >> dissemination, distribution, copying or other use of, or taking of any
> >> action in reliance upon this information is strictly prohibited. If you
> >> have received this communication in error, please contact the sender and
> >> delete the material from your computer.
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message