apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2087) Hive output module
Date Sun, 29 May 2016 01:03:12 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305715#comment-15305715
] 

ASF GitHub Bot commented on APEXMALHAR-2087:
--------------------------------------------

Github user tweise commented on a diff in the pull request:

    https://github.com/apache/incubator-apex-malhar/pull/289#discussion_r64996282
  
    --- Diff: hive/pom.xml ---
    @@ -0,0 +1,265 @@
    +<!--
    +
    +    Licensed to the Apache Software Foundation (ASF) under one
    +    or more contributor license agreements.  See the NOTICE file
    +    distributed with this work for additional information
    +    regarding copyright ownership.  The ASF licenses this file
    +    to you under the Apache License, Version 2.0 (the
    +    "License"); you may not use this file except in compliance
    +    with the License.  You may obtain a copy of the License at
    +
    +      http://www.apache.org/licenses/LICENSE-2.0
    +
    +    Unless required by applicable law or agreed to in writing,
    +    software distributed under the License is distributed on an
    +    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +    KIND, either express or implied.  See the License for the
    +    specific language governing permissions and limitations
    +    under the License.
    +
    +-->
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    +         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    +  <modelVersion>4.0.0</modelVersion>
    +
    +  <parent>
    +    <groupId>org.apache.apex</groupId>
    +    <artifactId>malhar</artifactId>
    +    <version>3.5.0-SNAPSHOT</version>
    +  </parent>
    +
    +  <artifactId>malhar-hive</artifactId>
    +  <name>Apache Apex Malhar Hive Support</name>
    +  <packaging>jar</packaging>
    +
    +  <properties>
    +    <checkstyle.console>true</checkstyle.console>
    +  </properties>
    +
    +  <build>
    +    <plugins>
    +      <!-- Publish tests jar -->
    +      <plugin>
    +        <groupId>org.apache.maven.plugins</groupId>
    +        <artifactId>maven-jar-plugin</artifactId>
    +        <version>2.4</version>
    +        <executions>
    +          <execution>
    +            <goals>
    +              <goal>test-jar</goal>
    +            </goals>
    +            <phase>package</phase>
    +          </execution>
    +        </executions>
    +      </plugin>
    +     <!-- create resource directory for xml javadoc-->
    +     <plugin>
    +         <groupId>org.apache.maven.plugins</groupId>
    +         <artifactId>maven-antrun-plugin</artifactId>
    +         <executions>
    +           <execution>
    +               <id>createJavadocDirectory</id>
    +               <phase>generate-resources</phase>
    +               <configuration>
    +                   <tasks>
    +                     <delete dir="${project.build.directory}/generated-resources/xml-javadoc"/>
    +                     <mkdir dir="${project.build.directory}/generated-resources/xml-javadoc"/>
    +                   </tasks>
    +               </configuration>
    +               <goals>
    +                   <goal>run</goal>
    +               </goals>
    +           </execution>
    +         </executions>
    +      </plugin>
    +     <!-- generate javdoc -->
    +     <plugin>
    +       <groupId>org.apache.maven.plugins</groupId>
    +       <artifactId>maven-javadoc-plugin</artifactId>
    +       <executions>
    +         <!-- generate xml javadoc -->
    +         <execution>
    +           <id>xml-doclet</id>
    +           <phase>generate-resources</phase>
    +           <goals>
    +             <goal>javadoc</goal>
    +           </goals>
    +           <configuration>
    +             <doclet>com.github.markusbernhardt.xmldoclet.XmlDoclet</doclet>
    +             <additionalparam>-d ${project.build.directory}/generated-resources/xml-javadoc
-filename ${project.artifactId}-${project.version}-javadoc.xml</additionalparam>
    +             <useStandardDocletOptions>false</useStandardDocletOptions>
    +             <docletArtifact>
    +               <groupId>com.github.markusbernhardt</groupId>
    +               <artifactId>xml-doclet</artifactId>
    +               <version>1.0.4</version>
    +             </docletArtifact>
    +           </configuration>
    +         </execution>
    +         <!-- generate default javadoc jar with custom tags -->
    +         <execution>
    +           <id>attach-sources</id>
    +           <goals>
    +             <goal>jar</goal>
    +           </goals>
    +           <configuration>
    +             <skip>true</skip>
    +             <tags>
    +               <tag>
    +                 <name>customTag1</name>
    +                 <placement>a</placement>
    +                 <head>Custom Tag One:</head>
    +               </tag>
    +               <tag>
    +                 <name>customTag2</name>
    +                 <placement>a</placement>
    +                 <head>Custom Tag two:</head>
    +               </tag>
    +               <tag>
    +                 <name>customTag3</name>
    +                 <placement>a</placement>
    +                 <head>Custom Tag three:</head>
    +               </tag>
    +             </tags>
    +           </configuration>
    +         </execution>
    +       </executions>
    +     </plugin>
    +     <!-- Transform xml javadoc to stripped down version containing only class/interface
comments and tags-->
    +     <plugin>
    +       <groupId>org.codehaus.mojo</groupId>
    +       <artifactId>xml-maven-plugin</artifactId>
    +       <version>1.0</version>
    +       <executions>
    +         <execution>
    +           <id>transform-xmljavadoc</id>
    +           <phase>generate-resources</phase>
    +           <goals>
    +             <goal>transform</goal>
    +           </goals>
    +         </execution>
    +       </executions>
    +       <configuration>
    +         <transformationSets>
    +           <transformationSet>
    +             <dir>${project.build.directory}/generated-resources/xml-javadoc</dir>
    +             <includes>
    +               <include>${project.artifactId}-${project.version}-javadoc.xml</include>
    +             </includes>
    +             <stylesheet>XmlJavadocCommentsExtractor.xsl</stylesheet>
    +             <outputDir>${project.build.directory}/generated-resources/xml-javadoc</outputDir>
    +           </transformationSet>
    +         </transformationSets>
    +       </configuration>
    +     </plugin>
    +     <!-- copy xml javadoc to class jar -->
    +     <plugin>
    +       <artifactId>maven-resources-plugin</artifactId>
    +       <version>2.6</version>
    +       <executions>
    +         <execution>
    +           <id>copy-resources</id>
    +           <phase>process-resources</phase>
    +           <goals>
    +             <goal>copy-resources</goal>
    +           </goals>
    +           <configuration>
    +             <outputDirectory>${basedir}/target/classes</outputDirectory>
    +             <resources>
    +               <resource>
    +                 <directory>${project.build.directory}/generated-resources/xml-javadoc</directory>
    +                 <includes>
    +                   <include>${project.artifactId}-${project.version}-javadoc.xml</include>
    +                 </includes>
    +                 <filtering>true</filtering>
    +               </resource>
    +             </resources>
    +           </configuration>
    +         </execution>
    +       </executions>
    +     </plugin>
    +      <plugin>
    +        <groupId>org.apache.maven.plugins</groupId>
    +        <artifactId>maven-checkstyle-plugin</artifactId>
    +        <configuration>
    +          <maxAllowedViolations>0</maxAllowedViolations>
    +          <logViolationsToConsole>${checkstyle.console}</logViolationsToConsole>
    +        </configuration>
    +      </plugin>
    +     <plugin>
    +       <artifactId>maven-surefire-plugin</artifactId>
    +       <configuration>
    +         <argLine>-Xmx2048m</argLine>
    +       </configuration>
    +     </plugin>
    +    </plugins>
    +  </build>
    +
    +  <dependencies>
    +    <dependency>
    +      <groupId>org.apache.hive</groupId>
    +      <artifactId>hive-exec</artifactId>
    +      <version>0.13.1</version>
    +      <optional>true</optional>
    --- End diff --
    
    Why optional? Dependencies should be defined so that user of module does not have to repeat
them (different from what was done in contrib).


> Hive output module
> ------------------
>
>                 Key: APEXMALHAR-2087
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2087
>             Project: Apache Apex Malhar
>          Issue Type: Improvement
>            Reporter: Yogi Devendra
>            Assignee: Yogi Devendra
>
> Currently, applications which needs to write to Hive have to include 2 operators FSPojoToHiveOperator,
HiveOperator from malhar-contrib in the application DAG. [First operator writes to HDFS; second
operator loads files from HDFS to hive].
> Proposal: to wrap these two operators into a Module/CompositeOperator. So that, end user
can directly include this module into application.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message