hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13952) tools dependency hooks are throwing errors
Date Fri, 04 Aug 2017 20:24:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114925#comment-16114925
] 

Allen Wittenauer commented on HADOOP-13952:
-------------------------------------------

*shrugs*  None of the solutions are great, honestly.  Most of the gains that we got by doing
the dedupe were quickly ate up by YARN's bloat anyway. :(

Code-wise:

1. using an array will be significantly faster. (Yes, getting rid of awk's and cut's is my
new thing, now that I've just thrown in the towel on arrays.)

{code}
function depline_to_jarname
{
  depline="${*}"
  IFS=':' read -ra fields <<< "${depline}"
  case "${#fields[@]}" in
    "5")
      # format: groupId:artifactId:type:version:scope
      jarname="${fields[1]}-${fields[3]}.jar"
      ;;
    "6")
      # format: groupId:artifactId:type:classifier:version:scope
      jarname="${fields[1]}-${fields[4]}-${fields[5]}.jar"
      ;;
    *)
      echo "Error: unrecognized number of fields in mvn dependency:list output"
      exit 1
      ;;
  esac
  echo "${jarname}"
}
{code}

2.  Probably better to do bash-style math:

{code}
declare -i errors=0
...
((errors=errors+1))
{code}

> tools dependency hooks are throwing errors
> ------------------------------------------
>
>                 Key: HADOOP-13952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13952
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Allen Wittenauer
>            Assignee: Sean Mackrory
>            Priority: Critical
>         Attachments: HADOOP-13952.000.patch, HADOOP-13952.preview.patch
>
>
> During build, we are throwing these errors:
> {code}
> ERROR: hadoop-aliyun has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> ERROR: hadoop-archive-logs has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-archives has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aws has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-azure has missing dependencies: jetty-util-ajax-9.3.11.v20160721.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar
> ERROR: hadoop-extras has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-gridmix has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-kafka has missing dependencies: lz4-1.2.0.jar
> ERROR: hadoop-kafka has missing dependencies: kafka-clients-0.8.2.1.jar
> ERROR: hadoop-openstack has missing dependencies: commons-httpclient-3.1.jar
> ERROR: hadoop-rumen has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: metrics-core-3.0.1.jar
> ERROR: hadoop-streaming has missing dependencies: jasper-compiler-5.5.23.jar
> {code}
> Likely a variety of reasons for the failures.  Kafka is HADOOP-12556, but others need
to be investigated.  Probably just need to look at more than just common/lib in dist-tools-hooks-maker
now that shading has gone in.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message