beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wesley Tanaka (JIRA)" <>
Subject [jira] [Commented] (BEAM-1859) sorter extension depends on hadoop but does not declare as such in repository artifact
Date Mon, 03 Apr 2017 05:05:41 GMT


Wesley Tanaka commented on BEAM-1859:

Adding org.apache.hadoop:hadoop-core:0.20.2 as a dependency explicitly does resolve the issue,
thanks, I'll just do that; I didn't know that it was a best practice to assume it was already

In case it helps to know it, my use case is one of learning the Beam API, not of trying to
actually accomplish something with it:

* I am trying to learn the Beam API
* So I am trying to create different toy composite PTransforms
* and I'd like to speed up the code/test/debug cycle relative to uploading code into a cluster
* so, despite this being nonsensical w.r.t. the actual use of Beam, I am trying to hack together
some code to get DirectRunner to read lines from stdin and write lines to stdout and run the
same code against different inputs to see how it behaves.

> sorter extension depends on hadoop but does not declare as such in repository artifact
> --------------------------------------------------------------------------------------
>                 Key: BEAM-1859
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>    Affects Versions: 0.6.0
>            Reporter: Wesley Tanaka
>            Assignee: Davor Bonaci
>             Fix For: Not applicable
> When SortValues is used via {{org.apache.beam:beam-sdks-java-extensions-sorter:0.6.0}},
this exception is raised:
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
> 	at org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(
> 	at org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
> 	at
> 	at java.lang.ClassLoader.loadClass(
> 	at sun.misc.Launcher$AppClassLoader.loadClass(
> 	at java.lang.ClassLoader.loadClass(
> 	at org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(
> 	at org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(
> 	at org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn$auxiliary$uK25yOmK.invokeProcessElement(Unknown
> 	at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(
> 	at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(
> 	at org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElement(
> 	at org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElementInReadyWindows(
> 	at
> 	at
> 	at
> 	at
> 	at java.util.concurrent.Executors$
> 	at
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(
> 	at java.util.concurrent.ThreadPoolExecutor$
> 	at
> {noformat}
> I think the issue is that beam-sdks-java-extensions-sorter should declare that it depends
on that hadoop library but does not?

This message was sent by Atlassian JIRA

View raw message