beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Sisk <s...@google.com>
Subject Re: Cannot find output with Apex runner
Date Fri, 16 Jun 2017 21:32:16 GMT
We've seen a couple reports involving the "Unable to find registrar for
hdfs"

The other potential cause is misconfiguration of HDFS/beam can't find the
HDFS config.

I filed https://issues.apache.org/jira/browse/BEAM-2457 - we don't believe
this is a bug in beam, but a number of users seem to be running into the
issues so there might be an undiagnosed issue or a common misconfiguration
problem.

Claire - if you figure out the root cause, it'd be helpful if you let us
know what solved the issue so we can improve the error message you saw.
(and if you can't figure it out, hopefully folks on this list will help you
figure it out)

S

On Fri, Jun 16, 2017 at 1:58 PM Kenneth Knowles <klk@google.com> wrote:

> Hi Claire,
>
> The 'hdfs' filesystem is registered when you include the artifact
> "org.apache.beam:beam-sdks-java-io-hadoop-file-system". Do you have this in
> your dependencies?
>
> Kenn
>
> On Fri, Jun 16, 2017 at 11:45 AM, Claire Yuan <claireyuan@yahoo-inc.com>
> wrote:
>
>> Hi all,
>>   I was following the instruction here Apache Apex Runner
>> <https://beam.apache.org/documentation/runners/apex/> to submit the work
>> into the cluster. The building seems to be successful. However, the thing
>> is that I could not find where the output is. I set my param in my maven
>> command with:
>> --output=/user/claire/output/
>> and I checked with hadoop dfs -ls /home/claire/output/ but seems no such
>> directory created.
>> I also checked my local directory with
>> --output=/home/claire/output/, and still no output there
>> Finally I set the output directory manually with:
>> --output=hdfs:///user/claireyuan/output
>> it gave exception as: Failed to execute goal
>> org.codehaus.mojo:exec-maven-plugin:1.5.0:java (default-cli) on project
>> beam-examples-java: An exception occured while executing the Java class.
>> null: InvocationTargetException: Unable to find registrar for hdfs -> [Help
>> 1]
>>
>> Apache Apex Runner
>> Apache Beam is an open source, unified model and set of language-specific
>> SDKs for defining and executing data p...
>> <https://beam.apache.org/documentation/runners/apex/>
>> I am wondering where I should check or modify my output directory to be?
>>
>> Claire
>>
>
>

Mime
View raw message