nemo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhiyuan Gao <gaozhiy...@ccns.ncku.edu.tw>
Subject Library version conflict on including AWS Lambda dependencies
Date Thu, 25 Jul 2019 09:42:26 GMT
Dear everyone,

As part of the Google Summer of Code program, I have been implementing a
"LambdaExecutor" to enable tasks to be executed by AWS Lambda Functions.

It includes modifications in the compiler middle end, extensions to the
user frontend, a new scheduler, and communication mechanisms between lambda
executors.

Currently, I am working on dispatching tasks to Lambda executors with no
communication mechanisms, on single-stage applications, see PR 216
<https://github.com/apache/incubator-nemo/pull/216>.

However, I failed to call AWS Lambda APIs due to a library version
conflict. The culprit seems to be org.apache.httpcomponents:httpclient:jar,
whose version is 4.3.6 for Spark, 4.1.2 for Hadoop and 4.5.5 for AWS lambda
and google-http-client package. Theoretically, we had various versions of
httpclient even before I included AWS Lambda, and it worked decently well
before. I am confused on the version issue shown in the driver stderr
(java.lang.NoSuchFieldError: INSTANCE).

I have been actively looking for solutions, and any suggestions from you
would be greatly appreciated. You can find the project dependency tree, and
the driver stderr in the attachment.

Thank you very much.
Kind regards,
Gao Zhiyuan

Mime
View raw message