gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Madhawa Kasun Gunasekara <madhaw...@gmail.com>
Subject Re: POC for hazelcast jet integration
Date Tue, 23 Apr 2019 08:45:11 GMT
Hi Lahiru,

Good initiative.
It seems like we need to use hadoop-hdfs and hadoop common with version
2.8.3. Try adding these dependencies, and exclude the old dependency from
the gora, or otherwise, we can do a version upgrade in gora for Hadoop. at
the moment we use Hadoop version 2.5.2 in gora but I prefer to upgrade the
Hadoop versions in the gora to 2.8.3.

Recently hazelcast developers released 3.0 version, We can try that also.

Thanks,
Madhawa


On Sat, Apr 6, 2019 at 9:53 PM Lahiru Jayasekera <mlpjayasekera@gmail.com>
wrote:

> Hi all,
> I have come up with a poc for hazelcast jet execution engine support for
> gora.
>
> https://github.com/LahiruJayasekara/gora/blob/poc-hazelcast-jet/gora-tutorial/src/main/java/org/apache/gora/tutorial/log/HazelcastJetPOC.java
>
> Here I have wrote a custom source for jet. What this does is read the
> AccessLog table created via LogManager example, and feed the PageView
> objects to jet.
>
> I have also looked into using hdfs connectors of jet. But I encountered a
> problem with conflicting hadoop versions. InputFormat class used in gora is
> from pacjage 'org.apache.hadoop.mapreduce', but in jet it is '
> org.apache.hadoop.mapred.InputFormat'.
>
> Anyway the proposed poc works as expected. Please add your suggestions
> here.
>
> Thanks and regards
>
> --
> Lahiru Jayasekara
> Batch 15
> Faculty of Information Technology
> University of Moratuwa
> 0716492170
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message