manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piergiorgio Lucidi <piergior...@apache.org>
Subject Re: GSOC 2018: MongoDB Output Connector
Date Sat, 28 Apr 2018 10:37:11 GMT
Hi Irindu,

I think that for this new development we should try to use the latest
stable version of MongoDB with its own related drivers and libraries.
We only have to check if some dependencies can be in conflict with other
libraries currently deployed in ManifoldCF.

Let's try to use the latest version and if it works we will also create an
issue for updating the existing MongoDB/GridFS Repository Connector.

Finally could you please share all your improvements and questions inside
our dev list or in the related issue on JIRA?
In this way you could receive update not only from me but also from any
other member in the community.

Your effort is related to the community so don't be shy to write directly
in the dev list :)

I would like to create a dedicated branch for your development that starts
from the latest release of ManifoldCF, so this means that after the 2.10
release I'll create the branch for you.

Cheers,
PJ




2018-04-28 4:12 GMT+02:00 Irindu Nugawela <nbiindeera@gmail.com>:

> Hi Piergiorgio,
> I have noticed that you are using  mongo-java-driver
>
> <dependency>
>     <groupId>org.mongodb</groupId>
>     <artifactId>mongo-java-driver</artifactId>
>     <version>2.11.3</version>
> </dependency>
>
> for Gridfs connector but  the latest stable is
>
> <dependency>
>     <groupId>org.mongodb</groupId>
>     <artifactId>mongo-java-driver</artifactId>
>     <version>3.6.3</version>
> </dependency>
>
> there are some API changes between the two which one do you want me to
> use. At the moment I cannot use the version 3.6.3 inside the manifoldCF
> framework even though I add it as a dependency for my project I encounter
> some problems with certain API calls which I suspect because version 2.11.3
> jar is already there. Please guide me with next steps, shall I stick to
> version 2.11.3?
>
>
>
> On 26 April 2018 at 16:27, Piergiorgio Lucidi <piergiorgio@apache.org>
> wrote:
>
>> Just shared on Twitter:
>> https://twitter.com/ApacheManifold/status/989457998715277312
>>
>> ^__^
>>
>> 2018-04-26 11:39 GMT+02:00 Irindu Nugawela <nbiindeera@gmail.com>:
>>
>>> Hi Piergiorgio,
>>> Thank you very much.  yes, you may publish about my involvement in our
>>> project and please let me know if I can help you in any way possible.  Yes,
>>> the twitter profile you have listed belongs to me.
>>>
>>> On 26 April 2018 at 12:23, Piergiorgio Lucidi <piergiorgio@apache.org>
>>> wrote:
>>>
>>>> Hi Irindu,
>>>>
>>>> no problem, take your time.
>>>>
>>>> In the meanwhile I'm wondering if you allow me to publish a post about
>>>> your involvement in the project on our Twitter profile:
>>>> https://twitter.com/apachemanifold
>>>>
>>>> I noticed that you have a Twitter profile, could you please to confirm
>>>> that your profile is the following?
>>>> https://twitter.com/IrinduPera
>>>>
>>>> Please let me know and if you give me your permission I'll write a post.
>>>> Thank you and take care.
>>>>
>>>> Cheers,
>>>> PJ
>>>>
>>>> 2018-04-26 3:33 GMT+02:00 Irindu Nugawela <nbiindeera@gmail.com>:
>>>>
>>>>> Hi Piergiorgio,
>>>>>
>>>>> Thank you very much for giving me this valuable opportunity, I am
>>>>> going to do my best with it. Secondly, I am very sorry that I was not
very
>>>>> active in the last few days that I have been suffering from a viral fever.
>>>>> I am recovering now and will work extra hours in the upcoming weeks to
get
>>>>> back on track. Please help me with the next steps :-)
>>>>>
>>>>> On 16 April 2018 at 13:52, Piergiorgio Lucidi <piergiorgio@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Irindu,
>>>>>>
>>>>>> very nice and great job!
>>>>>>
>>>>>> ^__^
>>>>>>
>>>>>> Cheers,
>>>>>> PJ
>>>>>>
>>>>>>
>>>>>> 2018-04-16 8:00 GMT+02:00 Irindu Nugawela <nbiindeera@gmail.com>:
>>>>>>
>>>>>>> Hi Piergiorgio ,
>>>>>>> I have come up with the implementation for edit configuration
and
>>>>>>> view configuration with the crawler-ui. It's a small step in
the long
>>>>>>> journey.​
>>>>>>>  Apache ManifoldCF_ View Output Connection Statu...
>>>>>>> <https://drive.google.com/file/d/1X3q32ml40yzJ1moTjoXew2K1g5wNk0Gs/view?usp=drive_web>
>>>>>>> ​
>>>>>>>
>>>>>>>
>>>>>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
Virus-free.
>>>>>>> www.avast.com
>>>>>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
>>>>>>> <#m_8005759486091203682_m_-3875022077604017546_m_2871376415022697920_m_-1171787217413983024_m_1069976683875003658_m_645939314292940779_m_2231955365225803757_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>>>>>
>>>>>>> On 15 March 2018 at 00:45, Irindu Nugawela <nbiindeera@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>> I was able to successfully build ManifoldCF with IntelliJ
Idea as
>>>>>>>> well.
>>>>>>>>
>>>>>>>> I have gone through MongoDB - Java
>>>>>>>> <https://www.tutorialspoint.com/mongodb/mongodb_java.htm>
and
>>>>>>>> identified the MongoDB parameters as follows
>>>>>>>>
>>>>>>>>
>>>>>>>>    1. MongoCredential by a useraname and a password
>>>>>>>>    2. MongoDatabase
>>>>>>>>    3. collection
>>>>>>>>
>>>>>>>> I am planning to have a single username and a password for
a Manifold-MongoDB connection
>>>>>>>>
>>>>>>>> I have identified the operations that we are going to support
as
>>>>>>>>
>>>>>>>>    1. Inserting documents
>>>>>>>>    2. Querying documents
>>>>>>>>    3. Updating documents
>>>>>>>>    4. Deleting documents
>>>>>>>>    5. Indexing documents based on a particular field
>>>>>>>>
>>>>>>>> I have also studied the document format we are going to store
>>>>>>>> available here
>>>>>>>> <https://github.com/apache/manifoldcf/blob/trunk/framework/agents/src/main/java/org/apache/manifoldcf/agents/interfaces/RepositoryDocument.java>
which
>>>>>>>> MongoDB accepts in BSON format,
>>>>>>>>
>>>>>>>> I have identified the following fields
>>>>>>>>
>>>>>>>>    1.  binaryFieldData
>>>>>>>>    2.  binaryLength
>>>>>>>>    3.  fileName
>>>>>>>>    4.  contentMimeType
>>>>>>>>    5.   createdDate
>>>>>>>>    6.   modifiedDate
>>>>>>>>    7.   indexingDate
>>>>>>>>    8.   originalSize
>>>>>>>>
>>>>>>>> of a Repository Document that I am going to convert to BSON
format
>>>>>>>> to write to MongoDB
>>>>>>>>
>>>>>>>> Please tell me am I on the correct path, and correct me if
I am
>>>>>>>> wrong
>>>>>>>>
>>>>>>>> On 27 February 2018 at 15:42, Piergiorgio Lucidi <
>>>>>>>> piergiorgio@apache.org> wrote:
>>>>>>>>
>>>>>>>>> Hi Irindu,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2018-02-25 13:53 GMT+01:00 Irindu Nugawela <nbiindeera@gmail.com>:
>>>>>>>>>
>>>>>>>>> > Hi All,
>>>>>>>>> > I am currently to working on the $subject.  I have
some issues
>>>>>>>>> that I need
>>>>>>>>> > to clarify.
>>>>>>>>> > First of all, why did we choose Mongo DB as an output
target? My
>>>>>>>>> first
>>>>>>>>> > thought of the project was that it was about writing
a
>>>>>>>>> repository connector
>>>>>>>>> > for MongoDB because MongoDB is a database programme.
( I am
>>>>>>>>> aware of its
>>>>>>>>> > NoSQL
>>>>>>>>> > architecture and its indexing capabilities ). But
it is not a
>>>>>>>>> search engine
>>>>>>>>> > per se.
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>> We want to extend the scenario of ManifoldCF and we would
like to
>>>>>>>>> add
>>>>>>>>> content migration capability.
>>>>>>>>> Imagine to have a tool that helps you not only for the
indexing
>>>>>>>>> process but
>>>>>>>>> also in migrating contents.
>>>>>>>>>
>>>>>>>>> In this way you could migrate contents from a content
repository
>>>>>>>>> to another
>>>>>>>>> one, for example you could migrate contents from Alfresco
to
>>>>>>>>> MongoDB.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> >
>>>>>>>>> > Then what are the expected operations other than
document
>>>>>>>>> addition and
>>>>>>>>> > deletion? (what activities should be supported?)
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>> The excepted operation are the same but related to migrate
>>>>>>>>> contents and not
>>>>>>>>> related only to manage the indexing process.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> >
>>>>>>>>> > What capabilities of MongoDB you have already identified
that
>>>>>>>>> you expect to
>>>>>>>>> > be useful for us. (what capabilities of MongoDB
you have
>>>>>>>>> identified as
>>>>>>>>> > useful That you've decided to write an Output connector
for it.)
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>> Currently we only have a repository connector for GridFS
that
>>>>>>>>> allow us to
>>>>>>>>> read binaries from MongoDB.
>>>>>>>>> We also want to store contents against a MongoDB instance,
in this
>>>>>>>>> way we
>>>>>>>>> can migrate contents on MongoDB from any repository with
metadata
>>>>>>>>> and
>>>>>>>>> binaries.
>>>>>>>>>
>>>>>>>>> I think it could be very cool :)
>>>>>>>>>
>>>>>>>>> In the enterprise context is very critical to migrate
contents /
>>>>>>>>> documents
>>>>>>>>> from different kind of repositories and typically it
is the first
>>>>>>>>> step for
>>>>>>>>> a digital transformation project where a company is trying
to
>>>>>>>>> orchestrate
>>>>>>>>> different silos in a smart way.
>>>>>>>>>
>>>>>>>>> Please let me know if you need more informations.
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> PJ
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> >
>>>>>>>>> > I have been through the ManifoldCF Architecture
with DaddyWri
>>>>>>>>> > <https://github.com/DaddyWri>/manifoldcfinaction
Chapter1.
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > Thanks and Regards,
>>>>>>>>> > Irindu Nugawela,
>>>>>>>>> > Computer Engineering <http://www.ce.pdn.ac.lk/>
Undergraduate,
>>>>>>>>> > Faculty of Engineering University of Peradeniya
>>>>>>>>> >
>>>>>>>>> > <https://www.avast.com/sig-email?utm_medium=email&utm_
>>>>>>>>> > source=link&utm_campaign=sig-email&utm_content=webmail&utm_t
>>>>>>>>> erm=icon>
>>>>>>>>> > Virus-free.
>>>>>>>>> > www.avast.com
>>>>>>>>> > <https://www.avast.com/sig-email?utm_medium=email&utm_
>>>>>>>>> > source=link&utm_campaign=sig-email&utm_content=webmail&utm_t
>>>>>>>>> erm=link>
>>>>>>>>> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Piergiorgio Lucidi
>>>>>>>>> https://www.open4dev.com
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks and Regards,
>>>>>>>> Irindu Nugawela,
>>>>>>>> Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate,
>>>>>>>> Faculty of Engineering University of Peradeniya
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks and Regards,
>>>>>>> Irindu Nugawela,
>>>>>>> Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate,
>>>>>>> Faculty of Engineering University of Peradeniya
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Piergiorgio Lucidi
>>>>>> https://www.open4dev.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks and Regards,
>>>>> Irindu Nugawela,
>>>>> Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate,
>>>>> Faculty of Engineering University of Peradeniya
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Piergiorgio Lucidi
>>>> https://www.open4dev.com
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Irindu Nugawela,
>>> Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate,
>>> Faculty of Engineering University of Peradeniya
>>>
>>
>>
>>
>> --
>> Piergiorgio Lucidi
>> https://www.open4dev.com
>>
>
>
>
> --
> Thanks and Regards,
> Irindu Nugawela,
> Computer Engineering <http://www.ce.pdn.ac.lk/> Undergraduate,
> Faculty of Engineering University of Peradeniya
>



-- 
Piergiorgio Lucidi
https://www.open4dev.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message