Mailing-List: contact dev-help@manifoldcf.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@manifoldcf.apache.org
MIME-Version: 1.0
Reply-To: rharo@apache.org
In-Reply-To: 
 <CAEawRSYoPkdmj1uPb+rVxPMPDx0cMZCx3OX554htY3AcAFhuPQ@mail.gmail.com>
References: 
 <CAEawRSa73gciEpTGO=uE1kAC1Hhhb7kBvJXKSVKHHxx=w9HsBg@mail.gmail.com>
	<CAEawRSYtZgt8ELmNiXtdRFHDwPfG5KvALN=sQF_B3+t3OeQKUw@mail.gmail.com>
	<CAEawRSYoPkdmj1uPb+rVxPMPDx0cMZCx3OX554htY3AcAFhuPQ@mail.gmail.com>
Date: Mon, 29 Jun 2015 17:55:55 +0200
Message-ID: 
 <CANW0eBdPZ0wvOm58RfNn4N9EF1VHP+watH91BeYd1Z-5eB5h=Q@mail.gmail.com>
Subject: Re: [GSoC] Confluence connector project status after bonding period
From: Rafa Haro <rharo@apache.org>
To: "dev@manifoldcf.apache.org" <dev@manifoldcf.apache.org>
Content-Type: multipart/alternative; boundary=089e011832a4ff54b60519aa1f32

--089e011832a4ff54b60519aa1f32
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Antonio,

Thanks a lot for the update and congratulations for reaching GSoC's midterm
successfully. I have been checking the proposal again carefully and, as I
have remarked in your evaluation, you are on schedule so far. Let me
comment your previous email:

On Sat, Jun 27, 2015 at 11:30 AM, Antonio David P=C3=A9rez Morales <
adperezmorales@gmail.com> wrote:

> Hi all
>
> Continuing with the development of Confluence repository connector [1] I
> have added support for processing attachments (configurable per job) for
> pages, ability to crawl each kind of pages and extract the page labels if
> they have been set.
>

Good. During this week, I will pull the last version from github and will
be testing it against a real Confluence instance on my current company. I
will also take a look to the code to check if some refactor is needed.
After testing, I should have a better idea about possible changes regarding
configuration, performance, integration details and so on. I will provide
my feedback ASAP both to you directly and here in the ManifoldCF's
developers list. In that way, anyone also interested in the connector
within the community can also provide feedback, ideas... I won't delay too
much the testing because this is the right moment to accomplish possible
changes/additions before continuing with the second part of the project
that, according to the proposal planning, should be focused on the
Authority connector, unit testing and final documentation. In summary,
let's close the repository connector first :-).


> Besides, a complete refactor of the code and client has been done, so now
> the connector has a better code flow and the specific components have bee=
n
> simplified.
>

Perfect, I will take a look to the code and will provide feedback also.


> Right now, the connector is able to process Page and attachments, being
> Page each supported confluence page type (blog, table, etc). As an
> improvement I want to add specific support for each kind of Page, giving
> the connector the ability to process Page-specific features. For example,
> for Blog pages, the connector is extracting the default/common properties=
,
> but adding the specific support for Blog page model (like it has been don=
e
> for attachments) would allow to extract Blog page-specific properties and
> set the specific type for that page to Blog instead of Page (by default).
>

Well, I suppose that we should decide a scope here, because to take into
account all types of Confluence Pages would probably complicate too much
the connector right now. This is something that we can progressively
improve or contribute after GSoC have finished. Which it is important now
is to come up with a good design for Confluence Pages in the connector in
order to ease the inclusion of concrete behavior for any kind of Page. Let
me take a look to the current code and also check how many different kind
of pages is currently supporting Confluence and will come back to you with
a suggestion.


>
> Another things that is being done in parallel is the development of the
> tests, to have a complete set of unit tests for the midterm.


> The code can be found at [1]. It is a git branch. After the midterm, I wi=
ll
> merge that branch with master one and create another one for the
> development of the authority connector to keep them separated at the
> moment.
>

Ok. I aim the rest of the community to take a look also to the code and
provide feedback. I will try to do the same with the other ManifoldCF's
GSoC project for integrating Kafka. If it is make it easier, once you merge
the connector to the master branch I could setup a branch at ManifoldCF's
svn space for easing the testing.

Cheers,
Rafa


>
> If you have any question/suggestion/comment, please drop a message here
> which will be more than welcome.
>
> Regards
> --
>
> [1]
>
> https://github.com/adperezmorales/confluence-manifold-connector/tree/feat=
ure/repository-connector
>
> 2015-06-05 18:48 GMT+02:00 Antonio David P=C3=A9rez Morales <
> adperezmorales@gmail.com>:
>
> > Hi devs and all
> >
> > As part of the development of the Atlassian Confluence connector for
> > Manifold, I have created a repository [1] on my GitHub account
> > Moreover I have developed and pushed the first version of the Confluenc=
e
> > repository connector on a branch called 'feature/repository-connector'.
> > This first version of the repository connector allows to crawl all the
> > Confluence pages contained in the spaces (only pages), with the
> possibility
> > to filter by a space.
> > At this moment, only one space or all of them can be configured to be
> > crawled, but I will continue improving the connector adding more featur=
es
> > like configuring more than one space or others you may see interesting =
to
> > be added.
> > The ACLs of the documents crawled are the space it belongs to, so that =
it
> > is aligned with the proposal for the Authority connector starting at
> Space
> > level and then going more fine grain if necessary.
> > There are no tests included at this moment, because I'm working on them=
.
> >
> > To see if the proposed Authority connector could be developed in a good
> > way, I have done a quick proof of concept with the logic of it and I wa=
s
> > able to get the spaces which the user has permission to access. So I ca=
n
> > confirm that at space level, we can add permissions to the documents
> > crawled in order to filter them later on the search engine being used.
> >
> > One important thing is that the new Confluence REST Api does not includ=
e
> > any endpoints for permissions yet, so legacy API's have to be used for
> that
> > while Confluence developers port completely the legacy methods to the n=
ew
> > api.
> >
> > I will continue improving the repository connector and thinking new
> > features to be added, but if you think some feature is interesting or
> good
> > to have, please let me know and I will take a look in order to include
> it.
> >
> > As stated in the proposal, the Authority Connector is planned for the
> > second part of the project, but I started to work a bit on it to make
> sure
> > we can have a general working version and then iteratively improving it=
.
> >
> > As always, comments and suggestions are more than welcome.
> >
> > Regards
> >
> >
> > [1] https://github.com/adperezmorales/confluence-manifold-connector/
> >
> > 2015-05-26 17:10 GMT+02:00 Antonio David P=C3=A9rez Morales <
> > adperezmorales@gmail.com>:
> >
> >> Hi all
> >>
> >> During the bonding period and these days I have been taking a look and
> >> familiarizing with Confluence API,
> >> doing some tests using CURL before start the implementation of the
> >> repository connector which is the first step as stated in the proposal=
.
> >>
> >> I have deployed a local instance of Confluence as well, so that I can =
do
> >> the development and tests using that instance.
> >>
> >> As stated in the proposal, Confluence is migrating its old APIs
> (rpc-xml,
> >> rpc-json) to the new REST API, so all the methods are not migrated yet=
.
> >> For getting the changes, fields and content of the documents there won=
't
> >> be any problem, but for permissions I have to check more in deep if th=
e
> new
> >> REST API already support it.
> >> If not, we will have to do a mix using the methods provided by the
> >> rpc-json api for that, and update it when the REST API contains all th=
e
> >> methods.
> >>
> >> After the first tests, there is no easy way to retrieve the user
> >> permissions because they are tied to documents and/or spaces. So in
> order
> >> to retrieve the user permissions,
> >> documentId or SpaceId and user have to be provided. I proposed two
> >> approaches to tackle this, one not so efficient, making many requests =
to
> >> Confluence and the other developing a Confluence plugin to get them
> >> (because at least at Java API level it is possible, but don't know yet
> what
> >> kind of permissions it returns)
> >>
> >> So I think, for that part, we can start using (trying) permissions at
> >> Space level and then try to go finer at document level.
> >> These problems are mainly related to the second part of the project
> >> (Authority Connector) but I think it is interesting to put here some
> >> results after the first overall tests I have performed.
> >>
> >> Regards
> >>
> >
> >
>

--089e011832a4ff54b60519aa1f32--