manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Sharepoint connector question
Date Wed, 15 Sep 2010 21:53:34 GMT
Any news on this issue?
Karl

On Mon, Sep 13, 2010 at 3:00 PM, Karl Wright <daddywri@gmail.com> wrote:

> I would expect that domain administrator privs would be sufficient. ;-)
>
> Unfortunately, SharePoint (and .NET services in general) often seem to have
> unusual security problems with internal communication.  I've seen cases
> where some of SharePoint's own web services have this issue.  Never was able
> to figure out the problem.  Perhaps a .NET guru could, but that's not me.
>
> Karl
>
>
> On Mon, Sep 13, 2010 at 2:55 PM, Martijn v Groningen <
> martijn.is.hier@gmail.com> wrote:
>
>> That could explain the error. I've installed and uninstalled the
>> webservice extension a few times, but I know for sure that the last
>> time I installed it as domain administrator. Last week I used a
>> plain-vanilla Sharepoint (trail version), the webservice extension
>> worked there without any problem.
>>
>> On 13 September 2010 19:30, Karl Wright <daddywri@gmail.com> wrote:
>> > The key error is the following:
>> >
>> >>>>>>>
>> > <soap:Fault><faultcode>soap:Client</faultcode><faultstring>The
>> > request failed with HTTP status 401:
>> > Unauthorized.</faultstring><faultactor>http://
>> [host]/_vti_bin/MCPermissions.asmx</faultactor><detail><Error><ErrorNumber>1000</ErrorNumber><ErrorMessage>The
>> > request failed with HTTP status 401:
>> >
>> Unauthorized.</ErrorMessage><ErrorSource>System.Web.Services</ErrorSource></Error></detail></soap:Fault>
>> > <<<<<<
>> >
>> > Clearly the MCPermissions web service does not have sufficient
>> permissions
>> > to perform its task.  I don't recall ever having seen this before, but
>> > perhaps during installation you were not logged in as a user that has
>> enough
>> > permission to perform security lookups.
>> >
>> > Karl
>> >
>> > On Mon, Sep 13, 2010 at 12:15 PM, Martijn v Groningen
>> > <martijn.is.hier@gmail.com> wrote:
>> >>
>> >> Hi Karl,
>> >>
>> >> Today I'm not at the environment where I can verify this, but I'll
>> >> definitely check this. But I ran into another issue with the
>> >> Sharepoint connector. In a another environment I installed the
>> >> Metacarta Sharepoint webservice extensions, but by executing the
>> >> following post:
>> >> HTTP POST http://[host]/rotterdamn/_vti_bin/MCPermissions.asmx
>> >> <?xml version="1.0" encoding="UTF-8"?><soapenv:Envelope
>> >> xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
>> >> xmlns:xsd="http://www.w3.org/2001/XMLSchema"
>> >>
>> >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
>> "><soapenv:Body><GetPermissionCollection
>> >>
>> >> xmlns="http://microsoft.com/sharepoint/webpartpages/
>> "><objectName>/</objectName><objectType>Web</objectType></GetPermissionCollection></soapenv:Body></soapenv:Envelope>
>> >>
>> >> I get back the following response (http 500):
>> >> <?xml version="1.0" encoding="utf-8"?><soap:Envelope
>> >> xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
>> >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >>
>> >> xmlns:xsd="http://www.w3.org/2001/XMLSchema
>> "><soap:Body><soap:Fault><faultcode>soap:Client</faultcode><faultstring>The
>> >> request failed with HTTP status 401:
>> >>
>> >> Unauthorized.</faultstring><faultactor>http://
>> [host]/_vti_bin/MCPermissions.asmx</faultactor><detail><Error><ErrorNumber>1000</ErrorNumber><ErrorMessage>The
>> >> request failed with HTTP status 401:
>> >>
>> >>
>> Unauthorized.</ErrorMessage><ErrorSource>System.Web.Services</ErrorSource></Error></detail></soap:Fault></soap:Body></soap:Envelope>
>> >>
>> >> I seems that I only get an error with the MCPermissions webservice
>> >> call. Other calls such as GetListCollection work fine. I'm
>> >> authentication with the domain administrator account. This environment
>> >> has also Sharepoint 3.0 installed. I'm making these posts to
>> >> Sharepoint with Firefox http poster plugin.  Also the url in the
>> >> response is without the subsite.
>> >>
>> >> Also important to note is that browsing to
>> >> http://[host]/[subsite]/_vti_bin/MCPermissions.asmx shows the
>> >> GetPermissionsCollection operation. That is what I checked after
>> >> installing the webservice extension. You have a clue what might be
>> >> wrong here?
>> >>
>> >> Thanks,
>> >>
>> >> Martijn
>> >>
>> >> On 13 September 2010 10:27, Karl Wright <daddywri@gmail.com> wrote:
>> >> > Hi Martijn,
>> >> >
>> >> > For the 401 error, here's something also worth trying, to remove the
>> >> > possibility that your error has anything to do with other recent
>> >> > changes.
>> >> > Can you check out the following:
>> >> >
>> >> > svn co -r987345
>> >> > https://svn.apache.org/repos/asf/incubator/lcf/trunk/modules/lib
>> >> >
>> >> > In the checkout lib area, you will see a jar called
>> >> > commons-httpclient-lcf.jar.  Replace commons-httpclient-acf.jar with
>> it
>> >> > (renaming to commons-httpclient-acf.jar, of course), and try running
>> >> > with
>> >> > it.  If your 401 error no longer happens, then it means something was
>> >> > messed
>> >> > up, and I'll need to do some research.
>> >> >
>> >> > Thanks,
>> >> > Karl
>> >> >
>> >> > On Sun, Sep 12, 2010 at 5:02 PM, Karl Wright <daddywri@gmail.com>
>> wrote:
>> >> >>
>> >> >> I confirmed that without any mappings set, the Solr Connector
>> *should*
>> >> >> just be passing the metadata through using the metadata's name
as
>> the
>> >> >> Solr
>> >> >> field name.
>> >> >>
>> >> >> For debugging, if you could post the Solr output from one update
>> >> >> operation, I'd love to see if any metadata seems to be in it.
>> >> >> Potentially
>> >> >> it's there but the Solr schema is not right somehow - that should
be
>> >> >> the
>> >> >> first thing we verify.
>> >> >>
>> >> >> Karl
>> >> >>
>> >> >>
>> >> >> On Sun, Sep 12, 2010 at 4:50 PM, Martijn v Groningen
>> >> >> <martijn.is.hier@gmail.com> wrote:
>> >> >>>
>> >> >>> Tomorrow I'll dive into code and do some more debugging. Last
week
>> I
>> >> >>> didn't specify any mappings in the mapping tab for the meta
data
>> >> >>> fields I selected in the metadata tab. But this shouldn't be
the
>> >> >>> problem, right?
>> >> >>>
>> >> >>> Thanks,
>> >> >>>
>> >> >>> Martijn
>> >> >>>
>> >> >>> On 12 September 2010 22:29, Karl Wright <daddywri@gmail.com>
>> wrote:
>> >> >>> > Martijn,
>> >> >>> >
>> >> >>> > (1) The precise svn url for the acf version of httpclient
is as
>> >> >>> > follows.  My
>> >> >>> > apologies for any earlier confusion - I was away from
my computer
>> at
>> >> >>> > the
>> >> >>> > time.
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> https://svn.apache.org/repos/asf/incubator/lcf/upstream/commons-httpclient-3x
>> >> >>> >
>> >> >>> > (2) Each time the solr connector posts into Solr, you
should see
>> a
>> >> >>> > set
>> >> >>> > of
>> >> >>> > argument names and values dumped to standard out (or the
log).
>> So
>> >> >>> > it
>> >> >>> > should
>> >> >>> > be easy to see what is being sent, and whether the arguments
in
>> fact
>> >> >>> > are the
>> >> >>> > correct ones for the extracting update request handler,
or not.
>> >> >>> > Furthermore, the Solr output connector recently had a
tab added
>> >> >>> > which
>> >> >>> > performs the mapping I alluded to.  This mapping is designed
to
>> >> >>> > translate
>> >> >>> > metadata coming from a connector like SharePoint, into
fields
>> that
>> >> >>> > you
>> >> >>> > presumably have in your Solr schema.  However, if you
don't set
>> >> >>> > anything,
>> >> >>> > the fields are not changed, and you should see an argument
for
>> every
>> >> >>> > metadata field, something like: literal.xxx=yyy.
>> >> >>> >
>> >> >>> > If you have a document that you *know* has metadata, and
you've
>> >> >>> > specified
>> >> >>> > that metadata in the job, and you run the job after you
specify
>> that
>> >> >>> > metadata, but still see no literal.xxx=yyy corresponding
to it in
>> >> >>> > the
>> >> >>> > Solr
>> >> >>> > output, then we should spend some time chasing this problem
down.
>> >> >>> > Be
>> >> >>> > wary
>> >> >>> > because incremental crawling means you'll probably not
see your
>> >> >>> > document
>> >> >>> > processed again unless you either change it in SharePoint,
or
>> delete
>> >> >>> > and
>> >> >>> > recreate the job.  But be reassured that SharePoint metadata
was
>> >> >>> > covered by
>> >> >>> > the old MetaCarta tests, and there have been no changes
of any
>> >> >>> > significance
>> >> >>> > to the SharePoint connector since then, so I have no explanation
>> why
>> >> >>> > it
>> >> >>> > would not work for you too.  That's why I'm spending time
trying
>> to
>> >> >>> > figure
>> >> >>> > out if this is a Solr connector issue instead.
>> >> >>> >
>> >> >>> > Please let me know if this helps you, or whether you need
to go
>> >> >>> > deeper
>> >> >>> > into
>> >> >>> > debugging.
>> >> >>> >
>> >> >>> > Karl
>> >> >>> >
>> >> >>> >
>> >> >>> > On Sun, Sep 12, 2010 at 4:05 PM, Martijn v Groningen
>> >> >>> > <martijn.is.hier@gmail.com> wrote:
>> >> >>> >>
>> >> >>> >> I didn't notice that I was under the upstream-changes
directory.
>> >> >>> >> Thanks for pointing that out.
>> >> >>> >>
>> >> >>> >> In Solr I have a wildcard (*) dynamic field, so everything
acf
>> >> >>> >> sends
>> >> >>> >> should end up in my index (or at least that is what
I assume). I
>> >> >>> >> also
>> >> >>> >> did some debugging in the Solr connecter and I noticed
that no
>> >> >>> >> metadata was send to Solr. I didn't create field mappings
in my
>> acf
>> >> >>> >> job. Do you always have to make mapping for metadata?
>> >> >>> >>
>> >> >>> >> Martijn
>> >> >>> >>
>> >> >>> >> On 12 September 2010 21:09, Karl Wright <daddywri@gmail.com>
>> wrote:
>> >> >>> >> > The source for upstream changes is under
>> >> >>> >> > lcf/upstream-changes/httpclient, not under trunk.
>> >> >>> >> >
>> >> >>> >> > As for the metadata, how are you determining
that no metadata
>> is
>> >> >>> >> > being
>> >> >>> >> > indexed?  If this is Solr you are indexing into,
have you set
>> up
>> >> >>> >> > the
>> >> >>> >> > appropriate metadata/field mappings?
>> >> >>> >> >
>> >> >>> >> > Karl
>> >> >>> >> >
>> >> >>> >> > On 9/12/10, Martijn v Groningen <martijn.is.hier@gmail.com>
>> >> >>> >> > wrote:
>> >> >>> >> >> To authenticate with Share point I had to
include the domain
>> as
>> >> >>> >> >> well.
>> >> >>> >> >> Also the ui reported an error if I didn't
specify the
>> username
>> >> >>> >> >> in a
>> >> >>> >> >> domain / username format. Maybe this http
client issue was
>> just
>> >> >>> >> >> particular with the Sharepoint / Domain Controller
>> installation
>> >> >>> >> >> I
>> >> >>> >> >> was
>> >> >>> >> >> working with. I also couldn't find the source
of afc version
>> of
>> >> >>> >> >> http
>> >> >>> >> >> client. Is it hosted in another source repository?
>> >> >>> >> >>
>> >> >>> >> >> I still don't understand why for the documents
I crawled, I
>> >> >>> >> >> didn't
>> >> >>> >> >> have any metadata associated with it. In
the job
>> configuration I
>> >> >>> >> >> was
>> >> >>> >> >> able to choose which metadata I wanted to
include. You have
>> an
>> >> >>> >> >> idea
>> >> >>> >> >> what might be the cause of this?
>> >> >>> >> >>
>> >> >>> >> >> Regards,
>> >> >>> >> >>
>> >> >>> >> >> Martijn
>> >> >>> >> >>
>> >> >>> >> >> On 12 September 2010 18:40, Karl Wright <daddywri@gmail.com>
>> >> >>> >> >> wrote:
>> >> >>> >> >>> Hi Martijn,
>> >> >>> >> >>>
>> >> >>> >> >>> The ACF version of httpclient has support
for NTLMv1,
>> NTLMv2,
>> >> >>> >> >>> and
>> >> >>> >> >>> NTLM2
>> >> >>> >> >>> protocols.  The standard client does
not.
>> >> >>> >> >>>
>> >> >>> >> >>> What this means practically for you depends
on how the
>> Windows
>> >> >>> >> >>> domain
>> >> >>> >> >>> controller you are working with is configured.
 You cannot
>> use
>> >> >>> >> >>> the
>> >> >>> >> >>> off-the-shelf httpclient and still authenticate
if the
>> domain
>> >> >>> >> >>> controller
>> >> >>> >> >>> is
>> >> >>> >> >>> configured to not allow LM connections,
which is what
>> Microsoft
>> >> >>> >> >>> recommends
>> >> >>> >> >>> people do.
>> >> >>> >> >>>
>> >> >>> >> >>> Since the ACF version of httpclient will
always try to
>> connect
>> >> >>> >> >>> using
>> >> >>> >> >>> NTLMv2,
>> >> >>> >> >>> this means that you must be more rigorous
about setting up
>> your
>> >> >>> >> >>> client
>> >> >>> >> >>> machine.  First, it must have a name,
and it must have a
>> >> >>> >> >>> machine
>> >> >>> >> >>> account
>> >> >>> >> >>> in
>> >> >>> >> >>> the domain.  Second, NTLMv2 is much more
picky about how you
>> >> >>> >> >>> specify
>> >> >>> >> >>> user
>> >> >>> >> >>> and domain.  The end user documentation
provides details
>> that
>> >> >>> >> >>> may
>> >> >>> >> >>> be
>> >> >>> >> >>> helpful
>> >> >>> >> >>> to you in this regard.
>> >> >>> >> >>>
>> >> >>> >> >>> Thanks,
>> >> >>> >> >>> Karl
>> >> >>> >> >>>
>> >> >>> >> >>>
>> >> >>> >> >>> On Sun, Sep 12, 2010 at 5:00 AM, Martijn
v Groningen
>> >> >>> >> >>> <martijn.is.hier@gmail.com> wrote:
>> >> >>> >> >>>>
>> >> >>> >> >>>> Hi All,
>> >> >>> >> >>>>
>> >> >>> >> >>>> I've configured the Sharepoint connector
(to connect to
>> >> >>> >> >>>> sharepoint
>> >> >>> >> >>>> 3.0), Solr connector and a job that
adds documents into
>> Solr.
>> >> >>> >> >>>> The
>> >> >>> >> >>>> only
>> >> >>> >> >>>> thing that I'm missing is the meta
data from Sharepoint.
>> Per
>> >> >>> >> >>>> document
>> >> >>> >> >>>> I need to know which users can access
it. In the metadata
>> tab
>> >> >>> >> >>>> on
>> >> >>> >> >>>> the
>> >> >>> >> >>>> job page I've configured the metadata
to be included, but
>> this
>> >> >>> >> >>>> doesn't
>> >> >>> >> >>>> end up in my Solr index. Does anybody
know what I should do
>> to
>> >> >>> >> >>>> also
>> >> >>> >> >>>> have the metadata in my index?
>> >> >>> >> >>>>
>> >> >>> >> >>>> I also had another issue with the
Sharepoint connector
>> which I
>> >> >>> >> >>>> managed
>> >> >>> >> >>>> to solve. But I'm curious to know
if someone else
>> encountered
>> >> >>> >> >>>> a
>> >> >>> >> >>>> similar issue.
>> >> >>> >> >>>> When I was setting up the sharepoint
connecter I always got
>> a
>> >> >>> >> >>>> 401
>> >> >>> >> >>>> message on the connectors page as
status. I was sure I
>> entered
>> >> >>> >> >>>> the
>> >> >>> >> >>>> correct credentials. After some debugging
I noticed that
>> the
>> >> >>> >> >>>> NLTM
>> >> >>> >> >>>> data
>> >> >>> >> >>>> that was send to Solr was different
then when I did a http
>> >> >>> >> >>>> post
>> >> >>> >> >>>> with
>> >> >>> >> >>>> Firefox poster plugin to a Sharepoint
webservice url (I
>> check
>> >> >>> >> >>>> this
>> >> >>> >> >>>> with Wireshark). After writing a
little test case with
>> >> >>> >> >>>> httpclient
>> >> >>> >> >>>> used
>> >> >>> >> >>>> in afc, I got the same 401 error.
I then ran the test with
>> a
>> >> >>> >> >>>> clean
>> >> >>> >> >>>> http client (version 3.1), that ran
as expected. I got a
>> >> >>> >> >>>> response
>> >> >>> >> >>>> code
>> >> >>> >> >>>> 200 back with a soap response. I
then used this version of
>> >> >>> >> >>>> http
>> >> >>> >> >>>> client
>> >> >>> >> >>>> (with some class filesfrom the afc
provided jar that were
>> >> >>> >> >>>> missing
>> >> >>> >> >>>> is
>> >> >>> >> >>>> the plain jar file) and the connector
worked as expected as
>> I
>> >> >>> >> >>>> was
>> >> >>> >> >>>> able
>> >> >>> >> >>>> to index documents. Did someone else
have this particular
>> >> >>> >> >>>> issue?
>> >> >>> >> >>>> I
>> >> >>> >> >>>> noticed that acf is using httpclient
3.1 (from the manifest
>> >> >>> >> >>>> file),
>> >> >>> >> >>>> but
>> >> >>> >> >>>> I'm curious to know why http client
was modified.
>> >> >>> >> >>>>
>> >> >>> >> >>>> BTW I've been using the latest trunk
version (I did a
>> checkout
>> >> >>> >> >>>> last
>> >> >>> >> >>>> tuesday). I'm also new to Sharepoint
>> >> >>> >> >>>>
>> >> >>> >> >>>> Cheers,
>> >> >>> >> >>>>
>> >> >>> >> >>>> Martijn
>> >> >>> >> >>>
>> >> >>> >> >>>
>> >> >>> >> >>
>> >> >>> >> >
>> >> >>> >>
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> --
>> >> >>> >> Met vriendelijke groet,
>> >> >>> >>
>> >> >>> >> Martijn van Groningen
>> >> >>> >
>> >> >>> >
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Met vriendelijke groet,
>> >> >>>
>> >> >>> Martijn van Groningen
>> >> >>
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Met vriendelijke groet,
>> >>
>> >> Martijn van Groningen
>> >
>> >
>>
>>
>>
>> --
>> Met vriendelijke groet,
>>
>> Martijn van Groningen
>>
>
>

Mime
View raw message