manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Problem crawling windows share
Date Mon, 12 Dec 2011 09:24:33 GMT
I forwarded this research again to Mr. Allen for further suggestions.  Thanks!

Karl

On Mon, Dec 12, 2011 at 1:58 AM, Swapna Vuppala
<swapna.kollipara@gmail.com> wrote:
> Hi Karl,
>
> I tried modifying the test code as you suggested.
>
> When I tried with empty credentials
>
> NtlmPasswordAuthentication pa = new NtlmPasswordAuthentication("","","");
> SmbFile smbconnection = new SmbFile("smb://" +
> "lon-data-arx.global.arup.com" + "/" + "stg-jobs$",pa);
>
> I got the same exception as before
>
> Exception in thread "main" jcifs.smb.SmbAuthException: Logon failure:
> unknown us
>
> er name or bad password.
>         at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:546)
>         at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
>
>         at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>         at jcifs.smb.SmbSession.send(SmbSession.java:218)
>         at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>         at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>
> ......
>
> When I tried with NtlmPasswordAuthentication.ANONYMOUS
>
> SmbFile smbconnection = new SmbFile("smb://" +
> "lon-data-arx.global.arup.com" + "/" +
> "stg-jobs$",NtlmPasswordAuthentication.ANONYMOUS);
>
> I got a different exception
>
> Exception in thread "main" jcifs.smb.SmbException: Incorrect function.
>         at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:563)
>         at jcifs.smb.SmbTransport.send(SmbTransport.java:663)
>         at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:316)
>
>         at jcifs.smb.SmbSession.send(SmbSession.java:218)
>         at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>         at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
> .....
>
> Thanks and Regards,
> Swapna.
> On Fri, Dec 9, 2011 at 11:46 PM, Karl Wright <daddywri@gmail.com> wrote:
>>
>> Michael Allen's response to your capture:
>>
>> "It looks like the non-JCIFS communication is successful because the
>> anonymous credential is used.
>>
>> Note that there is still no sign of DFS going on anywhere. Actually
>> JCIFS tries to do a referral for the DFS root and gets an error of
>> STATUS_FS_DRIVER_REQUIRED so it looks like DFS may not even be setup
>> at all in the target environment. At least not domain based DFS.
>>
>> So if you want to emulate Windows behavior, I guess you need to retry
>> the operation with credential NtlmPasswordAuthentication.ANONYMOUS (or
>> all empty strings should work too I think like ";:")."
>>
>> In other words, the issue is that the server in question does not
>> actually recognize the credentials given at all, and gives an error.
>> But while Windows then retries with no credentials whatsoever, nothing
>> in JCIFS does that retry.
>>
>> You can confirm that this is what is happening by using an empty user
>> name, password, and domain in the JCIFS test code you wrote to obtain
>> the captures.  If, then, the test code works against that server, I'll
>> create a ticket for downgrading to ANONYMOUS whenever there is an
>> authentication error in the JCIFS connector.
>>
>> Thanks,
>> Karl
>>
>> On Wed, Dec 7, 2011 at 6:41 AM, Karl Wright <daddywri@gmail.com> wrote:
>> > The next step is to obtain two packet captures using WireShark.  These
>> > should both be done from the machine that is not joined to the domain.
>> >  The first capture should be the interaction that takes place when you
>> > try to go to the foreign directory, and you supply the credentials,
>> > and then you list the folders.  The second capture will require you to
>> > do the same using JCIFS.  For that capture, you will need jcifs.jar
>> > and a simple example class that provides jcifs with credentials and
>> > tries to list the directory.  A simple class should be enough to do
>> > this, including code such as:
>> >
>> > import jcifs.smb.*;
>> >
>> > NtlmPasswordAuthentication pa = new
>> > NtlmPasswordAuthentication(domain,username,password);
>> > SmbFile smbconnection = new SmbFile("smb://" + server + "/" + share,pa);
>> > SmbFile[] files = smbconnection.listFiles();
>> >
>> > for (int i = 0 ; i < files.length ; i++)
>> > {
>> >        System.out.println(file.toString());
>> > }
>> >
>> > I suggest you put together your test class first and make sure it
>> > works, and then go after the captures.
>> >
>> > Let me know how it goes.
>> >
>> > Karl
>> >
>> > On Wed, Dec 7, 2011 at 6:10 AM, Swapna Vuppala
>> > <swapna.kollipara@gmail.com> wrote:
>> >> Hi Karl,
>> >>
>> >> Thanks for the detailed reply.
>> >>
>> >> As per your suggestion, I tried connecting to the problematic server
>> >> via
>> >> windows using a machine that is not in the domain. On supplying the
>> >> credentials via windows, I could connect to the server, although it did
>> >> not
>> >> list any folders. But when I tried accessing the folder of my interest
>> >> via
>> >> \\server\folder$, I could see all the sub folders. All this is when I
>> >> connected via windows from a machine that's not joined to the domain.
>> >>
>> >> The behavior is same when I connect to the server via windows from a
>> >> machine
>> >> that's joined to the domain.
>> >>
>> >> So I assume this means that through both Kerberos and NTLM, I see the
>> >> same
>> >> behavior. Whereas ManifoldCF throws an exception when trying to define
>> >> the
>> >> repository connection.
>> >>
>> >> Please advise me on what I should be doing next to resolve this.
>> >>
>> >> Thanks and Regards,
>> >> Swapna.
>> >>
>> >>
>> >>
>> >>
>> >> On Tue, Dec 6, 2011 at 12:55 PM, Karl Wright <daddywri@gmail.com>
>> >> wrote:
>> >>>
>> >>> About your capture - Michael Allen says the following:
>> >>>
>> >>> "Actually this has nothing to do with DFS. JCIFS does not get to the
>> >>> point where it does DFS anything. The capture shows a vanilla
>> >>> STATUS_LOGON_FAILURE when GLOBAL\swapna.vuppala tries to auth with
>> >>> l-carx01.global.arup.com. So the possible causes for this are 1) the
>> >>> account name is not valid 2) the supplied password is incorrect 3)
>> >>> some security policy is deliberately blocking that user or particular
>> >>> type of auth or 4) some server configuration is incompatible with
>> >>> JCIFS. I only mention this last option because I noticed the target
>> >>> server has security signatures disabled. That's strange. If they're
>> >>> messing around with things like that, who knows what their clients are
>> >>> expected to do.
>> >>>
>> >>> Try a Windows client that uses NTLM instead of Kerberos. Meaning try
a
>> >>> machine that is not joined to the domain so that when you try to
>> >>> access the target it asks you for credentials at which point you can
>> >>> test with GLOBAL\swapna.vuppala. Then it will use NTLM and you can
>> >>> actually compare captures. If the operator doesn't have a laptop or
>> >>> something not joined to the domain, it might be sufficient to log into
>> >>> a workstation using machine credentials and not domain credentials.
>> >>>
>> >>> Also when testing JCIFS you should use a simple stand-alone program
>> >>> like examples/ListFiles.java."
>> >>>
>> >>> In other words:
>> >>> (a) Since JCIFS does not use Kerberos for authentication, you need to
>> >>> try to log into the recalcitrant server via Windows without using
>> >>> Kerberos to be able to do a side-by-side comparison.  Michael has some
>> >>> ways of doing that, above.
>> >>> (b) You may find that it doesn't work, in which case JCIFS is not
>> >>> going to work either.
>> >>> (c) If it *does* work, then try to generate your side-by-side
>> >>> comparisons using a simpler example rather than ManifoldCF en toto;
>> >>> you can see how at jcifs.samba.org, or I can help you further.
>> >>>
>> >>> He also mentions that there is some bizarreness on the response that
>> >>> indicates that the server is configured in a way that he's never seen
>> >>> before.  And believe me, Michael has seen a *lot* of strange
>> >>> configurations...
>> >>>
>> >>> Hope this helps.
>> >>> Karl
>> >>>
>> >>> On Mon, Nov 28, 2011 at 4:12 AM, Karl Wright <daddywri@gmail.com>
>> >>> wrote:
>> >>> > That should read "properties.xml", not "properties.ini".  It looks
>> >>> > like this page needs updating.
>> >>> >
>> >>> > The debug property in the XML form is:
>> >>> >
>> >>> > <property name="org.apache.manifoldcf.connectors" value="DEBUG"/>
>> >>> >
>> >>> > I don't think it will provide you with any additional information
>> >>> > that
>> >>> > is useful for debugging your authentication issue, however, if
that
>> >>> > is
>> >>> > why you are looking at it.  There may be some jcifs.jar debugging
>> >>> > switches that might be of more help, but in the end I suspect you
>> >>> > will
>> >>> > need a packet capture of both a successful connection (via Windows)
>> >>> > and an unsuccessful one (via MCF).  The guy you will need to talk
>> >>> > with
>> >>> > after that is the jcifs author Michael Allen; I can give you his
>> >>> > email
>> >>> > address if you get that far.
>> >>> >
>> >>> > Karl
>> >>> >
>> >>> >
>> >>> > On Mon, Nov 28, 2011 at 1:30 AM, Swapna Vuppala
>> >>> > <swapna.kollipara@gmail.com> wrote:
>> >>> >> Hi Karl,
>> >>> >>
>> >>> >> I was planning to debug jCIFS repository connection using WireShark
>> >>> >> and
>> >>> >> I
>> >>> >> came across this
>> >>> >> https://cwiki.apache.org/CONNECTORS/debugging-connections.html
>> >>> >> Here, I see something as add
>> >>> >> "org.apache.manifoldcf.connectors=DEBUG"
>> >>> >> to the
>> >>> >> properties.ini file. Is it the properties.xml file that is
being
>> >>> >> referred
>> >>> >> here ? If not, where do I find properties.ini file ?
>> >>> >>
>> >>> >> Thanks and Regards,
>> >>> >> Swapna.
>> >>> >>
>> >>> >> On Thu, Nov 17, 2011 at 1:31 PM, Karl Wright <daddywri@gmail.com>
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> See http://jcifs.samba.org/src/docs/api/overview-summary.html#scp.
>> >>> >>> The properties jcifs.smb.lmCompatibility and
>> >>> >>> jcifs.smb.client.useExtendedSecurity are the ones you may
want to
>> >>> >>> change.  These two properties go together so certain combinations
>> >>> >>> make
>> >>> >>> sense and others don't, so there's really only combinations
you
>> >>> >>> need
>> >>> >>> but I'll need to look at what they are and get back to
you later
>> >>> >>> today.
>> >>> >>>
>> >>> >>> As far as setting the switches are concerned, if you are
using the
>> >>> >>> Quick Start you do this trivially by:
>> >>> >>>
>> >>> >>> <java> -Dxxx -Dyyy -jar start.jar
>> >>> >>>
>> >>> >>> If you are using the multi-process configuration, that
is what the
>> >>> >>> "defines" directory is for; you only need to create files
in that
>> >>> >>> directory with the names "jcifs.smb.lmCompatibility" and
>> >>> >>> "jcifs.smb.client.useExtendedSecurity" containing the values
you
>> >>> >>> want
>> >>> >>> to set.
>> >>> >>>
>> >>> >>> Karl
>> >>> >>>
>> >>> >>>
>> >>> >>> On Thu, Nov 17, 2011 at 1:11 AM, Swapna Vuppala
>> >>> >>> <swapna.kollipara@gmail.com> wrote:
>> >>> >>> > Hi Karl,
>> >>> >>> >
>> >>> >>> > Am able to access the folders on the problem server
through
>> >>> >>> > windows
>> >>> >>> > explorer, (\\server3\Folder1). I tried couple of things
with the
>> >>> >>> > credentials
>> >>> >>> > form, changing username, domain etc.. but I keep getting
the
>> >>> >>> > same
>> >>> >>> > error
>> >>> >>> > "Couldn't connect to server: Logon failure: unknown
user name or
>> >>> >>> > bad
>> >>> >>> > password"
>> >>> >>> >
>> >>> >>> > Can you tell me more about the -D switch you were
talking of ?
>> >>> >>> >
>> >>> >>> > Thanks and Regards,
>> >>> >>> > Swapna.
>> >>> >>> >
>> >>> >>> > On Tue, Nov 15, 2011 at 12:40 PM, Karl Wright
>> >>> >>> > <daddywri@gmail.com>
>> >>> >>> > wrote:
>> >>> >>> >>
>> >>> >>> >> Glad you chased it down this far.
>> >>> >>> >>
>> >>> >>> >> First thing to try is whether you can get into
the problem
>> >>> >>> >> server
>> >>> >>> >> using Windows Explorer.  Obviously ManifoldCF
is not going to
>> >>> >>> >> be
>> >>> >>> >> able
>> >>> >>> >> to do it if Windows can't.  If you *can* get
in, then just
>> >>> >>> >> playing
>> >>> >>> >> with the form of the credentials in the MCF connection
might do
>> >>> >>> >> the
>> >>> >>> >> trick.  Some Windows or net appliance servers
are picky about
>> >>> >>> >> this.
>> >>> >>> >> Try various things like leaving the domain blank
and specifying
>> >>> >>> >> the
>> >>> >>> >> user as "abc@domain.com", for instance. There's
also a
>> >>> >>> >> different
>> >>> >>> >> NTLM
>> >>> >>> >> mode you can operation jcifs in that some servers
may be
>> >>> >>> >> configured
>> >>> >>> >> to
>> >>> >>> >> require; this would need you to set a -D switch
on the command
>> >>> >>> >> line
>> >>> >>> >> to
>> >>> >>> >> enable.
>> >>> >>> >>
>> >>> >>> >> Karl
>> >>> >>> >>
>> >>> >>> >> On Tue, Nov 15, 2011 at 12:10 AM, Swapna Vuppala
>> >>> >>> >> <swapna.kollipara@gmail.com> wrote:
>> >>> >>> >> > Hi Karl,
>> >>> >>> >> >
>> >>> >>> >> > Thanks for the input. It looks like my problem
is related to
>> >>> >>> >> > the
>> >>> >>> >> > second
>> >>> >>> >> > one
>> >>> >>> >> > that you specified. One of the directories
in the path am
>> >>> >>> >> > trying
>> >>> >>> >> > to
>> >>> >>> >> > index is
>> >>> >>> >> > actually redirecting to a different server.
And when I
>> >>> >>> >> > specify
>> >>> >>> >> > this
>> >>> >>> >> > new
>> >>> >>> >> > server in defining the repository connection,
with my
>> >>> >>> >> > credentials,
>> >>> >>> >> > the
>> >>> >>> >> > connection fails with the message:  "Couldn't
connect to
>> >>> >>> >> > server:
>> >>> >>> >> > Logon
>> >>> >>> >> > failure: unknown user name or bad password"
>> >>> >>> >> >
>> >>> >>> >> > I'll look into why am not able to connect
to this server.
>> >>> >>> >> >
>> >>> >>> >> > Thanks and Regards,
>> >>> >>> >> > Swapna.
>> >>> >>> >> >
>> >>> >>> >> > On Mon, Nov 14, 2011 at 4:56 PM, Karl Wright
>> >>> >>> >> > <daddywri@gmail.com>
>> >>> >>> >> > wrote:
>> >>> >>> >> >>
>> >>> >>> >> >> There's two kinds of problem you might
be having.  The first
>> >>> >>> >> >> is
>> >>> >>> >> >> intermittent, and the second is not intermittent
but would
>> >>> >>> >> >> have
>> >>> >>> >> >> something to do with specific directories.
>> >>> >>> >> >>
>> >>> >>> >> >> Intermittent problems might include a
domain controller that
>> >>> >>> >> >> is
>> >>> >>> >> >> not
>> >>> >>> >> >> always accessible.  In such cases, the
crawl will proceed
>> >>> >>> >> >> but
>> >>> >>> >> >> will
>> >>> >>> >> >> tend to fail unpredictably.  On the
other hand, if you have
>> >>> >>> >> >> a
>> >>> >>> >> >> directory that is handled by a DFS redirection,
it is
>> >>> >>> >> >> possible
>> >>> >>> >> >> that
>> >>> >>> >> >> the redirection is indicating a new server
(lets call it
>> >>> >>> >> >> server3)
>> >>> >>> >> >> which may not like the precise form of
your login
>> >>> >>> >> >> credentials.
>> >>> >>> >> >>  Can
>> >>> >>> >> >> you determine which scenario you are
seeing?
>> >>> >>> >> >>
>> >>> >>> >> >> Karl
>> >>> >>> >> >>
>> >>> >>> >> >> On Mon, Nov 14, 2011 at 3:11 AM, Swapna
Vuppala
>> >>> >>> >> >> <swapna.kollipara@gmail.com> wrote:
>> >>> >>> >> >> > Hi,
>> >>> >>> >> >> >
>> >>> >>> >> >> > I have been using windows share
repository connection to
>> >>> >>> >> >> > crawl
>> >>> >>> >> >> > and
>> >>> >>> >> >> > get
>> >>> >>> >> >> > data
>> >>> >>> >> >> > from a particular server (server
1). Its working perfectly
>> >>> >>> >> >> > fine.
>> >>> >>> >> >> > However, am
>> >>> >>> >> >> > having trouble when I try with data
from another server
>> >>> >>> >> >> > (server
>> >>> >>> >> >> > 2).
>> >>> >>> >> >> >
>> >>> >>> >> >> > When I define a repository connection
of type windows
>> >>> >>> >> >> > share
>> >>> >>> >> >> > and
>> >>> >>> >> >> > specify
>> >>> >>> >> >> > the
>> >>> >>> >> >> > server name (server 2) with my credentials,
the connection
>> >>> >>> >> >> > status
>> >>> >>> >> >> > shows
>> >>> >>> >> >> > "Connection working". But when I
run a job to use this
>> >>> >>> >> >> > repository
>> >>> >>> >> >> > connection
>> >>> >>> >> >> > and index data from a location on
this server 2, I keep
>> >>> >>> >> >> > getting
>> >>> >>> >> >> > the
>> >>> >>> >> >> > exception below:
>> >>> >>> >> >> >
>> >>> >>> >> >> > JCIFS: Possibly transient exception
detected on attempt 3
>> >>> >>> >> >> > while
>> >>> >>> >> >> > checking
>> >>> >>> >> >> > if
>> >>> >>> >> >> > file exists: Logon failure: unknown
user name or bad
>> >>> >>> >> >> > password.
>> >>> >>> >> >> > jcifs.smb.SmbAuthException: Logon
failure: unknown user
>> >>> >>> >> >> > name
>> >>> >>> >> >> > or
>> >>> >>> >> >> > bad
>> >>> >>> >> >> > password.
>> >>> >>> >> >> >     at
>> >>> >>> >> >> > jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:544)
>> >>> >>> >> >> >     at jcifs.smb.SmbTransport.send(SmbTransport.java:661)
>> >>> >>> >> >> >     at
>> >>> >>> >> >> > jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)
>> >>> >>> >> >> >     at jcifs.smb.SmbSession.send(SmbSession.java:218)
>> >>> >>> >> >> >     at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)
>> >>> >>> >> >> >     at jcifs.smb.SmbFile.doConnect(SmbFile.java:911)
>> >>> >>> >> >> >     at jcifs.smb.SmbFile.connect(SmbFile.java:954)
>> >>> >>> >> >> >     at jcifs.smb.SmbFile.connect0(SmbFile.java:880)
>> >>> >>> >> >> >     at jcifs.smb.SmbFile.queryPath(SmbFile.java:1335)
>> >>> >>> >> >> >     at jcifs.smb.SmbFile.exists(SmbFile.java:1417)
>> >>> >>> >> >> >     at
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.fileExists(SharedDriveConnector.java:2064)
>> >>> >>> >> >> >     at
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> > org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector.getDocumentVersions(SharedDriveConnector.java:521)
>> >>> >>> >> >> >     at
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> >
>> >>> >>> >> >> > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:318)
>> >>> >>> >> >> >
>> >>> >>> >> >> > Am able to access this location
from windows explorer.
>> >>> >>> >> >> > What
>> >>> >>> >> >> > else
>> >>> >>> >> >> > should
>> >>> >>> >> >> > I be
>> >>> >>> >> >> > checking or what could be the reasons/factors
causing this
>> >>> >>> >> >> > to
>> >>> >>> >> >> > fail
>> >>> >>> >> >> > ?
>> >>> >>> >> >> >
>> >>> >>> >> >> > Thanks and Regards,
>> >>> >>> >> >> > Swapna.
>> >>> >>> >> >> >
>> >>> >>> >> >
>> >>> >>> >> >
>> >>> >>> >
>> >>> >>> >
>> >>> >>
>> >>> >>
>> >>
>> >>
>
>

Mime
View raw message