beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pei He (JIRA)" <>
Subject [jira] [Commented] (BEAM-1309) FileIOChannelFactory.match() traverses entire parent directory recursively
Date Fri, 27 Jan 2017 23:37:24 GMT


Pei He commented on BEAM-1309:

Where is match() is called, and what is the pattern?

We can probably use:

We should also make sure the pattern passed in to match() is not too broad.

> FileIOChannelFactory.match() traverses entire parent directory recursively
> --------------------------------------------------------------------------
>                 Key: BEAM-1309
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Eugene Kirpichov
>            Assignee: Pei He
> I was running a pipeline that reads a single file from my local home directory.
> The pipeline got stuck, and upon taking a stack snapshot, I noticed that it was stuck
in FileIOChannelFactory.match().
> The code currently works by traversing the whole parent directory of the requested filepattern
and checking which files match the filepattern. In my case, that means traversing everything
in my home directory, which is *a lot* (and includes remotely mounted directories).
> This is very wasteful and should be fixed.

This message was sent by Atlassian JIRA

View raw message