Python glob，但针对字符串列表而不是文件系统

Posted 2023-02-23

【中文标题】Python glob，但针对字符串列表而不是文件系统

【英文标题】：Python glob but against a list of strings rather than the filesystem

【发布时间】：2015-02-27 20:43:07

【问题描述】：

我希望能够将glob 格式的模式匹配到字符串列表，而不是文件系统中的实际文件。有什么方法可以做到这一点，或者将glob 模式轻松转换为正则表达式？

【参考方案1】：

glob 模块将fnmatch module 用于单独的路径元素。

这意味着路径分为目录名和文件名，如果目录名包含元字符（包含任何字符[、* 或?），那么这些将被扩展递归。

如果您有一个简单文件名的字符串列表，那么只需使用 fnmatch.filter() function 就足够了：

import fnmatch

matching = fnmatch.filter(filenames, pattern)

但如果它们包含完整路径，您需要做更多的工作，因为生成的正则表达式不考虑路径段（通配符不排除分隔符，也不针对跨平台路径匹配进行调整）。

您可以从路径构造一个简单的trie，然后将您的模式与之匹配：

import fnmatch
import glob
import os.path
from itertools import product


# Cross-Python dictionary views on the keys 
if hasattr(dict, 'viewkeys'):
    # Python 2
    def _viewkeys(d):
        return d.viewkeys()
else:
    # Python 3
    def _viewkeys(d):
        return d.keys()


def _in_trie(trie, path):
    """Determine if path is completely in trie"""
    current = trie
    for elem in path:
        try:
            current = current[elem]
        except KeyError:
            return False
    return None in current


def find_matching_paths(paths, pattern):
    """Produce a list of paths that match the pattern.

    * paths is a list of strings representing filesystem paths
    * pattern is a glob pattern as supported by the fnmatch module

    """
    if os.altsep:  # normalise
        pattern = pattern.replace(os.altsep, os.sep)
    pattern = pattern.split(os.sep)

    # build a trie out of path elements; efficiently search on prefixes
    path_trie = 
    for path in paths:
        if os.altsep:  # normalise
            path = path.replace(os.altsep, os.sep)
        _, path = os.path.splitdrive(path)
        elems = path.split(os.sep)
        current = path_trie
        for elem in elems:
            current = current.setdefault(elem, )
        current.setdefault(None, None)  # sentinel

    matching = []

    current_level = [path_trie]
    for subpattern in pattern:
        if not glob.has_magic(subpattern):
            # plain element, element must be in the trie or there are
            # 0 matches
            if not any(subpattern in d for d in current_level):
                return []
            matching.append([subpattern])
            current_level = [d[subpattern] for d in current_level if subpattern in d]
        else:
            # match all next levels in the trie that match the pattern
            matched_names = fnmatch.filter(k for d in current_level for k in d, subpattern)
            if not matched_names:
                # nothing found
                return []
            matching.append(matched_names)
            current_level = [d[n] for d in current_level for n in _viewkeys(d) & set(matched_names)]

    return [os.sep.join(p) for p in product(*matching)
            if _in_trie(path_trie, p)]

这个满嘴可以在路径上的任何地方使用 glob 快速找到匹配项：

>>> paths = ['/foo/bar/baz', '/spam/eggs/baz', '/foo/bar/bar']
>>> find_matching_paths(paths, '/foo/bar/*')
['/foo/bar/baz', '/foo/bar/bar']
>>> find_matching_paths(paths, '/*/bar/b*')
['/foo/bar/baz', '/foo/bar/bar']
>>> find_matching_paths(paths, '/*/[be]*/b*')
['/foo/bar/baz', '/foo/bar/bar', '/spam/eggs/baz']

【参考方案2】：

优秀的艺术家复制；伟大的艺术家steal。

我偷了;)

fnmatch.translate 将 glob ? 和 * 分别转换为正则表达式 . 和 .*。我调整它不。

import re

def glob2re(pat):
    """Translate a shell PATTERN to a regular expression.

    There is no way to quote meta-characters.
    """

    i, n = 0, len(pat)
    res = ''
    while i < n:
        c = pat[i]
        i = i+1
        if c == '*':
            #res = res + '.*'
            res = res + '[^/]*'
        elif c == '?':
            #res = res + '.'
            res = res + '[^/]'
        elif c == '[':
            j = i
            if j < n and pat[j] == '!':
                j = j+1
            if j < n and pat[j] == ']':
                j = j+1
            while j < n and pat[j] != ']':
                j = j+1
            if j >= n:
                res = res + '\\['
            else:
                stuff = pat[i:j].replace('\\','\\\\')
                i = j+1
                if stuff[0] == '!':
                    stuff = '^' + stuff[1:]
                elif stuff[0] == '^':
                    stuff = '\\' + stuff
                res = '%s[%s]' % (res, stuff)
        else:
            res = res + re.escape(c)
    return res + '\Z(?ms)'

这个fnmatch.filter，re.match 和 re.search 工作。

def glob_filter(names,pat):
    return (name for name in names if re.match(glob2re(pat),name))

在此页面上找到的 Glob 模式和字符串通过了测试。

pat_dict = 
            'a/b/*/f.txt': ['a/b/c/f.txt', 'a/b/q/f.txt', 'a/b/c/d/f.txt','a/b/c/d/e/f.txt'],
            '/foo/bar/*': ['/foo/bar/baz', '/spam/eggs/baz', '/foo/bar/bar'],
            '/*/bar/b*': ['/foo/bar/baz', '/foo/bar/bar'],
            '/*/[be]*/b*': ['/foo/bar/baz', '/foo/bar/bar'],
            '/foo*/bar': ['/foolicious/spamfantastic/bar', '/foolicious/bar']

        
for pat in pat_dict:
    print('pattern :\t\nstrings :\t'.format(pat,pat_dict[pat]))
    print('matched :\t\n'.format(list(glob_filter(pat_dict[pat],pat))))

【讨论】：

伟大的独家新闻！是的，将模式转换为忽略路径分隔符的模式是一个好主意。请注意，虽然它不处理 os.sep 或 os.altsep，但它应该很容易调整。

我通常只是将路径规范化为在任何处理之前先使用正斜杠。

【参考方案3】：

在 Python 3.4+ 上，您可以使用 PurePath.match。

pathlib.PurePath(path_string).match(pattern)

在 Python 3.3 或更早版本（包括 2.x）上，获取 pathlib from PyPI。

请注意，要获得独立于平台的结果（这将取决于为什么您要运行它），您需要明确声明 PurePosixPath 或 PureWindowsPath。

【讨论】：

这种方法的一个好处是，如果不需要，它不需要您指定全局语法 (**/*)。例如，如果您只是想根据文件名查找路径。

这仅适用于单个字符串。虽然有用，但它并不能完全回答 OP 的“glob 格式为 list 字符串”的问题。

找到了一种通过列表理解扩展此答案的方法，请参阅我的答案。

【参考方案4】：

虽然fnmatch.fnmatch 可直接用于检查模式是否与文件名匹配，但您也可以使用fnmatch.translate 方法从给定的fnmatch 模式中生成正则表达式：

>>> import fnmatch
>>> fnmatch.translate('*.txt')
'.*\\.txt\\Z(?ms)'

来自documenation：

fnmatch.translate(pattern)

返回转换为正则表达式的 shell 样式模式。

【参考方案5】：

没关系，我找到了。我想要fnmatch 模块。

【讨论】：

哦，等等——fnmatch 不处理路径名分段...叹息。

你能提供例子，fnmatch 不处理你的情况吗？

@BhargavRao: glob.glob() 将模式分别应用于路径元素

@MartijnPieters 是的。但是当您查看glob 的cpython 实现时，它们只是在路径sep 上拆分并递归使用fnmatch！所以使用fnmatch有什么问题是我的问题。 （基于一些假设，我的草稿答案已经准备好，只是等待 OP 澄清）

哎呀 Martijn，你已经回答了同样的问题，为此！没看到你的回答！立即丢弃我的草稿:)

【参考方案6】：

这是一个可以处理转义标点符号的 glob。它确实 not 在路径分隔符处停止。我在这里发布它是因为它与问题的标题相匹配。

在列表中使用：

rex = glob_to_re(glob_pattern)
rex = r'(?s:%s)\Z' % rex # Can match newline; match whole string.
rex = re.compile(rex)
matches = [name for name in names if rex.match(name)]

代码如下：

import re as _re

class GlobSyntaxError(SyntaxError):
    pass

def glob_to_re(pattern):
    r"""
    Given pattern, a unicode string, return the equivalent regular expression.
    Any special character * ? [ ! - ] \ can be escaped by preceding it with 
    backslash ('\') in the pattern.  Forward-slashes ('/') and escaped 
    backslashes ('\\') are treated as ordinary characters, not boundaries.

    Here is the language glob_to_re understands.
    Earlier alternatives within rules have precedence.  
        pattern = item*
        item    = '*'  |  '?'  |  '[!' set ']'  |  '[' set ']'  |  literal
        set     = element element*
        element = literal '-' literal  |  literal
        literal = '\' char  |  char other than \  [  ] and sometimes -
    glob_to_re does not understand "a,b...".
    """
    # (Note: the docstring above is r""" ... """ to preserve backslashes.)
    def expect_char(i, context):
        if i >= len(pattern):
            s = "Unfinished %s: %r, position %d." % (context, pattern, i)
            raise GlobSyntaxError(s)
    
    def literal_to_re(i, context="pattern", bad="[]"):
        if pattern[i] == '\\':
            i += 1
            expect_char(i, "backslashed literal")
        else:
            if pattern[i] in bad:
                s = "Unexpected %r in %s: %r, position %d." \
                    % (pattern[i], context, pattern, i)
                raise GlobSyntaxError(s)
        return _re.escape(pattern[i]), i + 1

    def set_to_re(i):
        assert pattern[i] == '['
        set_re = "["
        i += 1
        try:
            if pattern[i] == '!':
                set_re += '^'
                i += 1
            while True:
                lit_re, i = literal_to_re(i, "character set", bad="[-]")
                set_re += lit_re
                if pattern[i] == '-':
                    set_re += '-'
                    i += 1
                    expect_char(i, "character set range")
                    lit_re, i = literal_to_re(i, "character set range", bad="[-]")
                    set_re += lit_re
                if pattern[i] == ']':
                    return set_re + ']', i + 1
                
        except IndexError:
            expect_char(i, "character set")  # Trigger "unfinished" error.

    i = 0
    re_pat = ""
    while i < len(pattern):
        if pattern[i] == '*':
            re_pat += ".*"
            i += 1
        elif pattern[i] == '?':
            re_pat += "."
            i += 1
        elif pattern[i] == '[':
            set_re, i = set_to_re(i)
            re_pat += set_re
        else:
            lit_re, i = literal_to_re(i)
            re_pat += lit_re
    return re_pat

【参考方案7】：

不能说它的效率如何，但它比其他解决方案更简洁、更简单、更完整，并且可能更安全/可靠。

支持syntax:

* -- 匹配零个或多个字符。 **（实际上是**/ 或/**）——匹配零个或多个子目录。 ? -- 匹配一个字符。 [] -- 匹配括号内的一个字符。 [!] -- 匹配一个不在括号内的字符。 由于使用\进行转义，所以只有/可以用作路径分隔符。

操作顺序：

转义 glob 中的特殊 RE 字符。 为 escaped glob 的标记化生成 RE。 用等效的 RE 替换 转义 glob 标记。

import re
from sys import hexversion, implementation
# Support for insertion-preserving/ordered dicts became language feature in Python 3.7, but works in CPython since 3.6.
if hexversion >= 0x03070000 or (implementation.name == 'cpython' and hexversion >= 0x03060000):
    ordered_dict = dict
else:
    from collections import OrderedDict as ordered_dict

escaped_glob_tokens_to_re = ordered_dict((
    # Order of ``**/`` and ``/**`` in RE tokenization pattern doesn't matter because ``**/`` will be caught first no matter what, making ``/**`` the only option later on.
    # W/o leading or trailing ``/`` two consecutive asterisks will be treated as literals.
    ('/\*\*', '(?:/.+?)*'), # Edge-case #1. Catches recursive globs in the middle of path. Requires edge case #2 handled after this case.
    ('\*\*/', '(?:^.+?/)*'), # Edge-case #2. Catches recursive globs at the start of path. Requires edge case #1 handled before this case. ``^`` is used to ensure proper location for ``**/``.
    ('\*', '[^/]*?'), # ``[^/]*?`` is used to ensure that ``*`` won't match subdirs, as with naive ``.*?`` solution.
    ('\?', '.'),
    ('\[\*\]', '\*'), # Escaped special glob character.
    ('\[\?\]', '\?'), # Escaped special glob character.
    ('\[!', '[^'), # Requires ordered dict, so that ``\[!`` preceded ``\[`` in RE pattern. Needed mostly to differentiate between ``!`` used within character class ``[]`` and outside of it, to avoid faulty conversion.
    ('\[', '['),
    ('\]', ']'),
))

escaped_glob_replacement = re.compile('(%s)' % '|'.join(escaped_glob_tokens_to_re).replace('\\', '\\\\\\'))

def glob_to_re(pattern):
    return escaped_glob_replacement.sub(lambda match: escaped_glob_tokens_to_re[match.group(0)], re.escape(pattern))

if __name__ == '__main__':
    validity_paths_globs = (
        (True, 'foo.py', 'foo.py'),
        (True, 'foo.py', 'fo[o].py'),
        (True, 'fob.py', 'fo[!o].py'),
        (True, '*foo.py', '[*]foo.py'),
        (True, 'foo.py', '**/foo.py'),
        (True, 'baz/duck/bar/bam/quack/foo.py', '**/bar/**/foo.py'),
        (True, 'bar/foo.py', '**/foo.py'),
        (True, 'bar/baz/foo.py', 'bar/**'),
        (False, 'bar/baz/foo.py', 'bar/*'),
        (False, 'bar/baz/foo.py', 'bar**/foo.py'),
        (True, 'bar/baz/foo.py', 'bar/**/foo.py'),
        (True, 'bar/baz/wut/foo.py', 'bar/**/foo.py'),
    )
    results = []
    for seg in validity_paths_globs:
        valid, path, glob_pat = seg
        print('valid:', valid)
        print('path:', path)
        print('glob pattern:', glob_pat)
        re_pat = glob_to_re(glob_pat)
        print('RE pattern:', re_pat)
        match = re.fullmatch(re_pat, path)
        print('match:', match)
        result = bool(match) == valid
        results.append(result)
        print('result was expected:', result)
        print('-'*79)
    print('all results were expected:', all(results))
    print('='*79)

【参考方案8】：

@Veedrac PurePath.match 答案的扩展，可应用于字符串列表：

# Python 3.4+
from pathlib import Path

path_list = ["foo/bar.txt", "spam/bar.txt", "foo/eggs.txt"]
# convert string to pathlib.PosixPath / .WindowsPath, then apply PurePath.match to list
print([p for p in path_list if Path(p).match("ba*")])  # "*ba*" also works
# output: ['foo/bar.txt', 'spam/bar.txt']

print([p for p in path_list if Path(p).match("*o/ba*")])
# output: ['foo/bar.txt']

最好使用pathlib.Path() 而不是pathlib.PurePath()，因为这样您就不必担心底层文件系统。

【参考方案9】：

我想添加对递归 glob 模式的支持，即 things/**/*.py 并具有相对路径匹配，因此 example*.py 与 folder/example_stuff.py 不匹配。

这是我的方法：

from os import path
import re

def recursive_glob_filter(files, glob):
    # Convert to regex and add start of line match
    pattern_re = '^' + fnmatch_translate(glob)

    # fnmatch does not escape path separators so escape them
    if path.sep in pattern_re and not r'\'.format(path.sep) in pattern_re:
        pattern_re = pattern_re.replace('/', r'\/')

    # Replace `*` with one that ignores path separators
    sep_respecting_wildcard = '[^\]*'.format(path.sep)
    pattern_re = pattern_re.replace('.*', sep_respecting_wildcard)

    # And now for `**` we have `[^\/]*[^\/]*`, so replace that with `.*`
    # to match all patterns in-between
    pattern_re = pattern_re.replace(2 * sep_respecting_wildcard, '.*')
    compiled_re = re.compile(pattern_re)
    return filter(compiled_re.search, files)