当我们尝试在 Python 中使用 pysftp 从 SFTP 串行下载 50 多个文件时，下载失败并显示“身份验证失败”？

Posted 2023-02-18

【中文标题】当我们尝试在 Python 中使用 pysftp 从 SFTP 串行下载 50 多个文件时，下载失败并显示“身份验证失败”？

【英文标题】：Download fails with "Authentication failed" when we try to download 50+ files from SFTP serially using pysftp in Python?

【发布时间】：2021-11-21 00:46:55

【问题描述】：

for remote_path in list_of_stfp_paths:
    with pysftp.Connection(HOSTNAME, username=USERNAME, password=PASSWORD) as sftp:
        sftp.get(remote_path, str(local_path))

    #checks distinct count of a column for the csv downloaded, deletes it later
    df = pd.read_csv(str(local_path))
    print(df['taken_time'].value_counts())
    os.remove(str(local_path))

我使用的代码在上面。它只是在具有多个远程路径的 for 循环中运行。 有时，它会完成。有时，我会收到一条错误提示

异常：身份验证失败。

【参考方案1】：

不要为每个文件重新连接。仅循环下载，而不是连接：

with pysftp.Connection(HOSTNAME, username=USERNAME, password=PASSWORD) as sftp:
    for remote_path in list_of_stfp_paths:
        sftp.get(remote_path, str(local_path))

        #checks distinct count of a column for the csv downloaded, deletes it later
        df = pd.read_csv(str(local_path))
        print(df['taken_time'].value_counts())
        os.remove(str(local_path))

---

请注意，您甚至不必将文件下载到本地磁盘，只需直接从 SFTP 服务器读取它们即可：

with pysftp.Connection(HOSTNAME, username=USERNAME, password=PASSWORD) as sftp:
    for remote_path in list_of_stfp_paths:
        with sftp.open(remote_path) as f:
            f.prefetch()
            #checks distinct count of a column for the csv
            df = pd.read_csv(f)
            print(df['taken_time'].value_counts())

它甚至可能更快，因为它允许下载和解析并行进行，而不是按顺序进行。见Read CSV/Excel files from SFTP file, make some changes in those files using Pandas, and save back