下载几个文件后 Azure Blob 文件下载失败

Posted 2023-03-28

【中文标题】下载几个文件后 Azure Blob 文件下载失败

【英文标题】：Azure blob file download failed after downloading a few files

【发布时间】：2021-07-02 19:23:04

【问题描述】：

我尝试使用 BlobClient.downloadToFile(dest) 下载 blob 文件，但下载几个文件后失败。我在尝试从 Azure Blob 存储下载 3GB 单个文件时遇到了同样的问题。

package com.blob.download;
import com.azure.storage.blob.BlobContainerClient;
import com.azure.storage.blob.BlobServiceClient;
import com.azure.storage.blob.BlobServiceClientBuilder;
import com.azure.storage.blob.models.BlobItem;
import com.azure.storage.blob.models.ListBlobsOptions;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Iterator;
public class BlobDownload 
    public static void main(String args[])
        BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
        //Specify storage uri
        String storageBaseUri = "";
        ///Specify connection string
        String connectionString = "";
        //Specify blob list prefix
        String key = "";
        builder.connectionString(connectionString);
        BlobServiceClient client = builder.buildClient();
        if (!storageBaseUri.startsWith(client.getAccountUrl())) 
            throw new IllegalArgumentException(
                    "The given credential can not be used for the specified container.");
        
        String containerName = storageBaseUri.replace(client.getAccountUrl() + "/", "");
        BlobContainerClient blobContainerClient = client.getBlobContainerClient(containerName);
        System.out.println("Downloading " + blobContainerClient.getBlobContainerName() + "/" + key);
        Iterator<BlobItem> keyspaceBlobs =
                blobContainerClient.listBlobs(new ListBlobsOptions().setPrefix(key + "/"), null).iterator();
        while (keyspaceBlobs.hasNext())
            BlobItem blob = keyspaceBlobs.next();
            Path destFile = Paths.get("/opt/data/", blob.getName());
            try 
                Files.createDirectories(destFile.getParent());
             catch (IOException e) 
                System.out.println(e);
            
            blobContainerClient
                    .getBlobClient(blob.getName())
                    .downloadToFile(destFile.toString());
            System.out.println("Download file succeeded : " + destFile.toString());
        
    

依赖关系

nettyTcNativeVersion = '2.0.7.Final'
compile group: 'com.azure', name: 'azure-storage-blob', version: '12.4.0'
compile group: 'com.azure', name: 'azure-identity', version: '1.0.6'

我收到以下错误消息

Exception in thread "main" reactor.core.Exceptions$ReactiveException: java.io.IOException: Connection reset by peer
    at reactor.core.Exceptions.propagate(Exceptions.java:393)
    at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:97)
    at reactor.core.publisher.Mono.block(Mono.java:1678)
    at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:99)
    at com.azure.storage.blob.specialized.BlobClientBase.downloadToFileWithResponse(BlobClientBase.java:563)
    at com.azure.storage.blob.specialized.BlobClientBase.downloadToFile(BlobClientBase.java:488)
    at com.azure.storage.blob.specialized.BlobClientBase.downloadToFile(BlobClientBase.java:457)
    at com.blob.download.BlobDownload.main(BlobDownload.java:57)
    Suppressed: java.lang.Exception: #block terminated with an error
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
        ... 6 more
Caused by: java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:192)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
    at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:247)
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1147)
    at io.netty.channel.socket.nio.NiosocketChannel.doReadBytes(NioSocketChannel.java:347)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
    at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

如果我的实现中有任何错误的用法，请您指导我吗？

【参考方案1】：

我做了一个测试。你的代码基本没问题。

但是下载了几个文件就失败了。

您下载了多少文件？我使用你的代码，它似乎工作正常。

我在尝试下载 3GB 的单个文件时遇到了同样的问题 Azure Blob 存储。

根据您的代码，3gb 会导致问题。您需要分块下载文件，以防止握手超时问题。

【讨论】：

> 你下载了多少文件？以下是测试结果 * Azure blob 文件下载工作正常，1200 个文件（总文件大小为 52.32 MiB） * Azure blob 文件下载失败，单个 3GB 文件（java.io.IOException：对等连接重置） * Azure blob 文件下载下载 251 个文件后失败（167 Mb 是单个文件的最大大小，总下载文件大小为 3.6G） * 下载 5 个文件后 Azure blob 文件下载失败（每个文件大小为 167 Mb）