如何在sagemaker笔记本的S3存储桶中打开模型tarfile？

Posted 2021-04-30

我知道从S3存储桶将.csv文件加载到sagemaker笔记本中非常简单，但我想加载存储在S3存储桶中的model.tar.gz文件。我尝试执行以下操作

import botocore 
import sagemaker
from sagemaker import get_execution_role
from sagemaker.predictor import csv_serializer
import boto3

sm_client = boto3.client(service_name='sagemaker')
runtime_sm_client = boto3.client(service_name='sagemaker-runtime')

s3 = boto3.resource('s3')
s3_client = boto3.client('s3')

sagemaker_session = sagemaker.Session()
role = get_execution_role()

ACCOUNT_ID  = boto3.client('sts').get_caller_identity()['Account']
REGION      = boto3.Session().region_name
BUCKET      = 'sagemaker.prismade.net'
data_key    = 'DEMO_MME_ANN/multi_model_artifacts/axel.tar.gz'
loc = 's3://{}/{}'.format(BUCKET, data_key)
print(loc)
with tarfile.open(loc) as tar:
    tar.extractall(path='.')

我收到以下错误：

--------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-215-bfdddac71b95> in <module>()
     20 loc = 's3://{}/{}'.format(BUCKET, data_key)
     21 print(loc)
---> 22 with tarfile.open(loc) as tar:
     23     tar.extractall(path='.')

~/anaconda3/envs/python3/lib/python3.6/tarfile.py in open(cls, name, mode, fileobj, bufsize, **kwargs)
   1567                     saved_pos = fileobj.tell()
   1568                 try:
-> 1569                     return func(name, "r", fileobj, **kwargs)
   1570                 except (ReadError, CompressionError):
   1571                     if fileobj is not None:

~/anaconda3/envs/python3/lib/python3.6/tarfile.py in gzopen(cls, name, mode, fileobj, compresslevel, **kwargs)
   1632 
   1633         try:
-> 1634             fileobj = gzip.GzipFile(name, mode + "b", compresslevel, fileobj)
   1635         except OSError:
   1636             if fileobj is not None and mode == 'r':

~/anaconda3/envs/python3/lib/python3.6/gzip.py in __init__(self, filename, mode, compresslevel, fileobj, mtime)
    161             mode += 'b'
    162         if fileobj is None:
--> 163             fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
    164         if filename is None:
    165             filename = getattr(fileobj, 'name', '')

FileNotFoundError: [Errno 2] No such file or directory: 's3://sagemaker.prismade.net/DEMO_MME_ANN/multi_model_artifacts/axel.tar.gz'

这里的错误是什么，我该怎么做？

答案

并非每个旨在与文件系统一起工作的python库（在此示例中为tarfile.open）都知道如何将S3对象作为文件读取。

解决它的简单方法是先将对象作为文件复制到本地文件系统中。

import boto3

s3 = boto3.client('s3')
s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME')