如何在将文件上传到s3时处理互联网不可用？

Posted 2021-05-07

我已经在python中编写了调度程序，以便在特定时间将文件上传到AWS S3（使用调度程序）。

如果当时没有互联网连接，我将获得socket.gaierror: [Errno -3] Temporary failure in name resolution例外。

这会打破总流量，因此我需要重新启动程序。

如何以优雅的方式处理这种情况，以便当互联网恢复我的文件时，开始上传过程。

以下是我的调度程序作业代码：

def job():
    serialNo = getserial()
    print('schedule working.......')
    yesterday_date=date.today() - timedelta(1)

    # one day file path 
    file_path="/home/pi/Desktop/bme280_elf_new/reading/"+str(yesterday_date)+".csv" 

    # file move are upload file from cloud here
    AWS_ACCESS_KEY_ID = 'AWS_ACCESS_KEY_ID'
    AWS_SECRET_ACCESS_KEY = 'AWS_SECRET_ACCESS_KEY'
    Bucketname = 'Bucketname' 

    conn = boto.s3.connect_to_region('us-east-1',
       aws_access_key_id=AWS_ACCESS_KEY_ID,
       aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
       is_secure=True,               # uncomment if you are not using ssl
       calling_format = boto.s3.connection.OrdinaryCallingFormat(),
       )

    bucket = conn.get_bucket(Bucketname)

    print ('Uploading %s to Amazon S3 bucket %s' %
           (file_path, Bucketname))

    def percent_cb(complete, total):
        sys.stdout.write('.')
        sys.stdout.flush()

    k = Key(bucket)
    k.key = str(serialNo)+'-'+str(yesterday_date)+'.csv'
    k.set_contents_from_filename(file_path,
    cb=percent_cb, num_cb=10)

启动调度程序，

schedule.every().day.at("00:10").do(job) # run at midnight 00:10

提前致谢。

答案

它很简单，你可以在urllib2的帮助下实现它。这里是sample.Note urllib2仅支持python-2.x.

检查互联网连接

import urllib2

def is_connected():
        try:
            url = "http://www.google.com"
            urllib2.urlopen(url)
            return True
        except urllib2.URLError:
            pass
        return False

文件上传功能：

def fileUploadJob(BUCKET_NAME,AWS_REGION):
serialNo = getserial()
yesterday_date=date.today() - timedelta(1)

while True:
    if is_connected():
        conn = boto.s3.connect_to_region(<Aws-region>,<Access_key>
              ,<secert-key>,
            is_secure=True,
            calling_format = boto.s3.connection.OrdinaryCallingFormat(),
            )
        bucket = conn.get_bucket(BUCKET_NAME)

        FILE_PATH="/home/pi/Desktop/bme280_elf_new/reading/"+
                 str(yesterday_date)+".csv" 

        print ('Uploading %s to Amazon S3 bucket %s' % 
        (FILE_PATH, BUCKET_NAME))

        def percent_cb(complete, total):
            sys.stdout.write('.')
            sys.stdout.flush()


        k = Key(bucket)
        k.key = str(serialNo)+'-'+str(yesterday_date)+'.csv'
        k.set_contents_from_filename(FILE_PATH,
            cb=percent_cb, num_cb=10)
        time.sleep(10)

        break
    else:
        time.sleep(5)