App Engine 批量数据上传 URLError: "Connection reset by peer"

Posted

技术标签:

【中文标题】App Engine 批量数据上传 URLError: "Connection reset by peer"【英文标题】:App Engine Bulk Data Upload URLError: "Connection reset by peer" 【发布时间】:2010-08-08 16:10:42 【问题描述】:

我正在尝试使用 appcfg.py 将一些记录上传到我的本地数据存储中

实际上只插入了少量记录,我得到以下输出(有大量错误):

       $ appcfg.py upload_data --config_file=bulkloader.yaml --filename=/output.csv --kind=AutoCompleteIndex --url=http://localhost:8084/remote_api .
        Application: theapp1; version: dev.
        Uploading data records.
        [INFO    ] Logging to bulkloader-log-20100808.170223
        [INFO    ] Throttling transfers:
        [INFO    ] Bandwidth: 250000 bytes/second
        [INFO    ] HTTP connections: 8/second
        [INFO    ] Entities inserted/fetched/modified: 20/second
        [INFO    ] Batch Size: 10
        [INFO    ] Opening database: bulkloader-progress-20100808.170223.sql3
        Please enter login credentials for localhost
        Email: kjuyg
        Password for kjuyg: 
        [INFO    ] Connecting to localhost:8084/remote_api
        [INFO    ] Starting import; maximum 10 entities per post
        .......................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        ................................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .............[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        ................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .......................................................................................................................................................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put

.... etc .....
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C..[INFO    ] An error occurred. Shutting down...
        ..........
        [INFO    ] 19580 entites total, 0 previously transferred
        [INFO    ] 18570 entities (3593152 bytes) transferred in 225.1 seconds
        [INFO    ] Some entities not successfully transferred

知道可能是什么问题吗?

【问题讨论】:

【参考方案1】:

我在我的 Mac 上遇到了完全相同的问题。我通过创建代理解决了这个问题,将以下内容放入我的 httpd-vhost.conf 文件(在 etc/apache2/extra/ 中):

<Location /remote_api>
  ProxyPass http://localhost:8888/remote_api
</Location>

假设您的测试服务器在端口 8888 上运行。

然后我运行:

sudo apachectl restart

重启apache服务器。

现在这个命令(“Survey”是数据项之一):

/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/appcfg.py upload_data /
--kind=Survey --file=Survey.csv /
--url=http://localhost/remote_api --email=test@example.com /
--config_file=config.yml 

给出这个输出:

[INFO    ] Logging to bulkloader-log-20121114.100837
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
Password for test@example.com: 
[INFO    ] Opening database: bulkloader-progress-20121114.100837.sql3
[INFO    ] Connecting to localhost/remote_api
[INFO    ] Starting import; maximum 10 entities per post
....
[INFO    ] 36 entities total, 0 previously transferred
[INFO    ] 36 entities (15402 bytes) transferred in 1.7 seconds
[INFO    ] All entities successfully transferred

应该如此。

【讨论】:

【参考方案2】:

查看您的服务器日志。您是否收到超出堆大小或 OutOfMemory 异常?如果您的上传量很大,这将在本地开发服务器上发生,因为开发服务器将数据存储保存在主内存中。服务器将挂起一段时间,然后您将重置连接。我的开发服务器内存不足,只有 10MB 的数据存储数据,因为它使用大量额外的 RAM 来组织它和存储元数据。您可以使用 -Xmx 开关增加堆的大小。

【讨论】:

我们应该去哪里设置-Xmx开关?

以上是关于App Engine 批量数据上传 URLError: "Connection reset by peer"的主要内容,如果未能解决你的问题,请参考以下文章

GWT与Google App Engine-上传文件

如何在 Google App Engine 的 Datastore 模型中获取行数?

App-Engine (Java) 文件上传

Google App Engine 和 Amazon S3 文件上传

上传属性 Google App Engine - app.yaml

与 Google App Engine 一起上传图片