AWS Glue 作业内存不足

Posted 2023-02-16

【中文标题】AWS Glue 作业内存不足

【英文标题】：AWS Glue job running out of memory

【发布时间】：2021-10-23 10:25:27

【问题描述】：

我正在尝试运行 AWS Glue 作业（类型为 G.1X，有 15 个工人）。此粘合作业应生成具有以下架构的 spark 数据帧：

+-----------+-----------+-------+------------------+----------+--------+--------+--------+--------+
|client_id_x|client_id_y|   dist|              time|      date|   lat_x|   lng_x|   lat_y|   lng_y|
+-----------+-----------+-------+------------------+----------+--------+--------+--------+--------+
| 7120061434| 7120110123|84899.3| 70.67166666666667|2021-08-19|-23.6595|-52.6062|-23.7626|-53.3129|
| 7120061434| 7120094771|18679.8| 19.68166666666667|2021-08-19|-23.6595|-52.6062|-23.5373| -52.585|

我可以在本地生成多达 100 万行。我希望使用 AWS Glue 生成更多，但是，我什至无法生成 600k。 Glue 作业显示以下错误：

Driver logs

Aug 22, 2021, 6:04:36 PM Pending execution
Aug 22, 2021, 6:05:50 PM 21/08/22 23:05:50 WARN ApacheUtils: NoSuchMethodException was thrown when disabling normalizeUri. This indicates you are using an old version (< 4.5.8) of Apache http client. It is recommended to use http client version >= 4.5.9 to avoid the breaking change introduced in apache client 4.5.7 and the latency in exception handling. See https://github.com/aws/aws-sdk-java/issues/1919 for more information
Aug 22, 2021, 6:05:54 PM 21/08/22 23:05:54 INFO Utils: Successfully started service 'sparkDriver' on port 37887.
Aug 22, 2021, 6:05:55 PM 21/08/22 23:05:55 INFO GlueContext: GlueMetrics configured and enabled
Aug 22, 2021, 6:05:58 PM 21/08/22 23:05:58 INFO GlueContext: getCatalogSource: catalogId: null, nameSpace: cia_clients, tableName: clients, isRegisteredWithLF: false, isGoverned: false, isRowFilterEnabled: falseuseAdvancedFiltering: false
Aug 22, 2021, 6:05:58 PM 21/08/22 23:05:58 INFO GlueContext: getCatalogSource: transactionId: <not-specified> asOfTime: <not-specified> catalogPartitionIndexPredicate: <not-specified>
Aug 22, 2021, 6:05:58 PM 21/08/22 23:05:58 INFO GlueContext: classification parquet
Aug 22, 2021, 6:05:58 PM 21/08/22 23:05:58 INFO GlueContext: No of partitions from catalog are 0. consider catalogPartitionPredicate to reduce the number of partitions to scan through
Aug 22, 2021, 6:05:58 PM 21/08/22 23:05:58 INFO GlueContext: Glue secret manager integration: secretId is not provided.
Aug 22, 2021, 6:05:59 PM 21/08/22 23:05:59 WARN EC2MetadataUtils: Unable to retrieve the requested metadata (/latest/user-data/). The requested metadata is not found at http://169.254.169.254/latest/user-data/
Aug 22, 2021, 6:05:59 PM 21/08/22 23:05:59 ERROR UserData: Error encountered while try to get user data
Aug 22, 2021, 6:06:01 PM 21/08/22 23:06:01 INFO DAGScheduler: Got job 0 (resolveRelation at DataSource.scala:720) with 1 output partitions
Aug 22, 2021, 6:06:16 PM 21/08/22 23:06:16 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Aug 22, 2021, 6:06:31 PM 21/08/22 23:06:31 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Aug 22, 2021, 6:06:46 PM 21/08/22 23:06:46 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Aug 22, 2021, 6:07:01 PM 21/08/22 23:07:01 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Aug 22, 2021, 6:07:10 PM 21/08/22 23:07:10 INFO DAGScheduler: Job 0 finished: resolveRelation at DataSource.scala:720, took 69.248217 s
Aug 22, 2021, 6:07:29 PM 21/08/22 23:07:29 INFO DAGScheduler: Job 1 finished: run at ThreadPoolExecutor.java:1149, took 15.215954 s
Aug 22, 2021, 6:07:30 PM 21/08/22 23:07:30 INFO DAGScheduler: Got job 2 (count at NativeMethodAccessorImpl.java:0) with 1 output partitions
Aug 22, 2021, 6:07:35 PM 21/08/22 23:07:35 WARN TaskSetManager: Lost task 4.0 in stage 11.0 (TID 647, 172.35.108.199, executor 4): org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 377, in main process() File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 372, in process serializer.dump_stream(func(split_index, iterator), outfile) File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 286, in dump_stream for series in iterator: File "<string>", line 1, in <lambda> File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 113, in wrapped result = f(pd.concat(value_series, axis=1)) File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/util.py", line 99, in wrapper return f(*args, **kwargs) File "/tmp/test3", line 116, in osrm_map File "/home/spark/.local/lib/python3.7/site-packages/requests/models.py", line 898, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib64/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:172) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:122) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage19.agg_doAggregateWithoutKey_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage19.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Aug 22, 2021, 6:07:38 PM 21/08/22 23:07:38 ERROR TaskSetManager: Task 1 in stage 11.0 failed 4 times; aborting job
Aug 22, 2021, 6:07:38 PM 21/08/22 23:07:38 INFO DAGScheduler: ShuffleMapStage 11 (count at NativeMethodAccessorImpl.java:0) failed in 6.809 s due to Job aborted due to stage failure: Task 1 in stage 11.0 failed 4 times, most recent failure: Lost task 1.3 in stage 11.0 (TID 735, 172.35.108.199, executor 4): org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 377, in main process() File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 372, in process serializer.dump_stream(func(split_index, iterator), outfile) File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 286, in dump_stream for series in iterator: File "<string>", line 1, in <lambda> File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 113, in wrapped result = f(pd.concat(value_series, axis=1)) File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/util.py", line 99, in wrapper return f(*args, **kwargs) File "/tmp/test3", line 116, in osrm_map File "/home/spark/.local/lib/python3.7/site-packages/requests/models.py", line 898, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib64/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:172) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:122) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage19.agg_doAggregateWithoutKey_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage19.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Driver stacktrace:
Aug 22, 2021, 6:07:38 PM 21/08/22 23:07:38 INFO DAGScheduler: Job 2 failed: count at NativeMethodAccessorImpl.java:0, took 8.620489 s
Aug 22, 2021, 6:07:38 PM 21/08/22 23:07:38 WARN TaskSetManager: Lost task 4.3 in stage 11.0 (TID 736, 172.35.108.199, executor 4): TaskKilled (Stage cancelled)
Aug 22, 2021, 6:07:38 PM 21/08/22 23:07:38 ERROR ProcessLauncher: Error from Python:Traceback (most recent call last): File "/tmp/test3", line 163, in <module> print(cost_matrix.count()) File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 522, in count return int(self._jdf.count()) File "/opt/amazon/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__ answer, self.gateway_client, self.target_id, self.name) File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco return f(*a, **kw) File "/opt/amazon/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value format(target_id, ".", name), value) py4j.protocol.Py4JJavaError: An error occurred while calling o149.count. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 11.0 failed 4 times, most recent failure: Lost task 1.3 in stage 11.0 (TID 735, 172.35.108.199, executor 4): org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 377, in main process() File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 372, in process serializer.dump_stream(func(split_index, iterator), outfile) File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 286, in dump_stream for series in iterator: File "<string>", line 1, in <lambda> File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 113, in wrapped result = f(pd.concat(value_series, axis=1)) File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/util.py", line 99, in wrapper return f(*args, **kwargs) File "/tmp/test3", line 116, in osrm_map File "/home/spark/.local/lib/python3.7/site-packages/requests/models.py", line 898, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib64/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:172) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:122) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage19.agg_doAggregateWithoutKey_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage19.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126) at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:363) at org.apache.spark.rdd.RDD.collect(RDD.scala:944) at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:299) at org.apache.spark.sql.Dataset$$anonfun$count$1.apply(Dataset.scala:2830) at org.apache.spark.sql.Dataset$$anonfun$count$1.apply(Dataset.scala:2829) at org.apache.spark.sql.Dataset$$anonfun$53.apply(Dataset.scala:3364) at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73) at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3363) at org.apache.spark.sql.Dataset.count(Dataset.scala:2829) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 377, in main process() File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 372, in process serializer.dump_stream(func(split_index, iterator), outfile) File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 286, in dump_stream for series in iterator: File "<string>", line 1, in <lambda> File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 113, in wrapped result = f(pd.concat(value_series, axis=1)) File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/util.py", line 99, in wrapper return f(*args, **kwargs) File "/tmp/test3", line 116, in osrm_map File "/home/spark/.local/lib/python3.7/site-packages/requests/models.py", line 898, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib64/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:172) at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:122) at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage19.agg_doAggregateWithoutKey_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage19.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more
Aug 22, 2021, 6:07:38 PM 21/08/22 23:07:38 WARN TaskSetManager: Lost task 0.3 in stage 11.0 (TID 737, 172.35.108.199, executor 4): TaskKilled (Stage cancelled) 

某些台词引起了我的注意。有几行显示此错误： WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources。根据this question的说法，是集群资源不足造成的（因为worker没有启动）。

我还没有在 AWS Glue 上找到解决此问题的具体答案。如果有人遇到同样的问题，我将非常感谢您的帮助！

【问题讨论】：

源中的平均文件大小和文件总数是多少？

目前我的最终 spark 数据框应该有 640,000 行。使用 490,000 行的数据框似乎可以正常工作，但使用 640,000 行时，我总是会收到上述错误。

你能用脚本更新你的问题吗？

【参考方案1】：

我遇到了类似的问题，因为它太多了，所以无法完成 ETL 任务。我发现我可以从这个链接https://aws.amazon.com/blogs/big-data/best-practices-to-scale-apache-spark-jobs-and-partition-data-with-aws-glue/的文档中调整我的配置@

在我的情况下，我可以更改工作器的类型以完成我的任务，而不会引发关于我的 Spark 任务的 RAM 太少的异常。

还有其他答案和 cmets 应该明确地为粘合作业设置配置键值，但这对我不起作用，也不会随着工人的扩展而扩展。