看看spark的源码

Posted 2021-01-11 yangjiming

1. IntelliJ IDEA安装plugins,加入scala插件和SBT插件

2. 下载spark的源代码，下载地址 https://github.com/apache/spark，可以使用git下载或者下载zip包

3. 导入项目，开启 IntelliJ 之后选择 Import Project，而后选择 Spark 源代码，并将其导入为 SBT 项目，在之后的过程中 SBT 过程将会自动进行，下载作为依赖的 jar 包。

4. 源代码已经准备好了。开干吧

   private def schedule(): Unit = {
       if (state != RecoveryState.ALIVE) {
         return
       }
       // Drivers take strict precedence over executors
       val shuffledAliveWorkers = Random.shuffle(workers.toSeq.filter(_.state == WorkerState.ALIVE))
       val numWorkersAlive = shuffledAliveWorkers.size
       var curPos = 0
       for (driver <- waitingDrivers.toList) { // iterate over a copy of waitingDrivers
         // We assign workers to each waiting driver in a round-robin fashion. For each driver, we
         // start from the last worker that was assigned a driver, and continue onwards until we have
         // explored all alive workers.
         var launched = false
         var numWorkersVisited = 0
         while (numWorkersVisited < numWorkersAlive && !launched) {
           val worker = shuffledAliveWorkers(curPos)
           numWorkersVisited += 1
           if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= driver.desc.cores) {
             launchDriver(worker, driver)
             waitingDrivers -= driver
             launched = true
           }
           curPos = (curPos + 1) % numWorkersAlive
         }
       }
       startExecutorsOnWorkers()
     }