sbt总是重新编译CI中的完整项目，即使使用缓存？

Posted

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了sbt总是重新编译CI中的完整项目，即使使用缓存？相关的知识，希望对你有一定的参考价值。

我正在努力将SBT用于这个基本工作流程的CI流程：

  1. 编译测试
  2. 缓存~/.sbt和~/.ivy2/cache
  3. 缓存我项目中的所有target目录

在后续步骤中：

  1. 恢复~/.sbt和~/.ivy2/cache
  2. 恢复完整项目，包括以前生成的包含target文件和相同源代码的.class目录（它应该是相同的结帐）
  3. 通过sbt test运行测试

100％的时间，sbt test重新编译整个项目。我想理解或调试为什么会出现这种情况，因为自上次编译以来没有任何变化（好吧，什么都不应该改变，那么是什么导致它相信某些东西？）

我目前正在使用带有码头执行器的circleci。这意味着有一个新的docker实例，从同一个图像，运行每一步，虽然我希望缓存来解决这个问题。

.circleci/config.yml的相关部分（如果你不使用圆圈，这应该仍然是可以理解的;我已经注释了我能做到的）：

---
version: 2

jobs:
  # compile and cache compilation
  test-compile:
    working_directory: /home/circleci/myteam/myproj
    docker:
      - image: myorg/myimage:sbt-1.2.8
    steps:
      # the directory to be persisted (cached/restored) to the next step
      - attach_workspace:
          at: /home/circleci/myteam
      # git pull to /home/circleci/myteam/myproj
      - checkout
      - restore_cache:
          # look for a pre-existing set of ~/.ivy2/cache, ~/.sbt dirs 
          # from a prior build
          keys:
            - sbt-artifacts-{{ checksum "project/build.properties"}}-{{ checksum "build.sbt" }}-{{ checksum "project/Dependencies.scala" }}-{{ checksum "project/plugins.sbt" }}-{{ .Branch }}
      - restore_cache:
          # look for pre-existing set of 'target' dirs from a prior build
          keys:
            - build-{{ checksum "project/build.properties"}}-{{ checksum "build.sbt" }}-{{ checksum "project/Dependencies.scala" }}-{{ checksum "project/plugins.sbt" }}-{{ .Branch }}
      - run:
          # the compile step
          working_directory: /home/circleci/myteam/myproj
          command: sbt test:compile
      # per: https://www.scala-sbt.org/1.0/docs/Travis-CI-with-sbt.html
      # Cleanup the cached directories to avoid unnecessary cache updates
      - run:
          working_directory: /home/circleci
          command: |
            rm -rf /home/circleci/.ivy2/.sbt.ivy.lock
            find /home/circleci/.ivy2/cache -name "ivydata-*.properties" -print -delete
            find /home/circleci/.sbt -name "*.lock" -print -delete
      - save_cache:
          # cache ~/.ivy2/cache and ~/.sbt for subsequent builds
          key: sbt-artifacts-{{ checksum "project/build.properties"}}-{{ checksum "build.sbt" }}-{{ checksum "project/Dependencies.scala" }}-{{ checksum "project/plugins.sbt" }}-{{ .Branch }}-{{ .Revision }}
          paths:
            - /home/circleci/.ivy2/cache
            - /home/circleci/.sbt
      - save_cache:
          # cache the `target` dirs for subsequenet builds
          key: build-{{ checksum "project/build.properties"}}-{{ checksum "build.sbt" }}-{{ checksum "project/Dependencies.scala" }}-{{ checksum "project/plugins.sbt" }}-{{ .Branch }}-{{ .Revision }}
          paths:
            - /home/circleci/myteam/myproj/target
            - /home/circleci/myteam/myproj/project/target
            - /home/circleci/myteam/myproj/project/project/target
      # in circle, a 'workflow' undergoes several jobs, this first one 
      # is 'compile', the next will run the tests (see next 'job' section
      # 'test-run' below). 
      # 'persist to workspace' takes any files from this job and ensures 
      # they 'come with' the workspace to the next job in the workflow
      - persist_to_workspace:
          root: /home/circleci/myteam
          # bring the git checkout, including all target dirs
          paths:
            - myproj
      - persist_to_workspace:
          root: /home/circleci
          # bring the big stuff
          paths:
            - .ivy2/cache
            - .sbt

  # actually runs the tests compiled in the previous job
  test-run:
    environment:
      SBT_OPTS: -XX:+UseConcMarkSweepGC -XX:+UnlockDiagnosticVMOptions  -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Duser.timezone=Etc/UTC -Duser.language=en -Duser.country=US
    docker:
      # run tests in the same image as before, but technically 
      # a different instance
      - image: myorg/myimage:sbt-1.2.8
    steps:
      # bring over all files 'persist_to_workspace' in the last job
      - attach_workspace:
          at: /home/circleci/myteam
      # restore ~/.sbt and ~/.ivy2/cache via `mv` from the workspace 
      # back to the home dir
      - run:
          working_directory: /home/circleci/myteam
          command: |
            [[ ! -d /home/circleci/.ivy2 ]] && mkdir /home/circleci/.ivy2

            for d in .ivy2/cache .sbt; do
              [[ -d "/home/circleci/$d" ]] && rm -rf "/home/circleci/$d"
              if [ -d "$d"  ]; then
                mv -v "$d" "/home/circleci/$d"
              else
                echo "$d does not exist" >&2
                ls -la . >&2
                exit 1
              fi
            done
      - run:
          # run the tests, already compiled
          # note: recompiles everything every time!
          working_directory: /home/circleci/myteam/myproj
          command: sbt test
          no_output_timeout: 3900s

workflows:
  version: 2
  build-and-test:
    jobs:
      - test-compile
      - test-run:
          requires:
            - test-compile

第二阶段的输出通常如下所示：

#!/bin/bash -eo pipefail
sbt test

[info] Loading settings for project myproj-build from native-packager.sbt,plugins.sbt ...
[info] Loading project definition from /home/circleci/myorg/myproj/project
[info] Updating ProjectRef(uri("file:/home/circleci/myorg/myproj/project/"), "myproj-build")...
[info] Done updating.
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Compiling 1 Scala source to /home/circleci/myorg/myproj/project/target/scala-2.12/sbt-1.0/classes ...
[info] Done compiling.
[info] Loading settings for project root from build.sbt ...
[info] Set current project to Piranha (in build file:/home/circleci/myorg/myproj/)
[info] Compiling 1026 Scala sources to /home/circleci/myorg/myproj/target/scala-2.12/classes ...

我该怎么做才能确定为什么第二次重新编译所有来源并减轻它？

我在linux容器中使用scala 2.12.8运行sbt 1.2.8。

更新

我还没有解决这个问题，但我想我会在最糟糕的问题上分享一个解决方法。

主要问题：将“test compile”与“test run”分开。次要问题：更快的构建，无需在每次推送时重新编译所有内容

我没有辅助解决方案。对于小学：

我可以通过scalatest runner而不是通过scala -cp ... org.scalatest.tools.Runner从CLI运行sbt test，以避免任何重新编译的尝试。跑步者可以对.class文件目录进行操作。

变更摘要：

  1. 更新docker容器以包含scala cli安装。 （不幸的是，我现在需要保持这些版本同步）
  2. 建设阶段：sbt test:compile 'inspect run' 'export test:fullClasspath' | tee >(grep -F '.jar' > ~test-classpath.txt) 编译但也记录一个copy-patseable类路径字符串，适合传入scala -cp VALUE_HERE运行测试
  3. 测试阶段：scala -cp "$(cat test-classpath.txt)" org.scalatest.tools.Runner -R target/scala-2.12/test-classes/ -u target/test-reports -oD 使用.class中编译的target/scala-2.12/test-classes文件运行scalatest，使用编译阶段报告的类路径，printint到stdout以及报告目录

我不喜欢这个并且它有一些问题，但我想我会分享这个解决方法。

答案

我也在gitlab工作中使用sbt 1.2.8来解决这个问题。以前（使用sbt 0.13）缓存target目录工作正常。

现在我正试图通过设置手动调试：

logLevel := Level.Debug,
incOptions := incOptions.value.withApiDebug(true).withRelationsDebug(true),

在我的构建中。这应该打印失效的原因。虽然它产生的输出太多而无法在CI中运行，所以我很难再现我遇到问题的确切条件。

以上是关于sbt总是重新编译CI中的完整项目，即使使用缓存？的主要内容，如果未能解决你的问题，请参考以下文章

使用 SBT 进行慢速编译

sbt / maven是否包含jar文件中的完整依赖项？

如何防止SBT重新编译修改后的.class文件？

强制 sbt 重新加载整个构建定义

如何在scala中自动重新运行sbt项目

如何设置 IntelliJ Idea Scala 项目以识别本地 Ivy2 缓存？