Play2/Specs2 中的 Slick/Postgres 数据库测试问题

Posted 2023-04-11

【中文标题】Play2/Specs2 中的 Slick/Postgres 数据库测试问题

【英文标题】：Trouble with Slick/Postgres database testing in Play2/Specs2

【发布时间】：2017-07-07 10:41:44

【问题描述】：

我在构建应用程序测试时遇到了很多麻烦，希望有更多 Scala 经验的人可以为我指明正确的方向。

我有许多驻留在 Postgres 数据库中的数据模型，使用 Slick 映射到案例类。然后，我的 Play 应用程序为所述数据模型提供基于 JSON 的 REST 端点。 由于每个端点之间的大部分实际代码都是相似的，因此大部分代码都是作为一个 trait 实现的，该 trait 被混入到覆盖必要位的实际控制器中。

这很好用，但是当我尝试在每个控制器上运行单元测试时，大多数控制器都可以正常工作，然后我最终会遇到错误：

[error] Can't find a constructor for class helpers.DatabaseHelper
[warn] c.z.h.HikariConfig - The jdbcConnectionTest property is now deprecated, see the documentation for connectionTestQuery
[error]
[error] cannot create an instance for class FileControllerSpec
[error]   caused by java.sql.SQLTransientConnectionException: db - Connection is not available, request timed out after 1005ms.
[error]   caused by org.postgresql.util.PSQLException: FATAL: remaining connection slots are reserved for non-replication superuser connections
[error]
[error] STACKTRACE
[error]   sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error]   sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[error]   sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error]   java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[error]   org.specs2.reflect.Classes$$anonfun$org$specs2$reflect$Classes$$createInstanceForConstructor$1.apply(Classes.scala:104)
[error]   org.specs2.control.ActionT$$anonfun$safe$1.apply(ActionT.scala:88)
[error]   org.specs2.control.ActionT$$anonfun$reader$1$$anonfun$apply$6.apply(ActionT.scala:79)
[error]   org.specs2.control.Status$.safe(Status.scala:100)
[error]   org.specs2.control.StatusT$$anonfun$safe$1.apply(StatusT.scala:62)
[error]   org.specs2.control.StatusT$$anonfun$safe$1.apply(StatusT.scala:62)
[error]   scalaz.syntax.ToApplicativeOps$$anon$1.self$lzycompute(ApplicativeSyntax.scala:29)
[error]   scalaz.syntax.ToApplicativeOps$$anon$1.self(ApplicativeSyntax.scala:29)
[error]   scalaz.syntax.ToApplicativeOps$ApplicativeIdV$$anonfun$point$1.apply(ApplicativeSyntax.scala:33)
[error]   scalaz.WriterTApplicative$$anonfun$point$1.apply(WriterT.scala:282)
[error]   scalaz.WriterTApplicative$$anonfun$point$1.apply(WriterT.scala:282)
[error]   scalaz.effect.IO$$anonfun$apply$19$$anonfun$apply$20.apply(IO.scala:136)
[error]   scalaz.effect.IO$$anonfun$apply$19$$anonfun$apply$20.apply(IO.scala:136)
[error]   scalaz.FreeFunctions$$anonfun$return_$1.apply(Free.scala:326)
[error]   scalaz.FreeFunctions$$anonfun$return_$1.apply(Free.scala:326)
[error]   scalaz.std.FunctionInstances$$anon$1$$anonfun$map$1.apply(Function.scala:56)
[error]   scalaz.Free$$anonfun$run$1.apply(Free.scala:172)
[error]   scalaz.Free$$anonfun$run$1.apply(Free.scala:172)
[error]   scalaz.Free.go2$1(Free.scala:119)
[error]   scalaz.Free.go(Free.scala:122)
[error]   scalaz.Free.run(Free.scala:172)
[error]   scalaz.effect.IO$class.unsafePerformIO(IO.scala:22)
[error]   scalaz.effect.IOFunctions$$anon$6.unsafePerformIO(IO.scala:227)
[error]   org.specs2.reflect.Classes$$anonfun$createInstance$1$$anonfun$apply$1$$anonfun$3.apply(Classes.scala:37)
[error]   org.specs2.reflect.Classes$$anonfun$createInstance$1$$anonfun$apply$1$$anonfun$3.apply(Classes.scala:36)
[error]   scala.collection.immutable.List.map(List.scala:273)
[error]   org.specs2.reflect.Classes$$anonfun$createInstance$1$$anonfun$apply$1.apply(Classes.scala:36)
[error]   org.specs2.reflect.Classes$$anonfun$createInstance$1$$anonfun$apply$1.apply(Classes.scala:29)
[error]   scala.Function1$$anonfun$andThen$1.apply(Function1.scala:52)
[error]   org.specs2.control.Status$class.fold(Status.scala:30)
[error]   org.specs2.control.Ok.fold(Status.scala:95)
[error]   org.specs2.control.Status$class.flatMap(Status.scala:48)
[error]   org.specs2.control.Ok.flatMap(Status.scala:95)
[error]   org.specs2.control.Status$class.map(Status.scala:45)
[error]   org.specs2.control.Ok.map(Status.scala:95)
[error]   org.specs2.control.StatusT$$anonfun$map$1.apply(StatusT.scala:16)
[error]   org.specs2.control.StatusT$$anonfun$map$1.apply(StatusT.scala:16)
[error]   scalaz.WriterT$$anonfun$map$1.apply(WriterT.scala:46)
[error]   scalaz.WriterT$$anonfun$map$1.apply(WriterT.scala:46)
[error]   scalaz.effect.IO$$anonfun$map$1$$anonfun$apply$8.apply(IO.scala:56)
[error]   scalaz.effect.IO$$anonfun$map$1$$anonfun$apply$8.apply(IO.scala:55)
[error]   scalaz.Free$$anonfun$map$1.apply(Free.scala:52)
[error]   scalaz.Free$$anonfun$map$1.apply(Free.scala:52)
[error]   scalaz.Free$$anonfun$flatMap$1$$anonfun$apply$1.apply(Free.scala:60)
[error]   scalaz.Free$$anonfun$flatMap$1$$anonfun$apply$1.apply(Free.scala:60)
[error]   scalaz.Free.resume(Free.scala:72)
[error]   scalaz.Free.go2$1(Free.scala:118)
[error]   scalaz.Free.go(Free.scala:122)
[error]   scalaz.Free.run(Free.scala:172)
[error]   scalaz.effect.IO$class.unsafePerformIO(IO.scala:22)
[error]   scalaz.effect.IOFunctions$$anon$6.unsafePerformIO(IO.scala:227)
[error]   org.specs2.runner.SbtRunner$$anonfun$newTask$1$$anon$4.execute(SbtRunner.scala:37)
[error]   sbt.ForkMain$Run$2.call(ForkMain.java:294)
[error]   sbt.ForkMain$Run$2.call(ForkMain.java:284)
[error]   java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]   java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[error]   java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[error]   java.lang.Thread.run(Thread.java:745)

DatabaseHelper 是一个设置我的测试数据的对象。

减少规范的数量会使错误消失，所以我知道问题不在于特定的测试规范。

我按照 Play 文档的建议使用运行时 DI，并覆盖数据库绑定进行测试，以避免我的开发数据库空白。

我正在使用 Play evolutions 来管理数据库架构，但有意在设置/拆卸期间清空数据库并重新设置它，以确保在每个测试规范之前它都是原始的。

我认为发生的事情是我的所有测试规范都在同时初始化，这意味着它们都尝试同时连接到数据库，因此连接用完了。

我尝试使用 sbt 中的 parallelExecution 和 concurrentRestrictions 设置来设置一次只运行一个进程，但这无济于事。我尝试将每个规范也设置为按顺序运行，但这似乎不起作用。我也尝试过捕获异常并重试设置，但这似乎也不起作用。

我现在不知道如何尝试使我的测试有效！请帮忙。

非常感谢。

---

测试用例：

@RunWith(classOf[JUnitRunner])
class FileControllerSpec extends GenericControllerSpec 
  sequential

  override val componentName: String = "FileController"
  override val uriRoot: String = "/file"

  override def testParsedJsonObject(checkdata: JsLookupResult, parsed_test_json: JsValue) = 
    val object_keys = Seq("filepath","user","ctime","mtime","atime")
    val object_keys_int = Seq("storage","version")

    object_keys.map(key=>
      (checkdata \ key).as[String] must equalTo((parsed_test_json \ key).as[String])
    ) ++ object_keys_int.map(key=>
      (checkdata \ key).as[Int] must equalTo((parsed_test_json \ key).as[Int])
    )
  

  override val testGetId: Int = 3
  override val testGetDocument: String = """"filepath":"/path/to/a/video.mxf","storage":1,"user":"me","version":1,"ctime":"1970-01-01T04:25:45.678+0100","mtime":"1970-01-01T04:25:45.678+0100","atime":"1970-01-01T04:25:45.678+0100""""
  override val testCreateDocument: String =  """"filepath":"/path/to/some/other.project","storage":1,"user":"test","version":3,"ctime":"2017-03-17T13:51:00.123+0000","mtime":"2017-03-17T13:51:00.123+0000","atime":"2017-03-17T13:51:00.123+0000""""
  override val minimumNewRecordId = 3
  override val testDeleteId: Int = 2
  override val testConflictId: Int = -1

通用控制器规范：

@RunWith(classOf[JUnitRunner])
trait GenericControllerSpec extends Specification with BeforeAfterAll 
  //can over-ride bindings here. see https://www.playframework.com/documentation/2.5.x/ScalaTestingWithGuice
  val application:Application = new GuiceApplicationBuilder()
    .overrides(bind[DatabaseConfigProvider].to[TestDatabase.testDbProvider])
    .build

  val injector:Injector = new GuiceApplicationBuilder()
      .overrides(bind[DatabaseConfigProvider].to[TestDatabase.testDbProvider])
      .injector()

  def inject[T : ClassTag]: T = injector.instanceOf[T]

  //needed for body.consumeData
  implicit val system = ActorSystem("storage-controller-spec")
  implicit val materializer = ActorMaterializer()

  protected val databaseHelper:DatabaseHelper = inject[DatabaseHelper]

  val logger: Logger = Logger(this.getClass)

  override def beforeAll(): Unit =
    logger.warn(">>>> before all <<<<")
    val theFuture = databaseHelper.setUpDB().map(
      case Success(result)=>logger.info("DB setup successful")
      case Failure(error)=>logger.error(s"DB setup failed: $error")
    )

    Await.result(theFuture, 10.seconds)
  

  override def afterAll(): Unit =
    logger.warn("<<<< after all >>>>")
    Await.result(databaseHelper.teardownDB(), 10.seconds)
  

  val componentName:String
  val uriRoot:String

  def testParsedJsonObject(checkdata:JsLookupResult,test_parsed_json:JsValue):Seq[MatchResult[Any]]

  val testGetId:Int
  val testGetDocument:String
  val testCreateDocument:String
  val testDeleteId:Int
  val testConflictId:Int
  val minimumNewRecordId:Int

  def bodyAsJsonFuture(response:Future[play.api.mvc.Result]) = response.flatMap(result=>
    result.body.consumeData.map(contentBytes=> 
      logger.debug(contentBytes.decodeString("UTF-8"))
      Json.parse(contentBytes.decodeString("UTF-8"))
    
    )
  )

  componentName should 

    "return 400 on a bad request" in 
      logger.debug(s"$uriRoot/boum")
      val response = route(application,FakeRequest(GET, s"$uriRoot/boum")).get
      status(response) must equalTo(BAD_REQUEST)
    

    "return valid data for a valid record" in  
      logger.warn(s"Test URL is $uriRoot/1")
      val response:Future[play.api.mvc.Result] = route(application, FakeRequest(GET, s"$uriRoot/1")).get

      status(response) must equalTo(OK)
      val jsondata = Await.result(bodyAsJsonFuture(response), 5.seconds).as[JsValue]
      (jsondata \ "status").as[String] must equalTo("ok")
      (jsondata \ "result" \ "id").as[Int] must equalTo(1)
      testParsedJsonObject(jsondata \ "result", Json.parse(testGetDocument))
    

    "accept new data to create a new record" in 
      val response = route(application, FakeRequest(
        method="PUT",
        uri=uriRoot,
        headers=FakeHeaders(Seq(("Content-Type", "application/json"))),
        body=testCreateDocument)
      ).get

      status(response) must equalTo(OK)
      val jsondata = Await.result(bodyAsJsonFuture(response), 5.seconds).as[JsValue]
      (jsondata \ "status").as[String] must equalTo("ok")
      (jsondata \ "detail").as[String] must equalTo("added")
      (jsondata \ "id").as[Int] must greaterThanOrEqualTo(minimumNewRecordId) //if we re-run the tests without blanking the database explicitly this goes up

      val newRecordId = (jsondata \ "id").as[Int]
      val checkResponse = route(application, FakeRequest(GET, s"$uriRoot/$newRecordId")).get
      val checkdata = Await.result(bodyAsJsonFuture(checkResponse), 5.seconds)


      (checkdata \ "status").as[String] must equalTo("ok")
      (checkdata \ "result" \ "id").as[Int] must equalTo(newRecordId)
      testParsedJsonObject(checkdata \ "result", Json.parse(testCreateDocument))
    

    "delete a record" in 
      val response = route(application, FakeRequest(
        method="DELETE",
        uri=s"$uriRoot/$testDeleteId",
        headers=FakeHeaders(),
        body="")
      ).get

      status(response) must equalTo(OK)
      val jsondata = Await.result(bodyAsJsonFuture(response), 5.seconds).as[JsValue]
      (jsondata \ "status").as[String] must equalTo("ok")
      (jsondata \ "detail").as[String] must equalTo("deleted")
      (jsondata \ "id").as[Int] must equalTo(testDeleteId)
    

    "return conflict (409) if attempting to delete something with sub-objects" in 
      val response = route(application, FakeRequest(
        method = "DELETE",
        uri = s"$uriRoot/$testConflictId",
        headers = FakeHeaders(),
        body = "")
      ).get

      status(response) must equalTo(CONFLICT)
      val jsondata = Await.result(bodyAsJsonFuture(response), 5.seconds).as[JsValue]
      (jsondata \ "status").as[String] must equalTo("error")
      (jsondata \ "detail").as[String] must equalTo("This is still referenced by sub-objects")
    
  

数据库助手：

class DatabaseHelper @Inject()(configuration: Configuration, dbConfigProvider: DatabaseConfigProvider) 

  private val dbConfig = dbConfigProvider.get[JdbcProfile]
  private val logger: Logger = Logger(this.getClass)

  def setUpDB():Future[Try[Unit]] = 
    logger.warn("In setUpDB")
    dbConfig.db.run(
      DBIO.seq(
        (TableQuery[FileAssociationRow].schema ++
          TableQuery[FileEntryRow].schema ++
          TableQuery[ProjectEntryRow].schema ++
          TableQuery[ProjectTemplateRow].schema ++
          TableQuery[ProjectTypeRow].schema ++
          TableQuery[StorageEntryRow].schema
        ).create,
        TableQuery[StorageEntryRow] += StorageEntry(None,None,"filesystem",Some("me"),None,None,None),
        TableQuery[StorageEntryRow] += StorageEntry(None,None,"omms",Some("you"),None,None,None),
        TableQuery[FileEntryRow] += FileEntry(None,"/path/to/a/video.mxf",1,"me",1,new Timestamp(12345678),new Timestamp(12345678),new Timestamp(12345678)),
        TableQuery[FileEntryRow] += FileEntry(None,"/path/to/secondtestfile",1,"tstuser",1,new Timestamp(123456789),new Timestamp(123456789),new Timestamp(123456789)),
        //""""name": "Premiere test template 1","projectTypeId": 1,"filepath", "storageId": 1"""
        //""name":,"opensWith":"AdobePremierePro.app","targetVersion":"14.0""
        TableQuery[ProjectTypeRow] += ProjectType(None,"Premiere 2014 test","AdobePremierePro.app","14.0"),
        TableQuery[ProjectTypeRow] += ProjectType(None,"Cubase 7.0 test","Cubase.app","7.0"),
        TableQuery[ProjectTemplateRow] += ProjectTemplate(Some(1),"Premiere test template 1",1,"/srv/projectfiles/ProjectTemplatesDev/Premiere/premiere_template_2014.prproj",1)

      ).asTry
    )
  

  def teardownDB():Future[Try[Unit]] = 
    logger.warn("In teardownDB")
    dbConfig.db.run(
      DBIO.seq(
        (
          TableQuery[FileAssociationRow].schema ++
            TableQuery[FileEntryRow].schema ++
            TableQuery[ProjectEntryRow].schema ++
            TableQuery[ProjectTemplateRow].schema ++
            TableQuery[ProjectTypeRow].schema ++
            TableQuery[StorageEntryRow].schema
        ).drop
      ).asTry
    )
  

build.sbt 设置：

concurrentRestrictions in Global := Seq(
  Tags.limit(Tags.Test, 1),
  Tags.limitAll(1)
)

parallelExecution in Test := false

【参考方案1】：

异常告诉你你的连接池没有连接。

我在这里看到两个问题：

您没有在拆卸时关闭数据库连接。 您可以为每个连接池打开许多连接，并且由于您的每个规范都在创建一个新的连接池，因此您的 postgres 服务器上配置的连接最大值已用完。您可以通过减少slick config 的“numThreads”参数来减少每个连接池的连接数

由于性能原因，我还是建议您使用带有 postgres 设置的 H2DB 驱动程序进行单元测试，只要您不使用 H2DB 无法模拟的 postgres 特定功能。

【讨论】：

嗨大卫，非常感谢您的帮助。我已经添加了接近拆解的内容，并让东西在本地工作，但在 CI 中仍然存在问题。我已经开始使用 H2DB 驱动程序，但切换到 Postgres，因为我无法让 H2 与数据库 Evolutions 一起工作，但目前我决定放弃 Evolutions 并让应用程序正常工作，然后担心以后的数据迁移。这确实有助于提高我的理解:)