Scala: Practice in Real Project
Posted 楠爸自习室
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Scala: Practice in Real Project相关的知识，希望对你有一定的参考价值。
In this blog, I will share some practice in our Scala project, hope this can help you.
Project Management
Specify the sbt version
We need the sbt version of project to be always same on different device, then we can ensure the compiled jar of application are same between local, release and production.
Like .ruby-version, .python-version, sbt can also specify the sbt version in project/build.properties
sbt.version=1.4.1
It doesn't matter what's the sbt version in your local environment, sbt will always download the specified version.
Speed up dependency download
sbt will download dependencies in sequence which is very slow, it may take half hour.
To speed up this process, we can use the plugin sbt-coursier[1].
To also speed up the plugin download, add the following line in project/project/plugins.sbt
addSbtPlugin("io.get-coursier" % "sbt-coursier" % "2.0.0-RC6-8")
Add the following line in project/plugins.sbt
addSbtCoursier
Apply code formatter
It's necessary to use unified code style across all team members, all IDE and all device.
Add plugin scalafmt[2]
addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.4.0"
Add configuration .scalafmt.conf
version = 2.7.5
Check the code style before test
addCommandAlias("styleCheckAndTest", ";clean;scalafmtCheck;test")
Format code
sbt scalafmtAll
Use scalafmt in IntellJ IDEA
Optimise compiler option
A good set of compiler options can let us find lots of potential issue in compile process, you can find the best practice in scalac-flags[3]
Other useful plugins
FP
In FP code, sometimes we need to do some type projection to get partial applied type.
For example, We want all functions to return error or normal value, then we can define an unified return type.
type AppErrorOr[A] = Either[Throwable, A]
But what if we have a function like this
def identity[A, F[_]](value: F[A] "A, F[_]"): F[A] = value
And we want to apply it to Either
val either:Either[String, Int] = Right(1)
identity(either) // can't be compiled
Compiler tell us we need to give the type parameter explicitly, But we found it requires kind F[_], but Either is F[_, _].
It's not possible to define a partial applied type explicitly every time. Here we need the type to be defined anonymously, lucklily Scala support it
identity[Int, ({type L[A] = Either[String, A]})#L](either "Int, ({type L[A] = Either[String, A]})#L") // success
It's pretty hard and ugly to use this syntax, plugin kind-projector[4] give a better implementation.
identity[Int, Either[String, *]](either "Int, Either[String, *]")
Note: kind-projector involve * in this PR[5] to support Scala 3.0, you can still use ?.
To use this plugin, add the following line into build.sbt
addCompilerPlugin("org.typelevel" %% "kind-projector" % "0.11.0" cross CrossVersion.full)
Package
sbt-native-packager[6] can help us to package our app, then build an docker image.
Add the following line in project/plugins.sbt
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.7.6")
Add the following line in build.sbt to enable JavaAppPackaging and DockerPlugin
enablePlugins(JavaAppPackaging)
Note: JavaAppPackaging will enable DockerPlugin automatically
Add docker.sbt which is just like a Dockerfile written in Scala.
import com.typesafe.sbt.packager.docker.Cmd
defaultLinuxInstallLocation in Docker := s"/opt/rea/apps/${name.value}"
version in Docker := scala.util.Properties.envOrElse("VERSION", "v0." ++ scala.util.Properties.envOrElse("BUILDKITE_BUILD_NUMBER", "DEV"))
dockerBaseImage := ???
dockerRepository := ???
packageName in Docker := ???
dockerCommands ++= Seq(
Cmd("ENV", s"""JAVA_OPTS="-Xms1024m -Xmx1024m""""),
Cmd("RUN", s""" \\
| echo "RUN Command" \\
""".stripMargin)
)
dockerEntrypoint := ???
Note: we can also put the content of docker.sbt in build.sbt. If we put it in docker.sbt, they will be merged when we run sbt command.
Coding
Pattern
We tried Cake Pattern[7],Eff[8] and Tagless Final[9] in our projects, the winner is Tagless Final. But it's still hard to manage the dependency injection, we are trying to use Tagless Final and ReaderT pattern[10] together.
In the future, we may try ZIO[11] which is a combination of Tagless Final and ReaderT pattern.
Framework
FP
Definitely cats[12]
Add the following line in build.sbt
libraryDependencies += "org.typelevel" %% "cats-core" % "2.1.1"
Http
Http4s[13] supply both http server and client based on cats
libraryDependencies ++= Seq(
"org.http4s" %% "http4s-blaze-server" % "0.21.8",
"org.http4s" %% "http4s-blaze-client" % "0.21.8",
"org.http4s" %% "http4s-circe" % "0.21.8",
"org.http4s" %% "http4s-dsl" % "0.21.8"
)
JSON
Circe[14]
libraryDependencies ++= Seq(
"io.circe" %% "circe-core" % "0.12.3",
"io.circe" %% "circe-generic" % "0.12.3",
"io.circe" %% "circe-parser" % "0.12.3"
)
Database Connection
Doobie[15]
libraryDependencies ++= Seq(
"org.tpolecat" %% "doobie-core" % "0.9.0",
"org.tpolecat" %% "doobie-hikari" % "0.9.0", // HikariCP transactor.
"org.tpolecat" %% "doobie-postgres" % "0.9.0", // Postgres driver 42.2.12 + type mappings.
"org.tpolecat" %% "doobie-quill" % "0.9.0", // Support for Quill 3.5.1
"org.tpolecat" %% "doobie-specs2" % "0.9.0" % "test", // Specs2 support for typechecking statements.
)
Database Migration
Flyway[16]
libraryDependencies += "org.flywaydb" % "flyway-core" % "7.1.1"
Code
No var
We can modify a variable with var in any time without compiler warning, definitely should not be used.
Create a new instance with modified value instead.
No null
null can be assigned to any AnyRef variable. For a variable with given type, we don't know if it really store the value of given type or just null, which will confuse the meaning of type.
Use Option if your variable need to store null
No Any
Any type can be assigned to Any. For a variable with Any type, we really don't know what value it store. The code will be hard to read and maintain.
Use concrete type as much as possible, you don't need Any, trust me.
No return
In function, Scala will treat the value of last expression as return value, we don't need to return explicitly.
Can write less code, why not?
Use for instead of nested map/flatMap
Bad
f1()
.flatMap(x1 =>
f2(x1).flatMap(x2 =>
f3(x2).flatMap(x3 =>
f4(x3).map(x4 => x4))))
Good
for {
x1 <- f1()
x2 <- f2(x1)
x3 <- f3(x2)
x4 <- f4(x3)
} yield x4
for expression is easier to read and maintain.
Use implicits cautiously
implicits is a powerful tool, but it is very easy to be abused.
Most of time, it is implicits which make the code hard to read and maintain. It's also the biggest blocker for newbies to learn scala.
Don't use it if possible, except you can prove you have to do that.
Use pattern-matching instead of fold
Bad
val a:Option[Int] = ???
a.fold(
for {
x1 <- f1()
x2 <- f2(x1)
} yield x2
)(x =>
for {
x3 <- f3(x)
x4 <- f4(x3)
} yield x4
)
Good
val a:Option[Int] = ???
a match {
case None =>
for {
x1 <- f1()
x2 <- f2(x1)
} yield x2
case Some(x) =>
for {
x3 <- f3(x)
x4 <- f4(x3)
} yield x4
}
Most of time, pattern-matching will be easier to read.
Use sealed if possible
For data type with sealed, compiler can help us to check if we cover all the branch
Use trait group implicit instances and inject them into companion object
For a type T, we may have lots of implicit instances, usually they can be grouped like this
-
Instances
Define some implicit instances used by other components.
implicit val decoder: Decoder[T] = ???
implicit val ordering: Ordering[T] = ??? -
Syntax
Add extra methods to the given type.
implicit class TAddOps(v: T) {
def add(other: T): T = ???
}
implicit class THttpOps(v: T) {
def send: HttpResponse = ???
}
We don't want to import a package every time to use the implicit instances.
According to the rule of implicit, companion object is the fall back scope to find the implicit instances of T. So it make sense to put all of them into companion object.
We still want to group these instances better, so the pattern may look like this
trait TInstances {
implicit decoder: Decoder[T] = ???
implicit ordering: Ordering[T] = ???
}
trait TSyntax {
implicit class TAddOps(v: T) {
def add(other: T): T = ???
}
implicit class THttpOps(v: T) {
def send: HttpResponse = ???
}
}
object T extends TInstances with TSyntax
Use case class instead of class if possible
case class is easier to be copied and can be used in pattern-matching.
Most of time, algebraic data type are composed by trait and case class.
Don't use Option.get, List.head, Either.get and Try.get
These functions may throw exception and easy to be ignored,
There are safer function like Option.getOrElse, List.headOption, Either.getOrElse and Try.getOrElse.
Use F[_], G[_], A, B, C as type parameter
Using a set of unified symbol as type parameter can make the code easier to read.
Use @tailrec annotation
If we are writing a recursive function, add @tailrec annotation, then compiler can help us to check if it is really a tail recursive function.
Test
Framework
specs2[17]
libraryDependencies ++= Seq("org.specs2" %% "specs2-core" % "4.10.0" % "test")
spec2 have two type of test style, mutable style is similar to other language and also our preffered style.
import org.specs2.mutable.Specification
import org.specs2.specification.Scope
class HelloSpec extends Specification {
"Hello" should {
"should print hello" in new Scope {
1.toString should_==("1")
}
}
}
Mock
mockito-scala[18]
libraryDependencies ++= Seq(
"org.hamcrest" % "hamcrest" % "2.2" % Test,
"org.mockito" %% "mockito-scala-specs2" % "1.15.0" % Test,
)
specs2 also support mockito, but it doesn't support function with default parameter and not good at FP.They also suggest us to use mockito-scala[19].
Test Coverage
sbt-scoverage[20]
Add the following line in project/plugins.sbt
addSbtPlugin("org.scoverage" % "sbt-scoverage" % "1.6.1")
Generate coverage report when running test in build.sbt
addCommandAlias("TestWithCoverage", ";clean;coverage;test;coverageReport;coverageOff")
Tips
-
Stay in the sbt console to run compile/test repeatedly, which is faster. -
Use sbt instead of IntellJ IDEA to compile project, which will give more information. -
Declare type explicitly if you can not understand the error message. -
Use ammonite-repl [21] instead of scalac to run experiment code. -
Readability is more important than fantastic syntax.
参考资料
sbt-coursier: https://github.com/coursier/sbt-coursier
[2]scalafmt: https://github.com/scalameta/scalafmt
[3]scalac-flags: https://tpolecat.github.io/2017/04/25/scalac-flags.html
[4]kind-projector: https://github.com/typelevel/kind-projector
[5]PR: https://github.com/typelevel/kind-projector/pull/91
[6]sbt-native-packager: https://github.com/sbt/sbt-native-packager
[7]Cake Pattern: https://blog.shangjiaming.com/scala%20tutorial/cake-pattern/
[8]Eff: https://github.com/atnos-org/eff
[9]Tagless Final: https://degoes.net/articles/zio-environment
[10]ReaderT pattern: https://www.fpcomplete.com/blog/2017/06/readert-design-pattern/
[11]ZIO: https://github.com/zio/zio
[12]cats: https://github.com/typelevel/cats
[13]Http4s: https://github.com/http4s/http4s
[14]Circe: https://github.com/circe/circe
[15]Doobie: https://github.com/tpolecat/doobie
[16]Flyway: https://github.com/flyway/flyway
[17]specs2: https://github.com/etorreborre/specs2
[18]mockito-scala: https://github.com/mockito/mockito-scala
[19]They also suggest us to use mockito-scala: https://github.com/etorreborre/specs2/issues/854#issuecomment-674999804
[20]sbt-scoverage: https://github.com/scoverage/sbt-scoverage
[21]ammonite-repl: https://ammonite.io/
关注【楠爸自习室】
以上是关于Scala: Practice in Real Project的主要内容，如果未能解决你的问题，请参考以下文章
clickhouseClickHouse Practice in EOI
python:practice built-in function
Records of Pytorch in Practice