FunDA－ Reactive Streams：Play with IterateesEnumerator and Enumeratees

Posted 2020-08-29 雪川大虫

    在上一节我们介绍了Iteratee。它的功能是消耗从一些数据源推送过来的数据元素，不同的数据消耗方式代表了不同功能的Iteratee。所谓的数据源就是我们这节要讨论的Enumerator。Enumerator是一种数据源：它会根据下游数据消耗方（Iteratee）的具体状态主动向下推送数据元素。我们已经讨论过Iteratee的状态Step类型：

trait Step[E,+A]
case class Done[+A,E](a: A, remain: Input[E]) extends Step[E,A]
case class Cont[E,+A](k: Input[E] => InputStreamHandler[E,A]) extends Step[E,A]
case class Error[E](msg: String, loc:Input[E]) extends Step[E,Nothing]

这其中Iteratee通过Cont状态通知Enumerator可以发送数据元素，并提供了k函数作为Enumerator的数据推送函数。Enumerator推送的数据元素，也就是Iteratee的输入Input[E]，除单纯数据元素之外还代表着数据源状态： 

trait Input[+E]
case class EL[E](e: E) extends Input[E]
case object EOF extends Input[Nothing]
case object Empty extends Input[Nothing]

Enumerator通过Input[E]来通知Iteratee当前数据源状态，如：是否已经完成所有数据推送（EOF），或者当前推送了什么数据元素（El[E](e:E)）。Enumerator主动向Iteratee输出数据然后返回新状态的Iteratee。我们可以从Enumerator的类型款式看得出：

trait Enumerator[E] {

  /**
   * Apply this Enumerator to an Iteratee
   */
  def apply[A](i: Iteratee[E, A]): Future[Iteratee[E, A]]

}

这个Future的目的主要是为了避免占用线程。实际上我们可以最终通过调用Iteratee的fold函数来实现Enumerator功能，如：

 /**
   * Creates an enumerator which produces the one supplied
   * input and nothing else. This enumerator will NOT
   * automatically produce Input.EOF after the given input.
   */
  def enumInput[E](e: Input[E]) = new Enumerator[E] {
    def apply[A](i: Iteratee[E, A]): Future[Iteratee[E, A]] =
      i.fold {
        case Step.Cont(k) => eagerFuture(k(e))
        case _ => Future.successful(i)
      }(dec)
  }

又或者通过构建器（constructor, apply）来构建Eumerator：

/**
   * Create an Enumerator from a set of values
   *
   * Example:
   * {{{
   *   val enumerator: Enumerator[String] = Enumerator("kiki", "foo", "bar")
   * }}}
   */
  def apply[E](in: E*): Enumerator[E] = in.length match {
    case 0 => Enumerator.empty
    case 1 => new Enumerator[E] {
      def apply[A](i: Iteratee[E, A]): Future[Iteratee[E, A]] = i.pureFoldNoEC {
        case Step.Cont(k) => k(Input.El(in.head))
        case _ => i
      }
    }
    case _ => new Enumerator[E] {
      def apply[A](i: Iteratee[E, A]): Future[Iteratee[E, A]] = enumerateSeq(in, i)
    }
  }

  /**
   * Create an Enumerator from any TraversableOnce like collection of elements.
   *
   * Example of an iterator of lines of a file :
   * {{{
   *  val enumerator: Enumerator[String] = Enumerator( scala.io.Source.fromFile("myfile.txt").getLines )
   * }}}
   */
  def enumerate[E](traversable: TraversableOnce[E])(implicit ctx: scala.concurrent.ExecutionContext): Enumerator[E] = {
    val it = traversable.toIterator
    Enumerator.unfoldM[scala.collection.Iterator[E], E](it: scala.collection.Iterator[E])({ currentIt =>
      if (currentIt.hasNext)
        Future[Option[(scala.collection.Iterator[E], E)]]({
          val next = currentIt.next
          Some((currentIt -> next))
        })(ctx)
      else
        Future.successful[Option[(scala.collection.Iterator[E], E)]]({
          None
        })
    })(dec)
  }

  /**
   * An empty enumerator
   */
  def empty[E]: Enumerator[E] = new Enumerator[E] {
    def apply[A](i: Iteratee[E, A]) = Future.successful(i)
  }

  private def enumerateSeq[E, A]: (Seq[E], Iteratee[E, A]) => Future[Iteratee[E, A]] = { (l, i) =>
    l.foldLeft(Future.successful(i))((i, e) =>
      i.flatMap(it => it.pureFold {
        case Step.Cont(k) => k(Input.El(e))
        case _ => it
      }(dec))(dec))
  }

下面是个直接构建Enumerator的例子： 

 val enumUsers: Enumerator[String] = {
   Enumerator("Tiger","Hover","Grand","John")    
       //> enumUsers  : play.api.libs.iteratee.Enumerator[String] = [email protected]

在这个例子里的Enumerator就是用上面那个apply构建的。我们把enumUsers连接到costume Iteratee：

 

 val consume = Iteratee.consume[String]()        //> consume  : play.api.libs.iteratee.Iteratee[String,String] = Cont(<function1>)
 val consumeUsers = enumUsers.apply(consume)      //> consumeUsers  : scala.concurrent.Future[play.api.libs.iteratee.Iteratee[String,String]] = Success([email protected])

我们是用apply(consume)来连接Enumerator和Iteratees的。apply函数的定义如下：

/**
   * Attaches this Enumerator to an [[play.api.libs.iteratee.Iteratee]], driving the
   * Iteratee to (asynchronously) consume the input. The Iteratee may enter its
   * [[play.api.libs.iteratee.Done]] or [[play.api.libs.iteratee.Error]]
   * state, or it may be left in a [[play.api.libs.iteratee.Cont]] state (allowing it
   * to consume more input after that sent by the enumerator).
   *
   * If the Iteratee reaches a [[play.api.libs.iteratee.Done]] state, it will
   * contain a computed result and the remaining (unconsumed) input.
   */
  def apply[A](i: Iteratee[E, A]): Future[Iteratee[E, A]]

这是个抽象函数。举个例实现这个apply函数的例子：

/**
   * Creates an enumerator which produces the one supplied
   * input and nothing else. This enumerator will NOT
   * automatically produce Input.EOF after the given input.
   */
  def enumInput[E](e: Input[E]) = new Enumerator[E] {
    def apply[A](i: Iteratee[E, A]): Future[Iteratee[E, A]] =
      i.fold {
        case Step.Cont(k) => eagerFuture(k(e))
        case _ => Future.successful(i)
      }(dec)
  }

consumeUsers: Future[Iteratee[String,String]]，我们用Future的函数来显示发送数据内容：

 val futPrint = consumeUsers.flatMap { i => i.run }.map(println)
    //> futPrint  : scala.concurrent.Future[Unit] = List()
 Await.ready(futPrint,Duration.Inf)     //> TigerHoverGrandJohn res0: demo.worksheet.enumerator.futPrint.type = Success(()) 

另一种更直接的方式：

val futUsers = Iteratee.flatten(consumeUsers).run.map(println)
      //> futUsers  : scala.concurrent.Future[Unit] = List()
 Await.ready(futPrint,Duration.Inf)               
      //> TigerHoverGrandJohnres1: demo.worksheet.enumerator.futPrint.type = Success(())

我们也可以使用函数符号 |>> ：

 val futPrintUsers = {
  Iteratee.flatten(enumUsers |>> consume).run.map(println)
     //> futPrintUsers  : scala.concurrent.Future[Unit] = List()
 }
 Await.ready(futPrintUsers,Duration.Inf)          
     //> TigerHoverGrandJohn res2: demo.worksheet.enumerator.futPrintUsers.type = Success(())

我们还可以把两个Enumerator串联起来向一个Iteratee发送数据：

 val futEnums = {
   Iteratee.flatten {
     (enumUsers >>> enumColors) |>> consume
   }.run.map(println)                       //> futEnums  : scala.concurrent.Future[Unit] = List()
 }
  Await.ready(futEnums,Duration.Inf)              
      //> TigerHoverGrandJohnRedWhiteBlueYellow res3: demo.worksheet.enumerator.futEnums.type = Success(())

当然，最实用的应该是把InputStream的数据推送给一个Iteratee，如把一个文件内容发送给Iteratee：

/**
   * Create an enumerator from the given input stream.
   *
   * Note that this enumerator will block when it reads from the file.
   *
   * @param file The file to create the enumerator from.
   * @param chunkSize The size of chunks to read from the file.
   */
  def fromFile(file: java.io.File, chunkSize: Int = 1024 * 8)(implicit ec: ExecutionContext): Enumerator[Array[Byte]] = {
    fromStream(new java.io.FileInputStream(file), chunkSize)(ec)
  }

/**
   * Create an enumerator from the given input stream.
   *
   * This enumerator will block on reading the input stream, in the supplied ExecutionContext.  Care must therefore
   * be taken to ensure that this isn‘t a slow stream.  If using this with slow input streams, make sure the
   * ExecutionContext is appropriately configured to handle the blocking.
   *
   * @param input The input stream
   * @param chunkSize The size of chunks to read from the stream.
   * @param ec The ExecutionContext to execute blocking code.
   */
  def fromStream(input: java.io.InputStream, chunkSize: Int = 1024 * 8)(implicit ec: ExecutionContext): Enumerator[Array[Byte]] = {
    implicit val pec = ec.prepare()
    generateM({
      val buffer = new Array[Byte](chunkSize)
      val bytesRead = blocking { input.read(buffer) }
      val chunk = bytesRead match {
        case -1 => None
        case `chunkSize` => Some(buffer)
        case read =>
          val input = new Array[Byte](read)
          System.arraycopy(buffer, 0, input, 0, read)
          Some(input)
      }
      Future.successful(chunk)
    })(pec).onDoneEnumerating(input.close)(pec)
  }

这项功能的核心函数是这个generateM，它的函数款式如下：

/**
   * Like [[play.api.libs.iteratee.Enumerator.repeatM]], but the callback returns an Option, which allows the stream
   * to be eventually terminated by returning None.
   *
   * @param e The input function.  Returns a future eventually redeemed with Some value if there is input to pass, or a
   *          future eventually redeemed with None if the end of the stream has been reached.
   */
  def generateM[E](e: => Future[Option[E]])(implicit ec: ExecutionContext): Enumerator[E] = checkContinue0(new TreatCont0[E] {
    private val pec = ec.prepare()

    def apply[A](loop: Iteratee[E, A] => Future[Iteratee[E, A]], k: Input[E] => Iteratee[E, A]) = executeFuture(e)(pec).flatMap {
      case Some(e) => loop(k(Input.El(e)))
      case None => Future.successful(Cont(k))
    }(dec)
  })

checkContinue0函数是这样定义的：

trait TreatCont0[E] {

    def apply[A](loop: Iteratee[E, A] => Future[Iteratee[E, A]], k: Input[E] => Iteratee[E, A]): Future[Iteratee[E, A]]

  }

  def checkContinue0[E](inner: TreatCont0[E]) = new Enumerator[E] {

    def apply[A](it: Iteratee[E, A]): Future[Iteratee[E, A]] = {

      def step(it: Iteratee[E, A]): Future[Iteratee[E, A]] = it.fold {
        case Step.Done(a, e) => Future.successful(Done(a, e))
        case Step.Cont(k) => inner[A](step, k)
        case Step.Error(msg, e) => Future.successful(Error(msg, e))
      }(dec)

      step(it)
    }
  }

从这段代码 case Step.Cont(k)=>inner[A](step, k)可以推断操作模式应该是当下游Iteratee在Cont状态下不断递归式调用Cont函数k向下推送数据e。我们再仔细看看generateM的函数款式；

 def generateM[E](e: => Future[Option[E]])(implicit ec: ExecutionContext): Enumerator[E] 

实际上刚才的操作就是重复调用这个e:=>Future[Option[E]]函数。再分析fromStream代码：

  def fromStream(input: java.io.InputStream, chunkSize: Int = 1024 * 8)(implicit ec: ExecutionContext): Enumerator[Array[Byte]] = {
    implicit val pec = ec.prepare()
    generateM({
      val buffer = new Array[Byte](chunkSize)
      val bytesRead = blocking { input.read(buffer) }
      val chunk = bytesRead match {
        case -1 => None
        case `chunkSize` => Some(buffer)
        case read =>
          val input = new Array[Byte](read)
          System.arraycopy(buffer, 0, input, 0, read)
          Some(input)
      }
      Future.successful(chunk)
    })(pec).onDoneEnumerating(input.close)(pec)
  }

我们看到传入generateM的参数是一段代码，在Iteratee状态为Cont时会不断重复运行，也就是说这段代码会逐次从输入源中读取chunkSize个Byte。这种做法是典型的Streaming方式，避免了一次性上载所有数据。下面是一个文件读取Enumerator例子：

 import java.io._
 val fileEnum: Enumerator[Array[Byte]] = {
  Enumerator.fromFile(new File("/users/tiger/lines.txt"))
 }
 val futFile = Iteratee.flatten { fileEnum |>> consume }.run.map(println)

注意：fileEnum |>> consume并不能通过编译，这是因为fileEnum是个Enumerator[Array[Byte]]，而consume是个Iteratee[String,String]，Array[Byte]与String类型不符。我们可以用个Enumeratee来进行相关的转换。下面就介绍一下Enumeratee的功能。

Enumeratee其实是一种转换器。它把Enumerator产生的数据转换成能适配Iteratee的数据类型，或者Iteratee所需要的数据。比如我们想把一串字符类的数字汇总相加时，首先必须把字符转换成数字类型才能进行Iteratee的汇总操作：

val strNums = Enumerator("1","2","3")            //> strNums  : play.api.libs.iteratee.Enumerator[String] = play.a[email protected]
 val sumIteratee: Iteratee[Int,Int] = Iteratee.fold(0)((s,i) => s+i)
                                                 //> sumIteratee  : play.api.libs.iteratee.Iteratee[Int,Int] = Cont(<function1>)
 
 val strToInt: Enumeratee[String,Int] = Enumeratee.map {s => s.toInt}
                                                 //> strToInt  : play.api.libs.iteratee.Enumeratee[String,Int] = [email protected]
 strNums |>> strToInt.transform(sumIteratee)     //> res4: scala.concurrent.Future[play.api.libs.iteratee.Iteratee[String,Int]] = List()
 strNums |>> strToInt &>> sumIteratee            //> res5: scala.concurrent.Future[play.api.libs.iteratee.Iteratee[String,Int]] = List()
 strNums.through(strToInt) |>> sumIteratee       //> res6: scala.concurrent.Future[play.api.libs.iteratee.Iteratee[Int,Int]] = List()
 val futsum = Iteratee.flatten(strNums &> strToInt |>> sumIteratee).run.map(println)
                                                //> futsum  : scala.concurrent.Future[Unit] = List()
 Await.ready(futsum,Duration.Inf)               //> 6
                                                //| res7: demo.worksheet.enumerator.futsum.type = Success(())

在上面这个例子里Enumerator数据元素是String, Iteratee操作数据类型是Int, strToInt是个把String转换成Int的Enumeratee，我们用了几种转换方式的表达形式，结果都是一样的，等于6。我们可以用Enumerator.through或者Enumeratee.transform来连接Enumerator与Iteratee。当然，我们也可以筛选输入Iteratee的数据：

val sum2 = strNums &> Enumeratee.take(2) &> strToInt |>> sumIteratee
                 //> sum2  : scala.concurrent.Future[play.api.libs.iteratee.Iteratee[Int,Int]] =List()
 val futsum2 = Iteratee.flatten(sum2).run.map(println)
                                                  //> futsum2  : scala.concurrent.Future[Unit] = List()
 Await.ready(futsum2,Duration.Inf)                //> 3
                                                  //| res8: demo.worksheet.enumerator.futsum2.type = Success(())

上面例子里的Enumeratee.take(2)就是一个数据处理的Enumeratee。

现在Enumerator+Enumeratee+Iteratee从功能上越来越像fs2了，当然了，Iteratee就是一个流工具库。我们已经选择了fs2，因为它可以支持灵活的并行运算，所以再深入讨论Iteratee就没什么意义了。