Why Apache Beam? A data Artisans perspective

Posted 2020-08-01 fxjwind

https://cloud.google.com/dataflow/blog/dataflow-beam-and-spark-comparison

https://github.com/apache/incubator-beam

https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101

https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102

http://data-artisans.com/why-apache-beam/

 

As the Dataflow SDK and the Runners were moving to Apache Incubator as Apache Beam, we were asked by Google to bring the Flink runner into the codebase of Beam, and become committers and PMC members in the new project. We decided to go full st(r)eam ahead with this opportunity as we believe that

(1) the Beam model is the future reference programming model for writing data applications in both stream and batch, and

(2) Flink is the definitive platform to execute these data applications. As Beam is now taking shape, Flink is currently the only practical execution engine for Beam programs outside Google’s Cloud.

Beam包括，SDK部分和runner部分，而在Google外，当前Flink是一个可以作为runner的比较好的选择

 

Flink and Beam are completely aligned in their concepts, which makes the translation of Beam programs to Flink jobs both straightforward and very efficient.
With full support for concepts such as event time, watermarks, and triggers, and with the new features we are contributing to Flink, we believe that the superiority of the Flink runner will stick for the foreseeable future.

Flink和beam的设计和概念都很相似，所以从Beam programs翻译到Flink jobs 是非常直接的。

 

One question that remains is what is the relationship between Flink’s own native API (DataStream), and the Beam API?

Will both of these continue to be supported, and is it confusing for developers to have two different APIs that in the end generate Flink jobs?
Our committers at data Artisans will continue to fully support both the Flink DataStream API (which is, as of Flink 1.0, stable and backwards compatible), as well as the Beam API as it evolves for Beam programs that run on Flink.

The differences between the two APIs are largely syntactical (and a matter of taste), and, we are working together with Google towards unifying the APIs, with the end goal of making the Beam and Flink APIs source compatible.
We believe that the two communities can learn from each other, and we encourage users to use either of the two APIs to implement their Flink jobs for stream data processing.

With the native Flink DataStream API you get an already mature and backwards-compatible API, built-in libraries (e.g., CEP and upcoming SQL), mature tooling and connectors, key-value state (with the ability to query that state in the future), and an API which fully utilizes all the features of Flink’s powerful engine.
With the Beam API, you get the option of portability down the line as more Beam runners mature.

Flink API和Beam API的区别是什么？
区别主要是语法上的，并且会致力于unify两种接口，但两种接口会都有其存在的价值

API, model, and engine

To clarify our points above, we would like to explain what we mean by choice of API, choice of programming model, and choice of execution engine.

Currently, Beam has three available runners: the Google Cloud Dataflow proprietary runner by Google, as well as the Flink and Spark runners, included in the open source Apache Beam project. Let us look at this ecosystem, and add Flink and Spark themselves with their native APIs, as well as Storm:

可以看出Beam的好处，关键在于，API和Model的统一，虽然Engine可以是不一样的

 

Even more, Google and data Artisans are working together to make the two APIs semantically equivalent, ironing out any minor inconsistencies.
This means that users of either API can switch with relatively low effort.
Our long term goal is to make the Beam and Flink DataStream APIs source-compatible, so that programs written in one can natively run on the other with no code changes.

If you choose to invest in the Beam  programming model now, you have two options:

1. Use the Flink DataStream API in Java and Scala
2. Use the Beam API directly in Java (and  soon Python) with the Flink runner

 

长期看， Beam和Flink的API会兼容，所以如果想用Beam编程模型，可以有两种选择，直接用Flink DataStream API或用Beam API加上Flink Runner

 

We recommend option 1 to users that want to get started immediately, using an already mature and backwards-compatible API, access to libraries (e.g., the existing CEP library and the upcoming SQL functionality), mature tooling and connectors (e.g., to Kafka), as well as an API that fully and natively utilizes all the existing and upcoming features of the Flink engine. In addition, we recommend the Flink native API for use cases that use Flink’s key-value state abstraction, and in the future Flink’s facilities for querying that state.

We recommend option 2 to users that want to keep the option of engine portability (as other Beam runners progress).

选择BeamAPI的唯一好处是，可以做到各个runner的兼容；而显然，Flink的API更丰富和成熟

 

In a recent blog post, Google compared Beam and Spark Streaming from a programming model perspective. They took a mobile gaming scenario, and implemented several use cases in Beam and Spark Streaming, focusing their analysis on how well are the following concerns separated in the code:

• What results are calculated? Sums, joins, histograms, machine learning models?
• Where in event time are results calculated? Does the time each event originally occurred affect results? Are results aggregated in fixed windows, sessions, or a single global window?
• When in processing time are results materialized? Does the time each event is observed within the system affect results? When are results emitted? Speculatively, as data evolve? When data arrive late and results must be revised? Some combination of these?
• How do refinements of results relate? If additional data arrive and results change, are they independent and distinct, do they build upon one another, etc.?

 

 

用一个实际例子比较一下，Flink和Beam API的不同，不同颜色部分解决不同的问题，一共4个问题

 

Conclusion

We firmly believe that the Beam model is the correct programming model for streaming and batch data processing.

We encourage users to adopt this model for their future data applications, embodied in either the Beam API itself or the Flink DataStream API.

Further, we believe that Flink, with its current features and roadmap, is currently the most advanced open source stream processor, and at the same time the only practical solution for deploying Beam programs in production on on-premise or non-GCP clusters. We are looking forward to continue pushing the envelope in stream processing and enabling enterprises to use stream processing technology for their data applications.