
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.
Apache Flink 是一个框架和分布式处理引擎,用于在无边界和有边界数据流上进行有状态的计算。Flink 能在所有常见集群环境中运行,并能以内存速度和任意规模进行计算。
Here, we explain important aspects of Flink’s architecture.
接下来,我们来介绍一下 Flink 架构中的几个重要方面。
Any kind of data is produced as a stream of events. Credit card transactions, sensor measurements, machine logs, or user interactions on a website or mobile application, all of these data are generated as a stream.
任何类型的数据都是以事件流的形式产生的。信用卡交易、传感器测量、机器日志、网站或移动应用上的用户交互记录,所有这些数据都以流的形式产生。
Data can be processed as unbounded or bounded streams.
数据可以被作为无界流或者有界流来处理。
Unbounded streams have a start but no defined end. They do not terminate and provide data as it is generated. Unbounded streams must be continuously processed, i.e., events must be promptly handled after they have been ingested. It is not possible to wait for all input data to arrive because the input is unbounded and will not be complete at any point in time. Processing unbounded data often requires that events are ingested in a specific order, such as the order in which events occurred, to be able to reason about result completeness.无界流有定义流的开始,却没有定义流的结束。它们一旦被定义就会无休止地产生数据。无界流必须被持续处理,即事件需要在被摄取后迅速处理。不能等到所有数据都到达再处理,因为输入是无限的,并且在任何时候都不会完成。处理无界数据通常要求以特定的顺序摄取事件,例如事件发生的顺序,以便能够推断出结果的完整性。
Bounded streams have a defined start and end. Bounded streams can be processed by ingesting all data before performing any computations. Ordered ingestion is not required to process bounded streams because a bounded data set can always be sorted. Processing of bounded streams is also known as batch processing.有界流有定义流的开始,也有定义流的结束。有界流可以在摄取所有数据之后再进行计算。有界流数据集可以被排序,所以并不需要有序摄取。有界流处理通常被称为批处理。

Apache Flink excels at processing unbounded and bounded data sets. Precise control of time and state enable Flink's runtime to run any kind of application on unbounded streams. Bounded streams are internally processed by algorithms and data structures that are specifically designed for fixed sized data sets, yielding excellent performance.
Apache Flink 擅长处理无界和有界数据集。对时间和状态的精确控制使得 Flink Runtime 能够运行任何处理无界流的应用。有界流则由一些专为固定大小数据集特殊设计的算法和数据结构进行内部处理,产生了出色的性能。
Apache Flink is a distributed system and requires compute resources in order to execute applications. Flink integrates with all common cluster resource managers such as Hadoop YARN, Apache Mesos, and Kubernetes but can also be setup to run as a stand-alone cluster.
Apache Flink 是一个分布式系统,它需要计算资源来执行应用程序。Flink 集成了所有常见的集群资源管理器,例如 Hadoop YARN、 Apache Mesos 和 Kubernetes,但同时也可以作为独立集群运行。
Flink is designed to work well on each of the previously listed resource managers. This is achieved by resource-manager-specific deployment modes that allow Flink to interact with each resource manager in its idiomatic way.
Flink 被设计成能够在上述每个资源管理器中很好地工作。这是通过资源管理器特定的部署模式实现的,该部署模式允许 Flink 以其惯用方式与每个资源管理器进行交互。
When deploying a Flink application, Flink automatically identifies the required resources based on the application’s configured parallelism and requests them from the resource manager. In case of a failure, Flink replaces the failed container by requesting new resources. All communication to submit or control an application happens via REST calls. This eases the integration of Flink in many environments.
在部署 Flink 应用程序时,Flink 会根据应用程序配置的并行度自动标识所需的资源,并向资源管理器请求这些资源。在发生故障的情况下,Flink 通过请求新资源来替换发生故障的容器。提交或控制应用程序的所有通信都是通过 REST 调用进行的,这简化了 Flink 在各种环境中的集成。
Flink is designed to run stateful streaming applications at any scale. Applications are parallelized into possibly thousands of tasks that are distributed and concurrently executed in a cluster. Therefore, an application can leverage virtually unlimited amounts of CPUs, main memory, disk and network IO. Moreover, Flink easily maintains very large application state. Its asynchronous and incremental checkpointing algorithm ensures minimal impact on processing latencies while guaranteeing exactly-once state consistency.
Flink 旨在以任意规模运行有状态的流式应用程序。应用程序被并行化为可能是数以千计的任务,这些任务分布在集群中并发执行。所以,应用程序能够利用几乎无尽的 CPU、内存、磁盘和网络 IO。此外,Flink 可以轻松地维护非常大的应用程序状态。它的异步和增量检查点算法确保了对处理延迟的影响最小,同时保证精确一次的状态一致性。
Users reported impressive scalability numbers for Flink applications running in their production environments, such as
applications processing multiple trillions of events per day,
applications maintaining multiple terabytes of state,
applications running on thousands of cores.
Flink 用户报告了其生产环境中一些令人印象深刻的扩展性数字,例如
应用程序处理每天处理数万亿的事件,
应用程序维护几TB大小的状态,
应用程序在数千个内核上运行。
Stateful Flink applications are optimized for local state access. Task state is always maintained in memory or, if the state size exceeds the available memory, in access-efficient on-disk data structures. Hence, tasks perform all computations by accessing local, often in-memory, state yielding very low processing latencies. Flink guarantees exactly-once state consistency in case of failures by periodically and asynchronously checkpointing the local state to durable storage.
有状态的 Flink 程序针对本地状态访问进行了优化。任务的状态始终保留在内存中,如果状态大小超过可用内存,则会保存在能高效访问的磁盘数据结构中。因此,任务通过访问本地,通常是在内存中的,状态来进行所有的计算,从而产生非常低的处理延迟。Flink 通过定期和异步地检查点对本地状态进行持久化存储来保证故障场景下精确一次的状态一致性。
