StreamSets 设计Edge pipeline

Posted rongfengliang

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了StreamSets 设计Edge pipeline相关的知识,希望对你有一定的参考价值。

edge pipeline 运行在edge 执行模式,我们可以使用 data collector UI 进行edge pipeline 设计,
设计完成之后,你可以部署对应的pipeline到edge 设备

可以设计的edge pipeline

  • edge 发送pipeline
edge 发送pipeline 使用特定的orgin读取edge设备上的数据,这个pipeline 可以在将数据发送到data collector 之前进行数据的处理
  • edge 接收pipeline
接收pipeline可以接收来自edge 设备或者 data collector pipeline的数据

orgin 组件

  • Dev Random Record Source
  • Dev Raw Data Source
  • Directory Edge pipelines do not support multithreaded processing.
    In an edge pipeline, the Directory origin always creates a single thread to read the files even if you configure it to use multiple threads.
  • File Tail
    In edge pipelines, the File Tail origin can read a single set of files.
    If you configure multiple sets of files for the origin, the origin reads only the files configured in the first set.
  • HTTP Client
    In edge pipelines, the HTTP Client origin does not support batch processing mode, pagination, or OAuth2 authorization.
  • HTTP Server Edge pipelines do not support multithreaded processing.
    In an edge pipeline, the HTTP Server origin always creates a single thread to read the files even if you configure it to use multiple threads.
  • MQTT Subscriber Edge pipelines that use MQTT stages require using an intermediary MQTT broker.
    For example, an edge sending pipeline uses an MQTT Publisher destination to write to an MQTT broker. The MQTT broker temporarily stores the data until the MQTT Subscriber origin in the edge receiving pipeline reads the data.
  • Sensor Reader
  • System Metrics
  • WebSocket Client
  • Windows Event Log

processsor 组件

  • Delay
  • Dev Identity
  • Expression Evaluator
  • Field Remover
  • javascript Evaluator In edge pipelines, the JavaScript Evaluator processor does not support the sdcFunctions scripting object.
  • Stream Selector

destinations 组件

  • CoAP Client
  • HTTP Client
  • Kafka Producer
  • MQTT Publisher Edge pipelines that use MQTT stages require using an intermediary MQTT broker.
    For example, an edge sending pipeline uses an MQTT Publisher destination to write to an MQTT broker. The MQTT broker temporarily stores the data until the MQTT Subscriber origin in the Data Collector receiving pipeline reads the data.
  • Trash
  • WebSocket Client

错误记录处理

  • Discard 丢踢
    The pipeline discards the record.
  • Write to File 写到文件
    The pipeline writes error records and related details to a local directory on the edge device. Create another edge pipeline with a Directory origin to process the error records written to the file.
  • Write to MQTT 写到mqtt
    The pipeline publishes error records and related details to a topic on an MQTT broker. Create another edge or standalone Data Collector pipeline with an MQTT Subscriber origin to process the error records published to the broker.

支持的数据格式

  • json
  • text

限制

  • Email and webhook notifications cannot be sent by edge pipelines.
  • Rules and alerts cannot be defined for edge pipelines.
  • Edge pipelines support a limited number of record, math, pipeline, and string functions.
  • Edge pipelines do not support dataflow triggers.
  • Edge pipelines do not support multithreaded processing.
  • You cannot capture snapshots for edge pipelines.

参考资料

https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Edge_Mode/EdgePipelineTypes.html#concept_c14_m4r_4bb

以上是关于StreamSets 设计Edge pipeline的主要内容,如果未能解决你的问题,请参考以下文章

Streamsets官方文档--Pipeline的概念和设计

streamsets安装部署

StreamSets学习系列之StreamSets是什么?

StreamSets学习系列之StreamSets的集群安装(图文详解)

StreamSets学习系列之StreamSets的Create New Pipeline(图文详解)

StreamSets学习系列之StreamSets的Core Tarball方式安装(图文详解)