StreamSets 设计Edge pipeline

Posted 2020-12-31 rongfengliang

edge pipeline 运行在edge 执行模式，我们可以使用 data collector UI 进行edge pipeline 设计，
设计完成之后，你可以部署对应的pipeline到edge 设备

可以设计的edge pipeline

• edge 发送pipeline

edge 发送pipeline 使用特定的orgin读取edge设备上的数据，这个pipeline 可以在将数据发送到data collector 之前进行数据的处理

• edge 接收pipeline

接收pipeline可以接收来自edge 设备或者 data collector pipeline的数据

orgin 组件

• Dev Random Record Source
• Dev Raw Data Source
• Directory Edge pipelines do not support multithreaded processing.
  In an edge pipeline, the Directory origin always creates a single thread to read the files even if you configure it to use multiple threads.
• File Tail
  In edge pipelines, the File Tail origin can read a single set of files.
  If you configure multiple sets of files for the origin, the origin reads only the files configured in the first set.
• HTTP Client
  In edge pipelines, the HTTP Client origin does not support batch processing mode, pagination, or OAuth2 authorization.
• HTTP Server Edge pipelines do not support multithreaded processing.
  In an edge pipeline, the HTTP Server origin always creates a single thread to read the files even if you configure it to use multiple threads.
• MQTT Subscriber Edge pipelines that use MQTT stages require using an intermediary MQTT broker.
  For example, an edge sending pipeline uses an MQTT Publisher destination to write to an MQTT broker. The MQTT broker temporarily stores the data until the MQTT Subscriber origin in the edge receiving pipeline reads the data.
• Sensor Reader
• System Metrics
• WebSocket Client
• Windows Event Log

processsor 组件

• Delay
• Dev Identity
• Expression Evaluator
• Field Remover
• javascript Evaluator In edge pipelines, the JavaScript Evaluator processor does not support the sdcFunctions scripting object.
• Stream Selector

destinations 组件

• CoAP Client
• HTTP Client
• Kafka Producer
• MQTT Publisher Edge pipelines that use MQTT stages require using an intermediary MQTT broker.
  For example, an edge sending pipeline uses an MQTT Publisher destination to write to an MQTT broker. The MQTT broker temporarily stores the data until the MQTT Subscriber origin in the Data Collector receiving pipeline reads the data.
• Trash
• WebSocket Client

错误记录处理

• Discard 丢踢
  The pipeline discards the record.
• Write to File 写到文件
  The pipeline writes error records and related details to a local directory on the edge device. Create another edge pipeline with a Directory origin to process the error records written to the file.
• Write to MQTT 写到mqtt
  The pipeline publishes error records and related details to a topic on an MQTT broker. Create another edge or standalone Data Collector pipeline with an MQTT Subscriber origin to process the error records published to the broker.

支持的数据格式

• json
• text

限制

• Email and webhook notifications cannot be sent by edge pipelines.
• Rules and alerts cannot be defined for edge pipelines.
• Edge pipelines support a limited number of record, math, pipeline, and string functions.
• Edge pipelines do not support dataflow triggers.
• Edge pipelines do not support multithreaded processing.
• You cannot capture snapshots for edge pipelines.

参考资料

https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Edge_Mode/EdgePipelineTypes.html#concept_c14_m4r_4bb