Azure 流分析查询以检测特定 deviceId 的丢失活动事件
Posted
技术标签:
【中文标题】Azure 流分析查询以检测特定 deviceId 的丢失活动事件【英文标题】:Azure Stream Analytics query to detect missing alive event for a specific deviceId 【发布时间】:2018-11-21 16:30:23 【问题描述】:我看不到使用 azure 流分析查询语言分析流是否缺少特定事件的方法。 该流可能包含包含 DeviceId 的 DeviceAlive 和 BeaconDetected 事件,并且在 BeaconDetected 的情况下还包含 BeaconId。现在,如果 DeviceAlive 事件丢失,我想生成一个错误事件。
我怎样才能做到这一点? 我尝试将参考数据与所有有效的 deviceId 一起使用。 但我不允许像这样进行 linq-wise “包含”查询
SELECT * FROM
inputStream
WHERE DeviceId IN (SELECT Id FROM DeviceReferenceData)
我怎样才能做这样的查询。这可以通过将 inputStream 与 DeviceReferenceData 表连接起来吗? 我想我只是看不到明显的东西。
事件,例如看起来像这样:
"EventType": "DeviceAlive",
"DeviceId": "winiot-pi",
"EventEnqueuedUtcTime": "2018-11-19T11:00:20.5594584+01:00"
,
"EventType": "BeaconDetected",
"BeaconId": "2",
"DeviceId": "winiot-pi",
"EventEnqueuedUtcTime": "2018-11-19T11:00:20.5594584+01:00"
进行这样的查询,不会产生预期的结果:
SELECT
iothub.*,r.id as rerId
into output
FROM
iothub TIMESTAMP BY EventEnqueuedUtcTime
Left OUTER Join devicesReference r on iothub.DeviceId = r.Id
这只会为每个 DeviceAlive 事件返回一个 NULL referenceId。 csv中的输出:
eventenqueuedutctime;EventType;BeaconId;DeviceId;SignalStrength;rerid
19.11.2018 10:00:20;BeaconDetected;1;winiot-pi;-40;winiot-pi
19.11.2018 10:00:20;BeaconDetected;1;winiot-pi2;-80;winiot-pi2
19.11.2018 10:00:20;ReceiverDeviceAlive;winiot-pi;winiot-pi;;
19.11.2018 10:00:21;ReceiverDeviceAlive;winiot-pi2;winiot-pi2;;
19.11.2018 10:00:21;BeaconDetected;2;winiot-pi;-80;winiot-pi
19.11.2018 10:00:21;BeaconDetected;2;winiot-pi2;-40;winiot-pi2
19.11.2018 10:00:25;ReceiverDeviceAlive;winiot-pi;winiot-pi;;
19.11.2018 10:00:25;ReceiverDeviceAlive;winiot-pi2;winiot-pi2;;
19.11.2018 10:00:31;BeaconDetected;1;winiot-pi2;-40;winiot-pi2
19.11.2018 10:00:31;BeaconDetected;1;winiot-pi;-80;winiot-pi
19.11.2018 10:00:32;BeaconDetected;2;winiot-pi;-40;winiot-pi
19.11.2018 10:00:32;BeaconDetected;2;winiot-pi;-40;winiot-pi
19.11.2018 10:00:33;BeaconDetected;2;winiot-pi2;-80;winiot-pi2
19.11.2018 10:00:36;BeaconDetected;2;winiot-pi;-40;winiot-pi
19.11.2018 10:00:46;BeaconDetected;2;winiot-pi2;-80;winiot-pi2
19.11.2018 10:00:46;BeaconDetected;2;winiot-pi;-40;winiot-pi
19.11.2018 10:00:57;BeaconDetected;2;winiot-pi2;-80;winiot-pi2
19.11.2018 10:00:57;BeaconDetected;2;winiot-pi;-40;winiot-pi
19.11.2018 10:01:07;BeaconDetected;2;winiot-pi2;-80;winiot-pi2
19.11.2018 10:01:07;BeaconDetected;2;winiot-pi;-40;winiot-pi
19.11.2018 10:01:20;ReceiverDeviceAlive;winiot-pi2;winiot-pi2;;
19.11.2018 10:01:30;ReceiverDeviceAlive;winiot-pi2;winiot-pi2;;
但是如果窗口不包含任何事件,我需要的是每个时间窗口的信息。我们可以把它分解成我猜的那个问题: 如何查询和查看没有任何所需事件的时间窗口。 这完全可能吗?
感谢您的帮助。
【问题讨论】:
【参考方案1】:与同事交谈是一个很好的建议,即使他并不完全理解您在说什么。 ;-) 这是通过在参考设备表的帮助下检测专用设备的 30 秒窗口中是否存在活动事件来生成错误事件的解决方案。 这些链接帮助我更多地了解它:
azure stream analytics query to detect missing alive event for a specific device
how to find absence of signal in a stream analytics job
WITH OneEvent AS /* generate one event per period, any event */
(
SELECT
COUNT(*) As eventCount,
System.Timestamp as EndOfWindow
FROM iothub TIMESTAMP BY EventEnqueuedUtcTime
GROUP BY TumblingWindow(s, 30)
),
AllReferenceDevices AS /* generate one event per deviceId per period */
(
SELECT devicesReference.Id, OneEvent.EndOfWindow
FROM OneEvent JOIN devicesReference
ON OneEvent.EndOfWindow = OneEvent.EndOfWindow
),
/* Select only the devices where we cannot find an event for */
DeviceConnectivityErrorDetection AS
(
SELECT
'DeviceConnectivityErrorDetected' AS EventType,
AllReferenceDevices.Id as FromDeviceId,
AllReferenceDevices.Id as ToDeviceId,
AllReferenceDevices.EndOfWindow as EventEnqueuedUtcTime
FROM
AllReferenceDevices
LEFT join iothub TIMESTAMP BY EventEnqueuedUtcTime
ON DATEDIFF(s, iothub, AllReferenceDevices ) BETWEEN 0 and 30
AND iothub.DeviceId = AllReferenceDevices.Id
WHERE iothub IS NULL
)
SELECT *
INTO ReceiverDeviceConnectivityErrorDetectedOutput
FROM DeviceConnectivityErrorDetection
【讨论】:
【参考方案2】:根据您的要求,也许您可以使用JOIN reference Data 来查找参考数据集合中缺少的 deviceId。请尝试下面的sql:
SELECT
jsoninput.*,r.id as rerId
into output
FROM
jsoninput
Left OUTER Join jsonreference r on jsoninput.id = r.id
where r.id is null
【讨论】:
我对其进行了测试并在问题中添加了更多信息。 我想为了检测“什么都没有”,我需要这样的东西:how-to-query-for-all-events-and-no-event-scenarios,也许这个how-to-find-absence-of-signal-in-a-stream-analytics-job以上是关于Azure 流分析查询以检测特定 deviceId 的丢失活动事件的主要内容,如果未能解决你的问题,请参考以下文章