使用嵌套行（类型 STRUCT）对表 SQL 进行重复数据删除

Posted

技术标签:

【中文标题】使用嵌套行（类型 STRUCT）对表 SQL 进行重复数据删除【英文标题】：Deduplicate table SQL with nested rows (type STRUCT) 【发布时间】：2020-01-19 07:28:49 【问题描述】：

我有一个包含可能重复行的 SQL 表（在 BigQuery 中）。该表有 20 多列，其中一些是嵌套的（数据类型“STRUCT”）。我想对表进行重复数据删除。

我不能简单地查询SELECT DISTINCT *，因为我得到一个错误

查询错误：STRUCT 类型的列选项不能在 SELECT DISTINCT 中使用

到目前为止，我尝试根据某些列的哈希创建唯一 ID。 我现在有了这个唯一的 ID（称为sha256），但我想不出一种只选择具有唯一哈希的行的方法。

我尝试GROUP BY，但它不适用于STRUCT 类型，我还尝试INNER JOIN 使用仅包含唯一哈希的表，但我也得到重复。

作为参考，以下是数据集的 2 行示例：

  
    "sha256": "un2k3TUtzwzmQMvxfrjztsh/A/GW3WWzV4U4CezqceA=",
    "has_phone": true,
    "options": 
      "sub_toplist": true,
      "gallery": false,
      "urgent": false,
      "has_option": true,
      "photosup": true,
      "booster": false
    ,
    "calendar": 
      "dates": null
    ,
    "owner": 
      "siren": null,
      "pro_rates_link": null,
      "user_id": "f0d94687-1a24-4ed4-8adb-7faded053ca8",
      "type": "private",
      "no_salesmen": true,
      "name": "marius",
      "store_id": "5022456"
    ,
    "location": 
      "feature": 
        "properties": null,
        "geometry": 
          "coordinates": [
            "9.41733",
            "42.54701"
          ],
          "type": "Point"
        ,
        "type": "Feature"
      ,
      "is_shape": true,
      "provider": "here",
      "lng": "9.41733",
      "lat": "42.54701",
      "zipcode": "20290",
      "city_label": "Lucciana 20290",
      "city": "Lucciana",
      "region_name": "Corse",
      "department_name": null,
      "source": "city",
      "department_id": "0",
      "region_id": "9"
    ,
    "attributes": 
      "pro_rates_link": null,
      "immo_sell_type": "old",
      "ges": "a",
      "square": "92",
      "rooms": "4",
      "energy_rate": "b",
      "is_import": false,
      "custom_ref": null,
      "lease_type": "sell",
      "real_estate_type": "1",
      "fai_included": null,
      "type_real_estate_sale": null
    ,
    "price_calendar": null,
    "price": [
      "270000"
    ],
    "body": "text",
    "url": "https://www.example.fr/ventes_immobilieres/1729537955.htm",
    "category_name": "Ventes immobilières",
    "category_id": "9",
    "images": 
      "urls_thumb": [
        "https://img3.example.fr/ad-thumb/d63e236ce3546906b3ce661640a7cf858d0a0593.jpg"
      ],
      "urls": [
        "https://img3.example.fr/ad-image/ac6bd9ce0cc3aa507727ddece51f437d77ae4cfa.jpg",
      ],
      "nb_images": "7",
      "small_url": "https://img3.example.fr/ad-small/ac6bd9ce0cc3aa507727ddece51f437d77ae4cfa.jpg",
      "thumb_url": "https://img3.example.fr/ad-thumb/ac6bd9ce0cc3aa507727ddece51f437d77ae4cfa.jpg"
    ,
    "ad_type": "offer",
    "first_publication_date": "2020-01-02 15:00:46 UTC",
    "status": "active",
    "subject": "Villa à Lucciana",
    "index_date": "2020-01-16 15:00:45 UTC",
    "expiration_date": "2020-03-02 15:00:46 UTC",
    "list_id": "1729537955"
  ,
  
    "sha256": "wCMrggkqSJ3PgbkuWAgBpCMtFfkJDRlz6TOeO5Nngsg=",
    "has_phone": true,
    "options": 
      "sub_toplist": false,
      "gallery": false,
      "urgent": false,
      "has_option": false,
      "photosup": false,
      "booster": false
    ,
    "calendar": 
      "dates": null
    ,
    "owner": 
      "siren": null,
      "pro_rates_link": null,
      "user_id": "ae0f432d-0aa2-4828-a20b-3472255588b4",
      "type": "private",
      "no_salesmen": true,
      "name": "M.Milleliri",
      "store_id": "12132533"
    ,
    "location": 
      "feature": 
        "properties": null,
        "geometry": 
          "coordinates": [
            "9.1917",
            "41.54506"
          ],
          "type": "Point"
        ,
        "type": "Feature"
      ,
      "is_shape": true,
      "provider": "here",
      "lng": "9.1917",
      "lat": "41.54506",
      "zipcode": "20146",
      "city_label": "Sotta 20146",
      "city": "Sotta",
      "region_name": "Corse",
      "department_name": null,
      "source": "city",
      "department_id": "0",
      "region_id": "9"
    ,
    "attributes": 
      "pro_rates_link": null,
      "immo_sell_type": "old",
      "ges": "Non renseigné",
      "square": null,
      "rooms": null,
      "energy_rate": "Non renseigné",
      "is_import": false,
      "custom_ref": null,
      "lease_type": "sell",
      "real_estate_type": "3",
      "fai_included": null,
      "type_real_estate_sale": null
    ,
    "price_calendar": null,
    "price": [
      "100000"
    ],
    "body": "text",
    "url": "https://www.example.fr/ventes_immobilieres/1736199673.htm",
    "category_name": "Ventes immobilières",
    "category_id": "9",
    "images": 
      "urls_thumb": [
        "https://img3.example.fr/ad-thumb/4f3632dc8e5c50075aa6c6e4b559e2042546f009.jpg"
      ],
      "urls": [
        "https://img3.example.fr/ad-image/4f3632dc8e5c50075aa6c6e4b559e2042546f009.jpg"
      ],
      "urls_large": [
        "https://img3.example.fr/ad-large/4f3632dc8e5c50075aa6c6e4b559e2042546f009.jpg"
      ],
      "nb_images": "1",
      "small_url": "https://img3.example.fr/ad-small/4f3632dc8e5c50075aa6c6e4b559e2042546f009.jpg",
      "thumb_url": "https://img3.example.fr/ad-thumb/4f3632dc8e5c50075aa6c6e4b559e2042546f009.jpg"
    ,
    "ad_type": "offer",
    "first_publication_date": "2020-01-16 14:21:05 UTC",
    "status": "active",
    "subject": "Terrain 1250 m2 Sotta",
    "index_date": "2020-01-16 14:21:05 UTC",
    "expiration_date": "2020-03-16 14:21:05 UTC",
    "list_id": "1736199673"

以及我目前正在处理的查询：

WITH
  table_unique_hash AS (
  SELECT
    DISTINCT(SHA256(CONCAT(FORMAT_TIMESTAMP('%Y/%m/%d_%H:%M:%S_', index_date), CAST(list_id AS STRING)))) AS sha256
  FROM
    `test_bucket_data.daily_table`),

  table_hashed AS (
  SELECT
    SHA256(CONCAT(FORMAT_TIMESTAMP('%Y/%m/%d_%H:%M:%S_', index_date), CAST(list_id AS STRING))) AS sha256, *
  FROM
    `test_bucket_data.daily_table`)
SElECT * FROM table_hashed
limit 10;

解决方案是在 sha256 列上找到一种内部连接 ​​table_hashed 和 table_unique_hash 的方法...

感谢您的帮助！

【问题讨论】：

。 .我（也许是“the”）限制结构的一般解决方案是将值转换为 JSON。字符串一般都很好。也就是说，转换回嵌套结构可能有点痛苦。 【参考方案1】：

我找到了基于this topic 的解决方法。所有 STRUCT 列的 GROUP BY 和 ANY 函数的组合使其工作！

SELECT
  has_phone,
  ANY_VALUE(options) as options,
  ANY_VALUE(calendar) as calendar,
  ANY_VALUE(owner) as owner,
  ANY_VALUE(location) as location,
  ANY_VALUE(attributes) as attributes,
  price_calendar,
  price,
  body,
  url,
  category_name,
  category_id,
  ANY_VALUE(images) as images,
  ad_type,
  first_publication_date,
  status,
  subject,
  index_date,
  expiration_date,
  list_id,
FROM
  `table_name`
Group by
  has_phone,
  price_calendar,
  price,
  body,
  url,
  category_name,
  category_id,
  ad_type,
  first_publication_date,
  status,
  subject,
  index_date,
  expiration_date,
  list_id

注意：我的“价格”字段以前是array；我将它在我的源 json 中转换为 int

【讨论】：

以上是关于使用嵌套行（类型 STRUCT）对表 SQL 进行重复数据删除的主要内容，如果未能解决你的问题，请参考以下文章

如何在sqlite中对表中的行进行排名？

SQL基础教程（第2版）第7章 集合运算：7-2 联结（以列为单位对表进行联结）

实例成员不能用于 struct swift 之间的嵌套类型的实例

我想按姓氏对表进行分区？ [关闭]

如何将嵌套的Struct列展开为多列？

在 BigQuery 中使用双数组取消嵌套 STRUCT