尝试在 bigtable 中模拟单元级 TTL，但整个列族数据被垃圾收集删除

Posted 2023-02-24

【中文标题】尝试在 bigtable 中模拟单元级 TTL，但整个列族数据被垃圾收集删除

【英文标题】：Trying to simulate cell level TTL in bigtable but whole column family data is getting removed by garbage collection

【发布时间】：2020-04-09 14:07:13

【问题描述】：

使用以下规则创建了一个表： 因此，数据应在 1 秒后过期 (as per docs)

async function createTable() 
    console.log("Creating Table");
    const options = 
        families: [
            
                name: 'payloads',
                rule: 
                    age: 
                        seconds: 1,
                        nanos: 0,
                    ,
                ,
            ,
        ],
    ;
    try 
        await table.create(options);
        console.log("Successfully Created Table");
     catch (err) 
        console.error(`Error reading rows :`, err);
    

然后像这样插入数据：

      const rowsToInsert = 
            key: "SUMEET",
            data: 
                payloads: 
                    '1': 
                        value: "NOTIFICATIONS_PAYLOAD_1",
                        timestamp: 1576500927000,
                    ,
                    '2': 
                        value: "NOTIFICATIONS_PAYLOAD_2",
                        timestamp: 1576587327000,
                    ,
                    '3': 
                        value: "NOTIFICATIONS_PAYLOAD_3",
                        timestamp: 1576673727000,
                    ,
                    '4': 
                        value: "NOTIFICATIONS_PAYLOAD_4",
                        timestamp: 1576760127000,
                    ,
                ,
            ,
        ;

        await table.insert(rowsToInsert);

所以我添加了四个具有不同时间戳的单元格：

首先在写入数据时提前 5 分钟 第二个提前 1 小时 提前 1 天排名第三 第四名，提前 2 天

这里的问题是当 m 读取数据时，整个列族数据被删除，但它应该只根据规则集删除第一个和第二个单元格

有什么遗漏或做错了吗？？

【参考方案1】：

问题在于您的时间戳，您很可能设置了过去的日期。我建议您使用 date methods from javascript 设置日期，而不是像现在这样手动设置它们。

我用以下代码做了一些测试：

const Bigtable = require('@google-cloud/bigtable');
const bigtable = Bigtable();
const instance = bigtable.instance([instance]);
const table = instance.table([table]);
const now = new Date();


async function writeSimple() 
    var now = new Date(Date.now());
    var fiveMinutes = new Date(now.getTime() + 5 * 60000);
    var anHour = new Date(now.getTime() + 60 * 60000);
    var aDay = new Date(now.getTime() + 24 * 60 * 60000);
    var twoDays = new Date(now.getTime() + 48 * 60 * 60000);
  const rowsToInsert = 
            key: "SUMEET",
            data: 
                payloads: 
                    '1': 
                        value: "NOTIFICATIONS_PAYLOAD_1",
                        timestamp: now,
                    ,
                    '2': 
                        value: "NOTIFICATIONS_PAYLOAD_2",
                        timestamp: fiveMinutes,
                    ,
                    '3': 
                        value: "NOTIFICATIONS_PAYLOAD_3",
                        timestamp: anHour,
                    ,
                    '4': 
                        value: "NOTIFICATIONS_PAYLOAD_4",
                        timestamp: aDay,
                    ,
                    '5': 
                        value: "NOTIFICATIONS_PAYLOAD_5",
                        timestamp: twoDays,
                    ,
                ,
            ,
        ;

        await table.insert(rowsToInsert);
        console.log(`Successfully wrote row $rowsToInsert.key`);

得到如下一行：

2019/12/17 16:53:33 -creds flag unset, will use gcloud credential
----------------------------------------
SUMEET
  payloads:1                               @ 2019/12/17-16:30:34.343000
    "NOTIFICATIONS_PAYLOAD_1"
  payloads:2                               @ 2019/12/17-16:35:34.343000
    "NOTIFICATIONS_PAYLOAD_2"
  payloads:3                               @ 2019/12/17-17:30:34.343000
    "NOTIFICATIONS_PAYLOAD_3"
  payloads:4                               @ 2019/12/18-16:30:34.343000
    "NOTIFICATIONS_PAYLOAD_4"
  payloads:5                               @ 2019/12/19-16:30:34.343000
    "NOTIFICATIONS_PAYLOAD_5"

垃圾收集器通过后（大约 15 分钟后）我得到了你想要的结果：

2019/12/17 16:59:47 -creds flag unset, will use gcloud credential
----------------------------------------
SUMEET
  payloads:3                               @ 2019/12/17-17:30:34.343000
    "NOTIFICATIONS_PAYLOAD_3"
  payloads:4                               @ 2019/12/18-16:30:34.343000
    "NOTIFICATIONS_PAYLOAD_4"
  payloads:5                               @ 2019/12/19-16:30:34.343000
    "NOTIFICATIONS_PAYLOAD_5"

我希望你觉得这很有用！

【讨论】：

很高兴能帮上忙 :)

【参考方案2】：

您有以毫秒为单位的时间戳。 Cloud Bigtable 需要微秒（请参阅here）。基本上，将“000”添加到所有时间戳的末尾。