cassandra 2.2.6中的高读写延迟

Posted 2021-05-04

你好类似的问题已经被问过，但我想我们有一些不同的问题：

我们使用Cassandra 2.2.6单节点安装（并将升级到最新版本）。现在，我们遇到了可怕的查询时间，有时会写出超时。

    Read Count: 21554802
    Read Latency: 10.702975718589295 ms.
    Write Count: 19437551
    Write Latency: 27.806026818707767 ms.
    Pending Flushes: 0
            Table: -----
            SSTable count: 5
            Space used (live): 661310370
            Space used (total): 661310370
            Space used by snapshots (total): 704698632
            Off heap memory used (total): 845494
            SSTable Compression Ratio: 0.13491738106721324
            Number of keys (estimate): 179623
            Memtable cell count: 594836
            Memtable data size: 8816212
            Memtable off heap memory used: 0
            Memtable switch count: 3343
            Local read count: 21554802
            Local read latency: 11,744 ms
            Local write count: 19437551
            Local write latency: 30,506 ms
            Pending flushes: 0
            Bloom filter false positives: 387
            Bloom filter false ratio: 0,00024
            Bloom filter space used: 258368
            Bloom filter off heap memory used: 258328
            Index summary off heap memory used: 34830
            Compression metadata off heap memory used: 552336
            Compacted partition minimum bytes: 180
            Compacted partition maximum bytes: 12108970
            Compacted partition mean bytes: 23949
            Average live cells per slice (last five minutes): 906.8858219156                                                       92
            Maximum live cells per slice (last five minutes): 182785
            Average tombstones per slice (last five minutes): 1.432102507830                                                       9697
            Maximum tombstones per slice (last five minutes): 50

为了比较，有一个不同的表包含大约10M记录，并且构造与上面的非常类似

    Read Count: 815780599
    Read Latency: 0.1672932019580917 ms.
    Write Count: 3083462
    Write Latency: 1.5470194706469547 ms.
    Pending Flushes: 0
            Table: ------
            SSTable count: 9
            Space used (live): 5067447115
            Space used (total): 5067447115
            Space used by snapshots (total): 31810631860
            Off heap memory used (total): 19603932
            SSTable Compression Ratio: 0.2952622065160448
            Number of keys (estimate): 12020796
            Memtable cell count: 300611
            Memtable data size: 18020553
            Memtable off heap memory used: 0
            Memtable switch count: 97
            Local read count: 815780599
            Local read latency: 0,184 ms
            Local write count: 3083462
            Local write latency: 1,692 ms
            Pending flushes: 0
            Bloom filter false positives: 7
            Bloom filter false ratio: 0,00000
            Bloom filter space used: 15103552
            Bloom filter off heap memory used: 15103480
            Index summary off heap memory used: 2631412
            Compression metadata off heap memory used: 1869040
            Compacted partition minimum bytes: 925
            Compacted partition maximum bytes: 1916
            Compacted partition mean bytes: 1438
            Average live cells per slice (last five minutes): 1.0
            Maximum live cells per slice (last five minutes): 1
            Average tombstones per slice (last five minutes): 1.0193396020053622
            Maximum tombstones per slice (last five minutes): 3

差异是第一个包含大量地图和UDT。在开发中心进行简单测试select * from ... limit 999; （省略任何Lucene索引等）最后一个显示183ms，第一个显示1.8s。

任何人都可以帮助定义找到根本原因的方法？

答案

每片最大活细胞数（最后五分钟）：182785

这很大，可能来自你的地图和UDT。您的数据模型很可能是根本原因。步行直播180k单元格以满足单个查询将非常缓慢。

select * from ... limit 999;

范围查询本质上很慢。尝试设计表格，以便您可以从单个分区回答问题，您将获得更好的结果。

一个节点安装

每当有一个GC你会得到一个糟糕的查询，这可以通过添加更多的节点来减轻这种情况，这样暂停就不会受到伤害（如果在驱动程序上使用客户端推测性重试则更好）。