Redis数据结构之HperLogLog
Posted 三石雨
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Redis数据结构之HperLogLog相关的知识，希望对你有一定的参考价值。
一、HyperLogLog
HyperLogLog是用来做基数统计的。
其可以非常省内存的去统计各种计数，比如注册ip数、每日访问IP数、页面实时UV（PV肯定字符串就搞定了）、在线用户数等在对准确性不是很重要的应用场景。
HyperLogLog的优点是：
在输入元素的数量或者体积非常非常大时，计算基数所需的空间总是固定的、并且是很小的，
HyperLogLog的缺点:
它是估计基数的算法，所以会有一定误差0.81%。
每个HyperLogLog键只需要花费12KB内存，就可以计算接近264个不同元素的基数。这和计算基数时，元素越多耗费内存就越多的集合形成鲜明对比。
但是，因为 HyperLogLog 只会根据输入元素来计算基数，而不会储存输入元素本身，所以 HyperLogLog 不能像集合那样，返回输入的各个元素即无法知道统计的详细内容。
二、基数和估算值
1、基数
基数是集合中不同元素的数量。
比如数据集 {1, 3, 5, 7, 5, 7, 8}， 那么这个数据集的基数集为 {1, 3, 5 ,7, 8}, 基数(不重复元素)为5。
基数估计就是在误差可接受的范围内，快速计算基数。
2、估算值
算法给出的基数并不是精确的，可能会比实际稍微多一些或者稍微少一些，但会控制在合理的范围之内。
三、HperLogLog基本命令
redis HyperLogLog 的基本命令：
1 PFADD key element [element ...]
添加指定元素到 HyperLogLog 中。
2 PFCOUNT key [key ...]
返回给定 HyperLogLog 的基数估算值。
3 PFMERGE destkey sourcekey [sourcekey ...]
将多个 HyperLogLog 合并为一个 HyperLogLog
PFADD
将任意数量的元素添加到指定的 HyperLogLog 里面。在执行这个命令之后，HyperLogLog内部的结构会被更新，并有所反馈，
如果执行完之后HyperLogLog内部的基数估算发生了变化，那么就会返回1，否则（认为已经存在）就返回0。
这个命令还有一个比较神器的就是可以只有键，没有值，这样的意思就是只是创建空的键，不放值。
如果这个键存在，不做任何事情，返回0；不存在的话就创建，并返回1。
这个命令的时间复杂度为O(1)，所以就放心用吧~
PFCOUNT
当命令作用于单个键的时候，返回这个键的基数估算值。如果键不存在，则返回0。
当 PFCOUNT 命令作用于多个键时， 返回所有给定 HyperLogLog 的并集的近似基数， 这个近似基数是通过将所有给定 HyperLogLog 合并至一个临时 HyperLogLog 来计算得出的。
这个命令在作用于单个值的时候，时间复杂度为O(1)，并且具有非常低的平均常数时间；在作用于N个值的时候，时间复杂度为O(N)，这个命令的常数复杂度会比较低些。
命令返回的可见集合（observed set）基数并不是精确值， 而是一个带有 0.81% 标准错误（standard error）的近似值。
举个例子， 为了记录一天会执行多少次各不相同的搜索查询， 一个程序可以在每次执行搜索查询时调用一次 PFADD ， 并通过调用 PFCOUNT 命令来获取这个记录的近似结果。
PFMERGE
合并（merge）多个HyperLogLog为一个HyperLogLog。 合并后的 HyperLogLog 的基数接近于所有输入 HyperLogLog 的可见集合（observed set）的并集。
合并得出的 HyperLogLog 会被储存在 destkey 键里面， 如果该键并不存在， 那么命令在执行之前， 会先为该键创建一个空的 HyperLogLog 。
这个命令的第一个参数为目标键，剩下的参数为要合并的HyperLogLog。命令执行时，如果目标键不存在，则创建后再执行合并。
这个命令的时间复杂度为O(N)，其中N为要合并的HyperLogLog的个数。不过这个命令的常数时间复杂度比较高。
redis> PFADD ip:20170626 "192.168.0.10" "192.168.0.20" "192.168.0.30"
(integer) 1
redis> PFADD ip:20170626 "192.168.0.20" "192.168.0.40" "192.168.0.50" # 存在就只加新的
(integer) 1
redis> PFCOUNT ip:20170626 # 元素估计数量没有变化
(integer) 5
redis> PFADD ip:20170626 "192.168.0.20" # 存在就不会增加
(integer) 0
edis> PFMERGE ip:20170626 ip:20170627 ip:20170628
OK
redis> PFCOUNT ip:201706
(integer) 5
四、hperloglog 描述
由于hperloglog，这种数据结构在实际应用场景中并不多。因此，这里就不再详细讨论了。
我们看下hperloglog.c文件，对HperLogLog的描述
/* The Redis HyperLogLog implementation is based on the following ideas:
*
* * The use of a 64 bit hash function as proposed in [1], in order to don‘t
* limited to cardinalities up to 10^9, at the cost of just 1 additional
* bit per register.
* * The use of 16384 6-bit registers for a great level of accuracy, using
* a total of 12k per key.
* * The use of the Redis string data type. No new type is introduced.
* * No attempt is made to compress the data structure as in [1]. Also the
* algorithm used is the original HyperLogLog Algorithm as in [2], with
* the only difference that a 64 bit hash function is used, so no correction
* is performed for values near 2^32 as in [1].
*
* [1] Heule, Nunkesser, Hall: HyperLogLog in Practice: Algorithmic
* Engineering of a State of The Art Cardinality Estimation Algorithm.
*
* [2] P. Flajolet, éric Fusy, O. Gandouet, and F. Meunier. Hyperloglog: The
* analysis of a near-optimal cardinality estimation algorithm.
*
* Redis uses two representations:
*
* 1) A "dense" representation where every entry is represented by
* a 6-bit integer.
* 2) A "sparse" representation using run length compression suitable
* for representing HyperLogLogs with many registers set to 0 in
* a memory efficient way.
*
*
* HLL header
* ===
*
* Both the dense and sparse representation have a 16 byte header as follows:
*
* +------+---+-----+----------+
* | HYLL | E | N/U | Cardin. |
* +------+---+-----+----------+
*
* The first 4 bytes are a magic string set to the bytes "HYLL".
* "E" is one byte encoding, currently set to HLL_DENSE or
* HLL_SPARSE. N/U are three not used bytes.
*
* The "Cardin." field is a 64 bit integer stored in little endian format
* with the latest cardinality computed that can be reused if the data
* structure was not modified since the last computation (this is useful
* because there are high probabilities that HLLADD operations don‘t
* modify the actual data structure and hence the approximated cardinality).
*
* When the most significant bit in the most significant byte of the cached
* cardinality is set, it means that the data structure was modified and
* we can‘t reuse the cached value that must be recomputed.
*
* Dense representation
* ===
*
* The dense representation used by Redis is the following:
*
* +--------+--------+--------+------// //--+
* |11000000|22221111|33333322|55444444 .... |
* +--------+--------+--------+------// //--+
*
* The 6 bits counters are encoded one after the other starting from the
* LSB to the MSB, and using the next bytes as needed.
*
* Sparse representation
* ===
*
* The sparse representation encodes registers using a run length
* encoding composed of three opcodes, two using one byte, and one using
* of two bytes. The opcodes are called ZERO, XZERO and VAL.
*
* ZERO opcode is represented as 00xxxxxx. The 6-bit integer represented
* by the six bits ‘xxxxxx‘, plus 1, means that there are N registers set
* to 0. This opcode can represent from 1 to 64 contiguous registers set
* to the value of 0.
*
* XZERO opcode is represented by two bytes 01xxxxxx yyyyyyyy. The 14-bit
* integer represented by the bits ‘xxxxxx‘ as most significant bits and
* ‘yyyyyyyy‘ as least significant bits, plus 1, means that there are N
* registers set to 0. This opcode can represent from 0 to 16384 contiguous
* registers set to the value of 0.
*
* VAL opcode is represented as 1vvvvvxx. It contains a 5-bit integer
* representing the value of a register, and a 2-bit integer representing
* the number of contiguous registers set to that value ‘vvvvv‘.
* To obtain the value and run length, the integers vvvvv and xx must be
* incremented by one. This opcode can represent values from 1 to 32,
* repeated from 1 to 4 times.
*
* The sparse representation can‘t represent registers with a value greater
* than 32, however it is very unlikely that we find such a register in an
* HLL with a cardinality where the sparse representation is still more
* memory efficient than the dense representation. When this happens the
* HLL is converted to the dense representation.
*
* The sparse representation is purely positional. For example a sparse
* representation of an empty HLL is just: XZERO:16384.
*
* An HLL having only 3 non-zero registers at position 1000, 1020, 1021
* respectively set to 2, 3, 3, is represented by the following three
* opcodes:
*
* XZERO:1000 (Registers 0-999 are set to 0)
* VAL:2,1 (1 register set to value 2, that is register 1000)
* ZERO:19 (Registers 1001-1019 set to 0)
* VAL:3,2 (2 registers set to value 3, that is registers 1020,1021)
* XZERO:15362 (Registers 1022-16383 set to 0)
*
* In the example the sparse representation used just 7 bytes instead
* of 12k in order to represent the HLL registers. In general for low
* cardinality there is a big win in terms of space efficiency, traded
* with CPU time since the sparse representation is slower to access:
*
* The following table shows average cardinality vs bytes used, 100
* samples per cardinality (when the set was not representable because
* of registers with too big value, the dense representation size was used
* as a sample).
*
* 100 267
* 200 485
* 300 678
* 400 859
* 500 1033
* 600 1205
* 700 1375
* 800 1544
* 900 1713
* 1000 1882
* 2000 3480
* 3000 4879
* 4000 6089
* 5000 7138
* 6000 8042
* 7000 8823
* 8000 9500
* 9000 10088
* 10000 10591
*
* The dense representation uses 12288 bytes, so there is a big win up to
* a cardinality of ~2000-3000. For bigger cardinalities the constant times
* involved in updating the sparse representation is not justified by the
* memory savings. The exact maximum length of the sparse representation
* when this implementation switches to the dense representation is
* configured via the define server.hll_sparse_max_bytes.
*/
以上是关于Redis数据结构之HperLogLog的主要内容，如果未能解决你的问题，请参考以下文章