RedisCreating and Using Cluster Mode

Posted 2020-08-15 山高我为峰

1. Documents
Cluster will not support SELECT, it only contains database 0.

All the nodes use TCP bus and binary protocol to communicate. Every node connect to cluster bus. The use gossip to talk.

Some times we lost writing data
1. master node dies right before the slave node get the newly writing data.
2. master node dies, and slave node become master, and receive a writing data. when the old master recover, it does not have the newly writing data.

2. Configuration the Cluster mode
Check out the latest codes from here 
Switch to the branch unstable

>make distclean
>make

>mkdir /Users/carl/tool/redis-unstable
>cp redis-server /Users/carl/tool/redis-unstable/
>cp redis-cli /Users/carl/tool/redis-unstable/
>cp redis-benchmark /Users/carl/tool/redis-unstable/
>cd ..
>cp redis.conf /Users/carl/tool/redis-unstable/

>sudo ln -s /Users/carl/tool/redis-unstable /opt/redis-unstable
>sudo rm -fr /opt/redis
>sudo ln -s /opt/redis-unstable /opt/redis

Prepare the Configuration files

>mkdir cluster-conf
>cd cluster-conf
>mkdir 7000 7001 7002 7003 7004 7005 7006

Change the redis.conf there and properties as follow for references;

port 7000
appendonly yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000

Then start all the nodes one by one

>../../redis-server ./redis.conf

Every time, we will see these message after we start one node.
[23303] 01 May 15:28:47.602 * No cluster configuration found, I\'m c5a855fba006f6b3302f7c162ba3b6a71d548b58

After we start 7000 to 7005, 6 nodes, execute this command from src directory to configure cluster.

>./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

Error Message:
cannot load such file -- redis (LoadError)

Solution:

>ruby -v

ruby 2.0.0p451 (2014-02-24 revision 45167) [universal.x86_64-darwin13]

>gem -v

2.0.14

1. Use a recent ruby (1.9.3 or 2.0)
2. Install the redid gem

>sudo gem install redis

Then 

>ruby -rubygems ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

>>> Creating cluster Connecting to node 127.0.0.1:7000: OK Connecting to node 127.0.0.1:7001: OK Connecting to node 127.0.0.1:7002: OK Connecting to node 127.0.0.1:7003: OK Connecting to node 127.0.0.1:7004: OK Connecting to node 127.0.0.1:7005: OK >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 Adding replica 127.0.0.1:7003 to 127.0.0.1:7000 Adding replica 127.0.0.1:7004 to 127.0.0.1:7001 Adding replica 127.0.0.1:7005 to 127.0.0.1:7002 M: c5a855fba006f6b3302f7c162ba3b6a71d548b58 127.0.0.1:7000   slots:0-5460 (5461 slots) master M: d0a6d910adeffa2b0ba69ddc69d9e69bdda1a41e 127.0.0.1:7001   slots:5461-10922 (5462 slots) master M: 376e13f055d4119ca45824aff3e7722a3043ecf7 127.0.0.1:7002   slots:10923-16383 (5461 slots) master S: 5295fe0fae941d7f964a674c5e5a4de37738e412 127.0.0.1:7003   replicates c5a855fba006f6b3302f7c162ba3b6a71d548b58 S: 64a3ae73e370d1e016a01adaedce1606d3b7d2ea 127.0.0.1:7004   replicates d0a6d910adeffa2b0ba69ddc69d9e69bdda1a41e S: 4324a0c7c90a460b4fc107f66e61656ec23c316b 127.0.0.1:7005   replicates 376e13f055d4119ca45824aff3e7722a3043ecf7 Can I set the above configuration? (type \'yes\' to accept): yes >>> Nodes configuration updated >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join... >>> Performing Cluster Check (using node 127.0.0.1:7000) M: c5a855fba006f6b3302f7c162ba3b6a71d548b58 127.0.0.1:7000   slots:0-5460 (5461 slots) master M: d0a6d910adeffa2b0ba69ddc69d9e69bdda1a41e 127.0.0.1:7001   slots:5461-10922 (5462 slots) master M: 376e13f055d4119ca45824aff3e7722a3043ecf7 127.0.0.1:7002   slots:10923-16383 (5461 slots) master M: 5295fe0fae941d7f964a674c5e5a4de37738e412 127.0.0.1:7003   slots: (0 slots) master   replicates c5a855fba006f6b3302f7c162ba3b6a71d548b58 M: 64a3ae73e370d1e016a01adaedce1606d3b7d2ea 127.0.0.1:7004   slots: (0 slots) master   replicates d0a6d910adeffa2b0ba69ddc69d9e69bdda1a41e M: 4324a0c7c90a460b4fc107f66e61656ec23c316b 127.0.0.1:7005   slots: (0 slots) master   replicates 376e13f055d4119ca45824aff3e7722a3043ecf7 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.

—replicas 1 indicate that every master has a replica node.

It will create 3 masters and 3 slaves.

Right now, my version of branch unstable is 2.9.11.

Test and Verify the Cluster

>./redis-cli -c -p 7000

127.0.0.1:7000> set foo bar

-> Redirected to slot [12182] located at 127.0.0.1:7002 OK 127.0.0.1:7002> get foo "bar" 127.0.0.1:7002> set hello world -> Redirected to slot [866] located at 127.0.0.1:7000 OK 127.0.0.1:7000> get hello "world"

Add New Node

redis>cluster nodes 

This command will show all the nodes.
eg:

 

>./redis-trib.rb add-node 127.0.0.1:7006 127.0.0.1:7000

Connecting to node 127.0.0.1:7006: OK >>> Send CLUSTER MEET to node 127.0.0.1:7006 to make it join the cluster. [OK] New node added correctly.

Or add slave node

>./redis-trib.rb add-node --slave 127.0.0.1:7006 127.0.0.1:7000

Or 

>./redis-trib.rb add-node --slave --master-id adsfadsfadsfsadfsdaf 127.0.0.1:7006 127.0.0.1:7000

Remove a Node

>./redis-trib.rb del-node 127.0.0.1:7000 \'591d10c5251abd0ef655a458f8e16a6cca182da6\'

First parameter is any node host and port, the second parameter the id of the node we plan to remove.

Error Message:
[ERR] Node 127.0.0.1:7006 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

Solution:
I delete the node, I plan to add it again. But I meet that error message.

I do not know how to fix this right now. But an easy way, delete all the files under 7006 directory except redis.conf, they it will generate a new node id.

3. Scala Client
Check the Status

redis>info

redis_mode:cluster
redis_version:2.9.11

Open project easynosqlscala

It runs great.

package com.sillycat.easynosqlscala.app
import redis.clients.jedis.{ JedisCluster, HostAndPort, Jedis }
import Java.util.HashSet

object TestRedisDBConnectionApp extends App {
  //  val jedis = new Jedis("172.0.0.1", 7000)
  //  jedis.set("name", "sillycat")
  //  println(jedis.get("name"))

  val jedisClusterNodes = new HashSet[HostAndPort]
  jedisClusterNodes.add(new HostAndPort("127.0.0.1", 7000))
  jedisClusterNodes.add(new HostAndPort("127.0.0.1", 7001))
  jedisClusterNodes.add(new HostAndPort("127.0.0.1", 7002))
  jedisClusterNodes.add(new HostAndPort("127.0.0.1", 7003))
  jedisClusterNodes.add(new HostAndPort("127.0.0.1", 7004))
  jedisClusterNodes.add(new HostAndPort("127.0.0.1", 7005))
  jedisClusterNodes.add(new HostAndPort("127.0.0.1", 7006))
  val jc = new JedisCluster(jedisClusterNodes)
  jc.set("age", "32")
  println("name = " + jc.get("name") + " age = " + jc.get("age"))

}

 

Redis Monitor
https://github.com/LittlePeng/redis-monitor


References:
Redis 1~ 6
http://sillycat.iteye.com/blog/1549504   Installation redis on windows, ubuntu, redhat
http://sillycat.iteye.com/blog/1553507   Data Type, String, Int, List, Set
http://sillycat.iteye.com/blog/1553508   Data Type, Ordered Set, Hash, Publish/Subscribe, Data Expiration
http://sillycat.iteye.com/blog/1553509   Java DAO Layer
http://sillycat.iteye.com/blog/2028180   Installation on MAC, HA Solution sentinel master slave
http://sillycat.iteye.com/blog/2033094   Scala Client 

http://redis.io/topics/cluster-tutorial
http://redis.io/topics/cluster-spec

http://www.redis.cn/topics/cluster-tutorial.html
http://www.redis.cn/topics/cluster-spec.html

Tips
http://in.sdo.com/?p=1378
http://www.cnblogs.com/lulu/archive/2013/06/10/3130878.html
http://www.oschina.NET/translate/utopian-redis-cluster

 

Cluster

http://hot66hot.iteye.com/blog/2050676