记一次Redis Cluster 故障，迁移时--

Posted 2020-08-17

 

暂时记录在此 ！----- 待解决！

 

=== REDIS BUG REPORT START: Cut & paste starting from here ===
21516:M 04 Nov 01:25:16.448 # Redis 3.2.3 crashed by signal: 11
21516:M 04 Nov 01:25:16.448 # Crashed running the instuction at: 0x4644f2
21516:M 04 Nov 01:25:16.448 # Accessing address: 0x8
21516:M 04 Nov 01:25:16.448 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
/usr/local/bin/redis-server 0.0.0.0:6380 [cluster](migrateCloseSocket+0x52)[0x4644f2]

Backtrace:
/usr/local/bin/redis-server 0.0.0.0:6380 [cluster](logStackTrace+0x3c)[0x45bd5c]
/usr/local/bin/redis-server 0.0.0.0:6380 [cluster](sigsegvHandler+0xa1)[0x45cc41]
/lib64/libpthread.so.0[0x336b60f710]
/usr/local/bin/redis-server 0.0.0.0:6380 [cluster](migrateCloseSocket+0x52)[0x4644f2]
/usr/local/bin/redis-server 0.0.0.0:6380 [cluster](migrateCommand+0x7cd)[0x46744d]
/usr/local/bin/redis-server 0.0.0.0:6380 [cluster](call+0x72)[0x424192]
/usr/local/bin/redis-server 0.0.0.0:6380 [cluster](processCommand+0x365)[0x428d75]
/usr/local/bin/redis-server 0.0.0.0:6380 [cluster](processInputBuffer+0x109)[0x435089]
/usr/local/bin/redis-server 0.0.0.0:6380 [cluster](aeProcessEvents+0x13d)[0x41f86d]
/usr/local/bin/redis-server 0.0.0.0:6380 [cluster](aeMain+0x2b)[0x41fb6b]
/usr/local/bin/redis-server 0.0.0.0:6380 [cluster](main+0x370)[0x427220]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x336b21ed1d]
/usr/local/bin/redis-server 0.0.0.0:6380 [cluster][0x41d039]

------ INFO OUTPUT ------
# Server
redis_version:3.2.3
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:e6287c3620f90cfc
redis_mode:cluster
os:Linux 2.6.32-431.el6.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.7
process_id:21516
run_id:884f296f3415da21737749e523e448896b3c41d6
tcp_port:6380
uptime_in_seconds:1295041
uptime_in_days:14
hz:10
lru_clock:1798884
executable:/usr/local/bin/redis-server
config_file:/etc/redis/6380.conf

# Clients
connected_clients:67
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:1390635680
used_memory_human:1.30G
used_memory_rss:1430929408
used_memory_rss_human:1.33G
used_memory_peak:1599526384
used_memory_peak_human:1.49G
total_system_memory:67135774720
total_system_memory_human:62.53G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:14000000000
maxmemory_human:13.04G
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.03
mem_allocator:jemalloc-4.0.3

# Persistence
loading:0
rdb_changes_since_last_save:765
rdb_bgsave_in_progress:0
rdb_last_save_time:1478193409
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:14
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok

# Stats
total_connections_received:225838135
total_commands_processed:234213505
instantaneous_ops_per_sec:2670
total_net_input_bytes:10745915204
total_net_output_bytes:1922575953129
instantaneous_input_kbps:80.56
instantaneous_output_kbps:4166.00
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:0
expired_keys:35
evicted_keys:0
keyspace_hits:1381016
keyspace_misses:10610
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:20340
migrate_cached_sockets:1

# Replication
role:master
connected_slaves:1
slave0:ip=10.112.142.19,port=6379,state=online,offset=4388541123,lag=24
master_repl_offset:4388541123
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:4387997335
repl_backlog_histlen:543789

# CPU
used_cpu_sys:11945.34
used_cpu_user:43438.31
used_cpu_sys_children:0.77
used_cpu_user_children:12.24

# Commandstats
cmdstat_get:calls=277022,usec=589655,usec_per_call=2.13
cmdstat_set:calls=657229,usec=2681518,usec_per_call=4.08
cmdstat_setex:calls=804664,usec=3956968,usec_per_call=4.92
cmdstat_del:calls=49239,usec=682107,usec_per_call=13.85
cmdstat_exists:calls=3,usec=5,usec_per_call=1.67
cmdstat_mget:calls=1114331,usec=2787504,usec_per_call=2.50
cmdstat_rpush:calls=2227982,usec=2876069,usec_per_call=1.29
cmdstat_lpop:calls=2227988,usec=4906250,usec_per_call=2.20
cmdstat_llen:calls=1,usec=2,usec_per_call=2.00
cmdstat_select:calls=1,usec=0,usec_per_call=0.00
cmdstat_expire:calls=616450,usec=828737,usec_per_call=1.34
cmdstat_dbsize:calls=35402,usec=20732,usec_per_call=0.59
cmdstat_ping:calls=181066,usec=71024,usec_per_call=0.39
cmdstat_psync:calls=1,usec=20661,usec_per_call=20661.00
cmdstat_replconf:calls=469,usec=314,usec_per_call=0.67
cmdstat_flushall:calls=2,usec=12641015,usec_per_call=6320507.50
cmdstat_info:calls=493912,usec=22690391,usec_per_call=45.94
cmdstat_ttl:calls=216,usec=282,usec_per_call=1.31
cmdstat_config:calls=21250,usec=257271,usec_per_call=12.11
cmdstat_cluster:calls=225183697,usec=37325775030,usec_per_call=165.76
cmdstat_restore:calls=234950,usec=37891754,usec_per_call=161.28
cmdstat_command:calls=87630,usec=63312864,usec_per_call=722.50

# Cluster
cluster_enabled:1

# Keyspace
db0:keys=86125,expires=83431,avg_ttl=1104724913
hash_init_value: 1477220200

------ CLIENT LIST OUTPUT ------
id=225838116 addr=10.13.47.201:55401 fd=90 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225838064 addr=10.112.148.42:16039 fd=108 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225838086 addr=10.13.58.152:51606 fd=106 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837990 addr=10.13.47.153:51890 fd=97 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225838001 addr=10.112.146.29:30145 fd=104 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225838040 addr=10.112.148.42:15996 fd=82 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225838046 addr=10.112.148.42:16019 fd=53 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=225838023 addr=10.112.146.30:53010 fd=99 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837930 addr=10.112.148.40:41759 fd=92 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=225837963 addr=10.13.60.218:38670 fd=69 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837969 addr=10.13.42.120:56719 fd=101 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837980 addr=10.13.47.201:55386 fd=93 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837944 addr=10.13.62.91:40534 fd=57 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837946 addr=10.112.146.28:46732 fd=91 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837908 addr=10.13.51.43:48088 fd=83 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837918 addr=10.112.146.29:30055 fd=86 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837890 addr=10.13.53.58:43165 fd=76 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225817552 addr=?:0 fd=60 name= age=59 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=225625742 addr=?:0 fd=134 name= age=576 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225624941 addr=?:0 fd=56 name= age=581 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=225829873 addr=?:0 fd=51 name= age=23 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=225837794 addr=10.112.146.36:64465 fd=88 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=225837800 addr=10.13.44.109:53584 fd=85 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837641 addr=10.13.40.4:34940 fd=80 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837502 addr=10.13.42.3:47793 fd=74 name= age=2 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837474 addr=10.13.40.4:34901 fd=98 name= age=2 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=225836201 addr=10.112.146.34:56289 fd=65 name= age=5 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=225835920 addr=10.112.148.39:64749 fd=109 name= age=6 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=225624917 addr=?:0 fd=55 name= age=582 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225733424 addr=?:0 fd=59 name= age=294 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225625278 addr=?:0 fd=62 name= age=577 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225625196 addr=?:0 fd=61 name= age=578 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225625040 addr=?:0 fd=58 name= age=580 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225624572 addr=?:0 fd=52 name= age=587 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225625324 addr=10.112.150.12:17605 fd=63 name= age=576 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=225631906 addr=?:0 fd=66 name= age=561 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225832781 addr=10.112.142.19:56847 fd=73 name= age=15 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=migrate
id=225625599 addr=?:0 fd=102 name= age=576 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225837083 addr=?:0 fd=114 name= age=3 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225836916 addr=10.13.42.120:56465 fd=103 name= age=3 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=225625940 addr=?:0 fd=153 name= age=575 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225656957 addr=10.112.142.19:5941 fd=67 name= age=497 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=replconf
id=225626362 addr=?:0 fd=129 name= age=574 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225626435 addr=?:0 fd=179 name= age=574 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225626792 addr=?:0 fd=146 name= age=574 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225627239 addr=?:0 fd=123 name= age=573 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225628915 addr=?:0 fd=75 name= age=569 idle=15 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=225629803 addr=?:0 fd=167 name= age=566 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225628047 addr=?:0 fd=139 name= age=571 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225635480 addr=?:0 fd=47 name= age=552 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225689334 addr=?:0 fd=9 name= age=410 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225627796 addr=?:0 fd=96 name= age=571 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225627466 addr=?:0 fd=72 name= age=572 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get

id=225626764 addr=?:0 fd=138 name= age=574 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225665993 addr=?:0 fd=64 name= age=473 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225836753 addr=10.13.47.201:55300 fd=84 name= age=4 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=225626110 addr=?:0 fd=81 name= age=575 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225836704 addr=?:0 fd=70 name= age=4 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225837878 addr=10.112.146.30:52913 fd=14 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837901 addr=10.13.56.93:42620 fd=89 name= age=1 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=225837905 addr=10.112.146.35:14667 fd=79 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=225838109 addr=10.112.146.28:46837 fd=77 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=225838114 addr=10.112.146.30:53068 fd=71 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=225838130 addr=10.13.62.85:55263 fd=94 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=225838131 addr=10.112.146.26:43269 fd=54 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=225838134 addr=10.112.146.30:53079 fd=87 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225838135 addr=10.13.62.91:40577 fd=78 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get
id=225838136 addr=10.13.49.245:33953 fd=95 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget

------ CURRENT CLIENT INFO ------
id=225832781 addr=10.112.142.19:56847 fd=73 name= age=15 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=migrate
argv[0]: ‘DEL‘
argv[1]: ‘JIUKUIYOU_COM_GetCouponInfo_3479num‘

------ REGISTERS ------
21516:M 04 Nov 01:25:16.449 #
RAX:00007f067cc4d553 RBX:0000000000000000
RCX:000000000000003a RDX:0000000000000000
RDI:00007f067cc4d577 RSI:00000000004e747c
RBP:00007f06804fd150 RSP:00007fff8de3ae70
R8 :6d756e393734335f R9 :0000000000000000
R10:00007f07008206f0 R11:00007f06fa40d000
R12:0000000000000001 R13:0000000000000001
R14:00007f06783a2548 R15:0000000000000000
RIP:00000000004644f2 EFL:0000000000010202
CSGSFS:0000000000000033
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae7f) -> 00007f0700820698
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae7e) -> 0000000200000010
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae7d) -> 0000000000000000
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae7c) -> 0000000000600270
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae7b) -> 00007f0600000000
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae7a) -> 00007f06783a2548
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae79) -> 000000000000b000
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae78) -> 00007f067cc4d3c0
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae77) -> 00007f060000000a
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae76) -> 000000009fa02f20
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae75) -> 000001584f17ee01
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae74) -> 00007f069fa03760
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae73) -> 000000000046744d
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae72) -> 00007fff8de3b790
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae71) -> 00007f069f304c80
21516:M 04 Nov 01:25:16.449 # (00007fff8de3ae70) -> 0000000000000001

------ FAST MEMORY TEST ------
21516:M 04 Nov 01:25:16.449 # Bio thread for job type #0 terminated
21516:M 04 Nov 01:25:16.449 # Bio thread for job type #1 terminated
*** Preparing to test memory region 721000 (94208 bytes)
*** Preparing to test memory region aef000 (135168 bytes)
*** Preparing to test memory region 336ac21000 (4096 bytes)
*** Preparing to test memory region 336b58f000 (20480 bytes)
*** Preparing to test memory region 336b819000 (16384 bytes)
*** Preparing to test memory region 7f0657fff000 (2694844416 bytes)
*** Preparing to test memory region 7f06f8bff000 (10485760 bytes)
*** Preparing to test memory region 7f06f9600000 (14680064 bytes)
*** Preparing to test memory region 7f06fa400000 (2097152 bytes)
*** Preparing to test memory region 7f0700600000 (2097152 bytes)
*** Preparing to test memory region 7f0700820000 (16384 bytes)
*** Preparing to test memory region 7f070082c000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
=== REDIS BUG REPORT END. Make sure to include from START to END. ===

 

==================================

[[email protected] ~]# /usr/local/bin/redis-trib.rb rebalance 10.112.142.21:7211
>>> Performing Cluster Check (using node 10.112.142.21:7211)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 7 nodes. Total weight = 7
Moving 892 slots from 10.112.142.20:6380 to 10.112.142.21:7210

 

[ERR] Calling MIGRATE: ERR Target instance replied with error: BUSYKEY Target key name already exists.

 

[WARNING] Node 10.112.142.21:7210 has slots in importing state (45).
[WARNING] Node 10.112.142.20:6380 has slots in migrating state (45).

 

[WARNING] Node 10.112.142.20:6380 has slots in migrating state (45).
[WARNING] The following slots are open: 45

 

[OK] All nodes agree about slots configuration.
>>> Check for open slots...
[WARNING] Node 10.112.142.20:6380 has slots in migrating state (45).
[WARNING] The following slots are open: 45
>>> Fixing open slot 45
*** Found keys about slot 45 in node 10.112.142.21:7210!
Set as migrating in: 10.112.142.20:6380
Set as importing in: 10.112.142.21:7210
Moving slot 45 from 10.112.142.20:6380 to 10.112.142.21:7210:
*** Target key exists. Replacing it for FIX.
/usr/local/lib/ruby/gems/2.3.0/gems/redis-3.3.1/lib/redis/client.rb:121:in `call‘: MOVED 45 10.112.142.21:6380 (Redis::CommandError)
from /usr/local/bin/redis-trib.rb:942:in `rescue in move_slot‘
from /usr/local/bin/redis-trib.rb:937:in `move_slot‘
from /usr/local/bin/redis-trib.rb:607:in `fix_open_slot‘
from /usr/local/bin/redis-trib.rb:422:in `block in check_open_slots‘
from /usr/local/bin/redis-trib.rb:422:in `each‘
from /usr/local/bin/redis-trib.rb:422:in `check_open_slots‘
from /usr/local/bin/redis-trib.rb:360:in `check_cluster‘
from /usr/local/bin/redis-trib.rb:1140:in `fix_cluster_cmd‘
from /usr/local/bin/redis-trib.rb:1696:in `<main>‘