Lemcoden

来自于大数据攻城狮的分享

redis常见问题

1
jedis,luttce,springboot:low/high level

击穿

key过期造成并发访问数据库

graph LR
	id0((用户
client)) --> id2 id2[nginx] --> id3[ ] id3 --> id4 id4((client
server)) --1.null--> id5[redis
缓存
key过期时间,LRU,LFU] id4 --2.setInx--> id5 id4 --3.只有获得锁的去访问DB--> id1[DBMySQL] id4 --> id1 before[before
肯定发生了高并发]

解决:

并发有了:阻止并发到达DB,redis又没有key

redis是单进程单实例

setInx() -> 锁

setInx() -> 锁

1.get key

2.setInx

3-1. ok ,去DB

3-2.false,sleep -> 1

问题:

  1. 如果第一个人挂了?发生死锁

    可以设置锁的过期时间

  2. 没挂,但是,锁超时了…

    多线程,一个线程取库,一个线程监控,并延长时间

自己实现分布式协调很麻烦

穿透

从业务接受查询的是你系统根本不存在的数据

graph LR
 	id0{业务} --> id1((client
service)) id1 --> id2[redis
缓存] id1 --> id3[DB] id4[布隆过滤器] --> id5[client包含] id4 --> id6[client只包含算法
bitmap->redis
无状态] id4 --> id7[redis 集成布隆]

bloom过滤器问题:

  • [x] 只能增加,不能删除

  • [x] 布谷鸟过滤器

  • [ ] 空key

雪崩

大量的key同时失效,间接的造成大量的访问到达DB

graph LR
 	id0{业务} --> id1((client
service)) id1 --> id2[redis
缓存] id1 --> id3[DB] id4[随机过期时间] -.-> id5{零点} id4 --> id6[时点性无关] id5 --> id7[强依赖击穿方案] id5 --> id8[业务层加判断,零点延时...] style id5 fill:#0f0,stroke-width:2px,fill-opacity:0.5

分布式锁

1setnx

2.过期时间

3.多线程(守护线程)延长过期

redisson

zookeeper 做分布式锁!

API

1
2
3
127.0.0.1:6379> CONFIG GET *
127.0.0.1:6379> CONFIG SET protected-mode no
OK

AKF概述

graph TB
	subgraph single
	id1[redis 单机 单进程 缓存 数据库]
	id1 --> id((RDB AOF))
	end

单机,单节点,单实例

1.单点故障

2.容量有限

3.压力

AKF => xyz

graph LR	
	subgraph redis:x
	id{client} --> id1((redis:rw))
	id1 -- x --> id2((redis:r))
	id2 --> id3((redis:r))
	id --> id2
	id --> id3
	style id1 fill:#f9f,stroke:#333,stroke-width:4px,fill-opacity:0.5
	end
	subgraph redis2
	bu3[redis:rw] --y--- id1
	bu3 --- id11[redis]
	id11 --- id12[redis]
	style bu3 fill:#f9f,stroke:#333,stroke-width:4px,fill-opacity:0.5
	end
	subgraph redis1
	bu2[redis:rw] --- bu3
	bu2 --- id21[redis]
	id21 --- id22[redis]
	style bu2 fill:#f9f,stroke:#333,stroke-width:4px,fill-opacity:0.5
	end
	subgraph redis
	bu1[redis:rw] --- bu2
	bu1 --- id31[redis]
	id31 --- id32[redis]
	style bu1 fill:#f9f,stroke:#333,stroke-width:4px,fill-opacity:0.5
	end
	id1 --z--> id11

AKF一变多

数据一致性问题

所有节点阻塞直到数据全部一致

  • 同步方式

强一致性

极其容易破坏可用性!反问自己:为什么一边多? 解决可用性

  • 通过异步方式

容忍数据丢失一部分

graph LR
	client --set k1 a--> id1((redis))
	id1 --阻塞--> id2((redis))
	id1 --阻塞--> id3((redis))
	sum[强一致性,破坏可用性]
graph LR
	client --set k1 a--> id1((redis))
	id1 --非阻塞--> id2((redis))
	id1 --非阻塞--> id3((redis))
	sum[弱一致性
丢失数据]
graph LR
 	client --set k1 a--> id1((redis))
 	id1 --同步阻塞--> id4[kafka 
可靠,集群
响应速度够快] id4 --> id2((redis)) id4 --> id3((redis)) sum[最终数据会一致
有可能数据不一致]

主从:

主从复制,主机有的数据从机也有

客户端主从都可以访问

主备:

客户端只访问master

备用机不参业务

主机

负责读写

自己又是一个:单点

一般对主做高可用,自动的故障转移:代替人

人是怎么做监控的
由一个技术,程序实现
只要是一个程序就会有单点故障的问题,一变多的集群
有一些不一样的地方
graph TB
	id((redis)) --> id1[监控]
	id --> id2[监控]
	id --> id3[监控]
graph BT
	id1[监控] --> id((redis)) 
	id2[监控] --> id
	id3[监控] --> id

都给出 OK

强一致性

一部分给出OK

另一部分不算数

几个?

1,2

推导:

1台统计不准确,不够势力范围

问题:网络分区

脑裂

2 在3个节点成功解决脑裂问题

3 在4个节点成功解决脑裂问题

4 在5个节点成功解决脑裂问题

n/2+1 过半!

使用奇数台!

x实际操作:主从复制

启动安装redis-server脚本,6379,6380,6381

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[lemcoden@hadoop01 utils]$ sudo ./install_server.sh 
Welcome to the redis service installer
This script will help you easily set up a running redis server

Please select the redis port for this instance: [6379] 6380
Please select the redis config file name [/etc/redis/6380.conf]
Selected default - /etc/redis/6380.conf
Please select the redis log file name [/var/log/redis_6380.log]
Selected default - /var/log/redis_6380.log
Please select the data directory for this instance [/var/lib/redis/6380]
Selected default - /var/lib/redis/6380
Please select the redis executable path [] /opt/bigdata/module/redis5/bin/redis-server
Selected config:
Port : 6380
Config file : /etc/redis/6380.conf
Log file : /var/log/redis_6380.log
Data dir : /var/lib/redis/6380
Executable : /opt/bigdata/module/redis5/bin/redis-server
Cli Executable : /opt/bigdata/module/redis5/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied /tmp/6380.conf => /etc/init.d/redis_6380
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!

拷贝配置文件到测试目录

1
2
3
4
5
6
7
[lemcoden@hadoop01 test]$ cp /etc/redis/* ./
[lemcoden@hadoop01 test]$ ls
6379.conf 6380.conf 6381.conf
[lemcoden@hadoop01 test]$ vim 6379.conf
daemonize no
#logfile /var/lib/redis/6379.log
appendonly no

启动redis-server

1
2
3
4
5
6
终端01
[lemcoden@hadoop01 bin]$ sudo ./redis-server /home/lemcoden/test/6379
终端02
[lemcoden@hadoop01 bin]$ sudo ./redis-server /home/lemcoden/test/6379
终端03
[lemcoden@hadoop01 bin]$ sudo ./redis-server /home/lemcoden/test/6379

启动redis-cli(略过)

REPLICAOF命令 降级主机slave

1
2
127.0.0.1:6380> REPLICAOF 127.0.0.1 6379
OK
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
127.0.0.1:6379> set k1 hello
OK


127.0.0.1:6380> get k1
"hello"
127.0.0.1:6380> set k1 222
(error) READONLY You can't write against a read only replica.

127.0.0.1:6381> set key2 ccc
OK
127.0.0.1:6381> get key2
"ccc"

127.0.0.1:6379> get key2
(nil)

127.0.0.1:6381> REPLICAOF 127.0.0.1 6379
OK

127.0.0.1:6381> keys *
1) "k1"

kill 6381主机后,6379设置多个key,然后启动6381(增量同步)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
127.0.0.1:6379> set k3 aaa
OK
127.0.0.1:6379> set k4 444
OK
127.0.0.1:6379> set k2 4123313
OK
127.0.0.1:6379> set k5 141jddpfipj
OK
127.0.0.1:6379> set k6 sada
OK


[lemcoden@hadoop01 bin]$ sudo ./redis-server /home/lemcoden/test/6381.conf --replicaof 127.0.0.1 6379

127.0.0.1:6381> keys *
1) "k3"
2) "k5"
3) "k4"
4) "k1"
5) "k6"
6) "k2"

如果6381开启AOF ,则会产生RDB sync

!!!且RDB文件中不存在replicaID号

1
2
3
4
5
6
7
8
9
10
[lemcoden@hadoop01 bin]$ sudo ./redis-server /home/lemcoden/test/6381.conf  --replicaof 127.0.0.1 6379 --appendonly yes
16071:S 21 Nov 2020 05:05:29.820 * Full resync from master: d93b72c43fc1777195b07d7546baf1b34a92fb4f:12772
16071:S 21 Nov 2020 05:05:29.870 * MASTER <-> REPLICA sync: receiving 240 bytes from master
16071:S 21 Nov 2020 05:05:29.870 * MASTER <-> REPLICA sync: Flushing old data
16071:S 21 Nov 2020 05:05:29.874 * MASTER <-> REPLICA sync: Loading DB in memory
16071:S 21 Nov 2020 05:05:29.874 * MASTER <-> REPLICA sync: Finished with success
16071:S 21 Nov 2020 05:05:29.919 * Background AOF rewrite terminated with success

[lemcoden@hadoop01 6381]$ vim dump.rdb

假如Master宕掉,SLAVE升级为Master

1
2
3
4
5
6
127.0.0.1:6381> REPLICAOF no one 
OK
16192:M 21 Nov 2020 05:18:36.965 * MASTER MODE enabled
127.0.0.1:6380> REPLICAOF 127.0.0.1 6381
OK

关于主从架构,conf文件的一些配置

1
2
3
4
5
6
7
8
9
10
11
12
13
# replicaof <masterip> <masterport>
# masterauth <master-password>
如果redis刚启动,从Master复制数据期间,是否还支持本机old数据查询
replica-serve-stale-data yes
备机是否只支持查询
replica-read-only yes
是否直接使用网络复制
repl-diskless-sync no
增量复制文件大小,salve短时间下线情况发生时
# repl-backlog-size 1mb
最少几个replica写成功
# min-replicas-to-write 3
# min-replicas-max-lag 10

主从复制配置,需要人工维护主的故障问题

x实际操作:HA哨兵

新建哨兵conf文件

1
2
port 26379
sentinel monitor mymaster 127.0.0.1 6379 2

启动服务器,启动哨兵26379,26380,26381

1
2
3
4
5
6
[lemcoden@hadoop01 bin]$ sudo ./redis-server /home/lemcoden/test/26379.conf --sentinel
16806:X 21 Nov 2020 05:57:09.434 # +monitor master mymaster 127.0.0.1 6379 quorum 2
16806:X 21 Nov 2020 05:57:09.435 * +slave slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster 127.0.0.1 6379
16806:X 21 Nov 2020 05:57:09.442 * +slave slave 127.0.0.1:6380 127.0.0.1 6380 @ mymaster 127.0.0.1 6379
16806:X 21 Nov 2020 06:00:34.614 * +sentinel sentinel aea8334513cd9abf6b18c40262915bd337033350 127.0.0.1 26380 @ mymaster 127.0.0.1 6379
16806:X 21 Nov 2020 06:00:54.877 * +sentinel sentinel f95f1b3b18e56f7bda1bdf5e8b39baa9400cb3d3 127.0.0.1 26381 @ mymaster 127.0.0.1 6379

退出6379,master,sentinal推演出新的leader,

1
16806:X 21 Nov 2020 06:04:32.642 * +failover-state-send-slaveof-noone slave 127.0.0.1:6380 127.0.0.1 6380 @ mymaster 127.0.0.1 6379

sentinel通过发布订阅知道其他sentinel

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
127.0.0.1:6380> PSUBSCRIBE *
Reading messages... (press Ctrl-C to quit)
1) "psubscribe"
2) "*"
3) (integer) 1
1) "pmessage"
2) "*"
3) "__sentinel__:hello"
4) "127.0.0.1,26380,aea8334513cd9abf6b18c40262915bd337033350,1,mymaster,127.0.0.1,6380,1"
1) "pmessage"
2) "*"
3) "__sentinel__:hello"
4) "127.0.0.1,26381,f95f1b3b18e56f7bda1bdf5e8b39baa9400cb3d3,1,mymaster,127.0.0.1,6380,1"
1) "pmessage"
2) "*"

sentinel 配置文件

1
2
3
4
$REDIS_SRC/sentinel.conf
port 26379
daemonize no
.....

y概述

单节点副本解决了单点故障和压力的问题,但是容量问题没有解决

容量client端解决方式

graph LR
	id[2-1:client
融入逻辑,业务拆分
] --> redis1 id --> redis2

sharding分片方式

hash+取模

graph LR
	id[2-2:client
算法:hash+取模
] --> redis1 id --> redis2 id1((弊端:取模的值必须固定,
%3,
%4
影响分布式下的扩展性))

random lpush 适用于消息队列场景

graph LR
	id[2-2:client] --> id2[逻辑:
random
lpush] id2--> redis1 id2 --> redis2 redis1 --> id3([2-2:client rpop]) redis2 --> id3

一致性hash算法

graph LR
	id[2-2:client] --> id2[逻辑:
kemata
一致性hash
没有取模
data,node] id2--> redis1 id2 --> redis2 redis1 --> id3([2-2:client rpop]) redis2 --> id3 id2 --> id4((规划一个环形哈希:虚拟节点)) subgraph 映射算法 id1[hash
crc
fmv
md5
] end

一致性Hash优点:

你加节点,的确可以分担其他节点的压力,不会造成全局洗牌

缺点:

新增节点造成一小部分数据不能命中

1.问题,缓存击穿,压到mysql

2.方案,每次取离我最近的2个物理节点

更倾向于作为缓存,而不是数据库

弊端:

3个模式不能做数据库用

预分区

twemproxy安装

大部分直接按照官网的README操作

编译完成之后,进入scrips目录

将nutcracker.init copy到/etc/ini.d目录下

1
2
[lemcoden@hadoop01 scripts]$ sudo cp nutcracker.init /etc/init.d/twemproxy
[lemcoden@hadoop01 init.d]$ sudo chmod +x twemproxy

新建/etc/nutcracker/

将源码的config目录下的所有文件copy过来

1
2
3
4
[lemcoden@hadoop01 conf]$ sudo mkdir /etc/nutcracker/
[lemcoden@hadoop01 conf]$ ls
nutcracker.leaf.yml nutcracker.root.yml nutcracker.yml
[lemcoden@hadoop01 conf]$ cp ./* /etc/nutcracker/

将src文件夹下的nutcracker可执行文件copy到/usr/bin/目录下

1
[lemcoden@hadoop01 src]$ sudo cp nutcracker /usr/bin/

新建redis数据文件夹,

并手动启动redis-server

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
[lemcoden@hadoop01 nutcracker]$ cd ~
[lemcoden@hadoop01 ~]$ mkdir data
[lemcoden@hadoop01 ~]$ cd data/
[lemcoden@hadoop01 data]$ mkdir 6379
[lemcoden@hadoop01 data]$ mkdir 6380
[lemcoden@hadoop01 data]$ cd 6379
[lemcoden@hadoop01 6379]$ redis-server --port 6379
[lemcoden@hadoop01 data]$ cd 6380
[lemcoden@hadoop01 6379]$ redis-server --port 6380
启动twemproxy代理
[lemcoden@hadoop01 ~]$ systemctl start twemproxy
客户端链接代理
[lemcoden@hadoop01 ~]$ redis-cli -p 22121
127.0.0.1:22121> set k1 sss
(error) ERR Connection refused
127.0.0.1:22121> set k2 sss
(error) ERR Connection refused
127.0.0.1:22121> set k1 sss
OK
127.0.0.1:22121> set k2 sss
OK
127.0.0.1:22121> keys *
Error: Server closed the connection


[lemcoden@hadoop01 ~]$ redis-cli -p 6380
127.0.0.1:6380> keys *
1) "k2"
2) "k1"
127.0.0.1:6380> keys *
1) "k3"
2) "k2"
3) "k1"
127.0.0.1:6380> keys *
1) "k3"
2) "k2"
3) "k4"
4) "k1"

[lemcoden@hadoop01 ~]$ redis-cli -p 6379
127.0.0.1:6379> keys *
(empty list or set)
127.0.0.1:6379> keys *
(empty list or set)
127.0.0.1:6379> keys *
(empty list or set)
127.0.0.1:22121> MULTI
Error: Server closed the connection

predixy 安装

上github,找到release包,下载下来,解压

1
wget https://github.com/joyieldInc/predixy/releases/download/1.0.5/predixy-1.0.5-bin-amd64-linux.tar.gz
1
2
3
4
5
修改conf文件
[lemcoden@hadoop01 conf]$ vim predixy.conf
#cluster.conf
Include sentinel.conf
#try.conf
1
2
3
4
5
6
7
8
9
10
11
12
修改conf文件
[lemcoden@hadoop01 conf]$ vim sentinel.conf
Sentinels {
+ 127.0.0.1:26379
+ 127.0.0.1:26380
+ 127.0.0.1:26381
}
Group ooxx {
}
Group xxoo {
}

集群文件配置(sentinel)

1
2
3
4
5
6
7
8
9
10
11
port 26379
sentinel monitor ooxx 127.0.0.1 36379 2
sentinel monitor xxoo 127.0.0.1 46379 2

port 26380
sentinel monitor ooxx 127.0.0.1 36379 2
sentinel monitor xxoo 127.0.0.1 46379 2

port 26381
sentinel monitor xxoo 127.0.0.1 46379 2
sentinel monitor ooxx 127.0.0.1 36379 2

集群启动

1
2
3
4
5
6
7
8
[lemcoden@hadoop01 test]$ redis-server 26379.conf --sentinel
[lemcoden@hadoop01 test]$ redis-server 26380.conf --sentinel
[lemcoden@hadoop01 test]$ redis-server 26381.conf --sentinel
[lemcoden@hadoop01 36379]$ redis-server --port 36379
[lemcoden@hadoop01 36379]$ redis-server --port 36380
[lemcoden@hadoop01 36379]$ redis-server --port 46379
[lemcoden@hadoop01 36379]$ redis-server --port 46380
[lemcoden@hadoop01 predixy-1.0.5]$ ./bin/predixy conf/predixy.conf

redis-cli实验

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[lemcoden@hadoop01 ~]$ redis-cli -p 7617
127.0.0.1:7617> set k1 sdfsdf
OK
127.0.0.1:7617> get k1"sdfsdf"
127.0.0.1:7617> set k2 sdfsdf
OK
127.0.0.1:7617> get k2
"sdfsdf"
127.0.0.1:7617> set {oo}k1 sdfsdfsdfs
OK
127.0.0.1:7617> set {oo}k2 sdasdasfasfqad
OK
127.0.0.1:7617> WATCH {oo}k1
(error) ERR forbid transaction in current server pool
127.0.0.1:7617>
[lemcoden@hadoop01 ~]$ redis-cli -p 46379
127.0.0.1:46379> keys *
1) "k2"
127.0.0.1:46379> keys *
1) "{oo}k1"
2) "k2"
3) "{oo}k2"
127.0.0.1:46379>
[lemcoden@hadoop01 ~]$ redis-cli -p 36379
127.0.0.1:36379> keys *
1) "k1"
127.0.0.1:36379> keys *
1) "k1"
127.0.0.1:36379>

不支持事务,所以试验单主从

修改config

1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

127.0.0.1:7617> set k1 sdasfa
QUEUED
127.0.0.1:7617> set k2 asdada
QUEUED

127.0.0.1:36379> keys *
1) "k1"
2) "k2"

127.0.0.1:7617> MULTI
OK
127.0.0.1:7617> get k1
QUEUED
127.0.0.1:7617> set k2 2222
QUEUED
127.0.0.1:7617> exec
1) "sdasfa"
2) OK

kill掉36379进程,哨兵会自动选择,在代理层是感觉不到的

1
2
3
kill掉36379进程之后
127.0.0.1:7617> get k1
"sdasfa"

redis官方集群样例

去redis源码的utils/create-cluster目录下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
[lemcoden@hadoop01 create-cluster]$ vim create-cluster 
[lemcoden@hadoop01 create-cluster]$ ./create-cluster start
Starting 30001
Starting 30002
Starting 30003
Starting 30004
Starting 30005
Starting 30006
[lemcoden@hadoop01 create-cluster]$ ./create-cluster create
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:30005 to 127.0.0.1:30001
Adding replica 127.0.0.1:30006 to 127.0.0.1:30002
Adding replica 127.0.0.1:30004 to 127.0.0.1:30003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: a403574ce40177866a188d16a615848bd5b5459a 127.0.0.1:30001
slots:[0-5460] (5461 slots) master
M: 638ac8a174e8a66b1e46bb19155860a5c3aa415b 127.0.0.1:30002
slots:[5461-10922] (5462 slots) master
M: 6a2c3dd007e7d44c95c51d33002e9cc0fdc429ba 127.0.0.1:30003
slots:[10923-16383] (5461 slots) master
S: 69313ced470329f55a5bdf408e4285f345a24154 127.0.0.1:30004
replicates 6a2c3dd007e7d44c95c51d33002e9cc0fdc429ba
S: 967818092ca216fe22a90160587fbdbfc23dd70b 127.0.0.1:30005
replicates a403574ce40177866a188d16a615848bd5b5459a
S: 00e70c8bebff2e59e89ed3e1e152d6805ad36eff 127.0.0.1:30006
replicates 638ac8a174e8a66b1e46bb19155860a5c3aa415b
Can I set the above configuration? (type 'yes' to accept):
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 127.0.0.1:30001)
M: a403574ce40177866a188d16a615848bd5b5459a 127.0.0.1:30001
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 967818092ca216fe22a90160587fbdbfc23dd70b 127.0.0.1:30005
slots: (0 slots) slave
replicates a403574ce40177866a188d16a615848bd5b5459a
S: 00e70c8bebff2e59e89ed3e1e152d6805ad36eff 127.0.0.1:30006
slots: (0 slots) slave
replicates 638ac8a174e8a66b1e46bb19155860a5c3aa415b
S: 69313ced470329f55a5bdf408e4285f345a24154 127.0.0.1:30004
slots: (0 slots) slave
replicates 6a2c3dd007e7d44c95c51d33002e9cc0fdc429ba
M: 638ac8a174e8a66b1e46bb19155860a5c3aa415b 127.0.0.1:30002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
M: 6a2c3dd007e7d44c95c51d33002e9cc0fdc429ba 127.0.0.1:30003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


客户端
[lemcoden@hadoop01 create-cluster]$ redis-cli -p 30001
127.0.0.1:30001> set k1 sdfsdf
(error) MOVED 12706 127.0.0.1:30003
127.0.0.1:30001>
[lemcoden@hadoop01 create-cluster]$ redis-cli -p 30001
127.0.0.1:30001> set k1 sdfsdf
(error) MOVED 12706 127.0.0.1:30003
127.0.0.1:30001>
[lemcoden@hadoop01 create-cluster]$ redis-cli -c -p 30001 (cluster集群模式)
127.0.0.1:30001> set k1 sdfsdf
-> Redirected to slot [12706] located at 127.0.0.1:30003
OK
127.0.0.1:30003> get k1
"sdfsdf"
127.0.0.1:30003> set k2 sdfsdsf
-> Redirected to slot [449] located at 127.0.0.1:30001
OK
127.0.0.1:30001> get k2
"sdfsdsf"
127.0.0.1:30001> get k1
-> Redirected to slot [12706] located at 127.0.0.1:30003
"sdfsdf"
127.0.0.1:30003> WATCH k2
-> Redirected to slot [449] located at 127.0.0.1:30001
OK
127.0.0.1:30001> MULTI
OK
127.0.0.1:30001> set k1 232323456
-> Redirected to slot [12706] located at 127.0.0.1:30003
OK
127.0.0.1:30003> set sdfsdf sdfsdf
OK
127.0.0.1:30003> exec
(error) ERR EXEC without MULTI
(事务中重定向不同的进程的命令会失败)

127.0.0.1:30003> set {oo}k1 sdfsdf
-> Redirected to slot [1629] located at 127.0.0.1:30001
OK
127.0.0.1:30001> set {oo}k2 sdfsdf
OK
127.0.0.1:30001> set {oo}k3 sdfsdf
OK
127.0.0.1:30001> WATCH {oo}k1
OK
127.0.0.1:30001> MULTI
OK
127.0.0.1:30001> set {oo}k2 234234
QUEUED
127.0.0.1:30001> get {oo}k3
QUEUED
127.0.0.1:30001> exec
1) OK
2) "sdfsdf"
(可以在事务中给key添加相同的前缀)

客户端分赃

1
2
[lemcoden@hadoop01 create-cluster]$ ./create-cluster start
[lemcoden@hadoop01 create-cluster]$ redis-cli --cluster create 127.0.0.1:30001 127.0.0.1:30002 127.0.0.1:30003 127.0.0.1:30004 127.0.0.1:30005 127.0.0.1:30006 --cluster-replicas 1

客户端重新分片

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
[lemcoden@hadoop01 create-cluster]$ redis-cli --cluster reshard 127.0.0.1:30001
>>> Performing Cluster Check (using node 127.0.0.1:30001)
M: 8271c8dcb899de726fe6c6ffe0ebcdb9a494eb0a 127.0.0.1:30001
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: c41870d1a555d7674b8a13f2e52cfd926113fd1e 127.0.0.1:30004
slots: (0 slots) slave
replicates 0d0f70ad6b408f7c56897450eb605aa9e95e10ab
M: 0d0f70ad6b408f7c56897450eb605aa9e95e10ab 127.0.0.1:30002
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
M: 55c02c59f51913c654087fbf0ab2e8ee2464d017 127.0.0.1:30003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 8ccf40a81064e64afb56999dbd931636e7b49519 127.0.0.1:30005
slots: (0 slots) slave
replicates 55c02c59f51913c654087fbf0ab2e8ee2464d017
S: 85aa8ac690ba144599e7fb47b733d22dc365c036 127.0.0.1:30006
slots: (0 slots) slave
replicates 8271c8dcb899de726fe6c6ffe0ebcdb9a494eb0a
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 2000
What is the receiving node ID? 0d0f70ad6b408f7c56897450eb605aa9e95e10ab
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1: 8271c8dcb899de726fe6c6ffe0ebcdb9a494eb0a
Source node #2: done

检查使用info或者check命令
[lemcoden@hadoop01 ~]$ redis-cli --cluster info 127.0.0.1:30001
127.0.0.1:30001 (8271c8dc...) -> 0 keys | 3461 slots | 1 slaves.
127.0.0.1:30002 (0d0f70ad...) -> 0 keys | 7462 slots | 1 slaves.
127.0.0.1:30003 (55c02c59...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.

[lemcoden@hadoop01 ~]$ redis-cli --cluster check 127.0.0.1:30001
127.0.0.1:30001 (8271c8dc...) -> 0 keys | 3461 slots | 1 slaves.
127.0.0.1:30002 (0d0f70ad...) -> 0 keys | 7462 slots | 1 slaves.
127.0.0.1:30003 (55c02c59...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 127.0.0.1:30001)
M: 8271c8dcb899de726fe6c6ffe0ebcdb9a494eb0a 127.0.0.1:30001
slots:[2000-5460] (3461 slots) master
1 additional replica(s)
S: c41870d1a555d7674b8a13f2e52cfd926113fd1e 127.0.0.1:30004
slots: (0 slots) slave
replicates 0d0f70ad6b408f7c56897450eb605aa9e95e10ab
M: 0d0f70ad6b408f7c56897450eb605aa9e95e10ab 127.0.0.1:30002
slots:[0-1999],[5461-10922] (7462 slots) master
1 additional replica(s)
M: 55c02c59f51913c654087fbf0ab2e8ee2464d017 127.0.0.1:30003
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 8ccf40a81064e64afb56999dbd931636e7b49519 127.0.0.1:30005
slots: (0 slots) slave
replicates 55c02c59f51913c654087fbf0ab2e8ee2464d017
S: 85aa8ac690ba144599e7fb47b733d22dc365c036 127.0.0.1:30006
slots: (0 slots) slave
replicates 8271c8dcb899de726fe6c6ffe0ebcdb9a494eb0a
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


数据库设计

缓存:数据可以丢 急速!

数据库:数据绝对不能丢 速度+持久性

掉电易失!

redis+mysql > 数据库 < 不太对

redis如何持久化

存储层:

1.快照/副本

2.日志

从linux进程开始聊

graph LR
	id1[单机持久化] --> id2{RDB} 
	id2 --> id4[时点性]
	id2 --> id6[save]
	id6 --> id10[明确:比如,关机维护]
	id2 --> id7[bgsave]
	id7 --> id8[fork创建子进程]
	id2 --> id9[配置文件中给出bgsave的规则:save这个标识]
	id2 --> id11[弊端

1.不支持拉链,只有一个dump.rdb

2.丢失数据相对多,
时点与时点之间窗口数据容易丢失
8点得到一个rdb,9点刚要落一个rdb,挂机了] id2 --> id12[优点
类似java中序列化,恢复的速度相对快]
graph LR
	id101[如果果开启了AOF,只会用AOF恢复,
4.0之后,AOF中包含RDB全量,增加记录新的写操作]
1
2
3
4
5
6
7
8
[lemcoden@hadoop01 ~]$ num=0
[lemcoden@hadoop01 ~]$ ((num++))
[lemcoden@hadoop01 ~]$ echo $num
1
[lemcoden@hadoop01 ~]$ ((num++)) | echo ok
ok
[lemcoden@hadoop01 ~]$ echo $num
1

管道

  1. 衔接,前一个命令的输出作为后一个命令的输入

  2. 管道会触发子进程

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[lemcoden@hadoop01 ~]$ num=0
[lemcoden@hadoop01 ~]$ ((num++))
[lemcoden@hadoop01 ~]$ echo $num
1
[lemcoden@hadoop01 ~]$ ((num++)) | echo ok
ok
[lemcoden@hadoop01 ~]$ echo $num
1
[lemcoden@hadoop01 ~]$ echo $$
2236
[lemcoden@hadoop01 ~]$ echo $$ | more
2236
[lemcoden@hadoop01 ~]$ echo $BASHPID
2236
[lemcoden@hadoop01 ~]$ echo $BASHPID | more
8130
[lemcoden@hadoop01 ~]$ echo $BASHPID | more
8132
[lemcoden@hadoop01 ~]$ echo $BASHPID | more
8134
[lemcoden@hadoop01 ~]$ echo $BASHPID | more
8136

进程演示

graph LR
	id1[使用linux的时候:

父子进程

常规思想,进程数据隔离的

进阶思想,父进程其实可以让子进程看到数据

linux中
export的环境变量,子进程的修改不会影响到父进程

父进程不会破坏子进程
] --> id2[创建子进程应该是什么程度
如果父进程是redis,内存数据10G
1.速度
2.内存空间够不够] id2-->id3(("fork")) id3 --> id4["1.速度:快
2.空间:小"]
1
2
3
4
5
6
7
8
[lemcoden@hadoop01 ~]$ echo $num

[lemcoden@hadoop01 ~]$ exit
exit
[lemcoden@hadoop01 ~]$ export num
[lemcoden@hadoop01 ~]$ /bin/bash
[lemcoden@hadoop01 ~]$ echo $num
1

test.sh

1
2
3
4
5
6
7
8
9
#!/bin/bash
echo $$
echo $num
num=999
echo num:$num

sleep 20

echo $num

执行状况(子进程不会修改父进程)

1
2
3
4
5
6
7
8
9
[lemcoden@hadoop01 ~]$ ./test.sh &
[1] 8558
[lemcoden@hadoop01 ~]$ 8558
1
num:999

[lemcoden@hadoop01 ~]$ echo $$
8454
[lemcoden@hadoop01 ~]$ 999

执行状况(父进程不会破坏子进程)

1
2
3
4
5
6
7
8
9
10
11
12
[lemcoden@hadoop01 ~]$ ./test.sh &
[1] 8623
[lemcoden@hadoop01 ~]$ 8623
1
num:999

[lemcoden@hadoop01 ~]$ echo $num
1
[lemcoden@hadoop01 ~]$ num=888
[lemcoden@hadoop01 ~]$ echo $num
888
[lemcoden@hadoop01 ~]$ 999

RDB

redis实现持久化需要解决两个问题:

  1. 非阻塞,redis继续对外提供服务

  2. 将数据落地

通过fork和copy on write实现

时点.png

RDB配置要点


1
2
3
4
5
6
7
8
9
10
11
12
13
14
######################## SNAPSHOTTING  ########################
# save ""

save 900 1
save 300 10
save 60 10000

rdbcompression yes

rdbchecksum yes 文件末尾写入校验位

dbfilename dump.rdb

dir /var/lib/redis/6379

为什么一般redis设置1到10个G,而不占满内存?

因为做数据持久化的时候,磁盘IO的瓶颈,为了保证redis的速度,不能有太大的数据量做持久化.

AOF

graph LR
	id[主从复制] --> id3{AOF}
	id3 --> id5[redis的写操作记录到文件中]
	id5 --> id22[丢失数据少]
	id5 --> id23[rdb和aof可以同时开启]
	id5 --> id24[弊端:体量无限变大,恢复慢]
	id24 --> id25[日志:优点如果能保住
还是可以用的

设计方案->AOF足够小]
graph LR
	hdfs[hdfs:fsimage+edits.log

让日志记录增量

合并过程] hdfs --> id0[4.0以前] id0 --> id2[重写
删除抵消的命令
合并重复的命令] id2 --> id4[最终也是纯指令文件] hdfs --> id10[4.0以后] id10 -->id11[重写
将老的数据RDB到aof文件中
将增量以指令的方式
append到AOF] id11 --> id12[AOF是一个混合体,利用RDB的快,日志的全量]

redis运行了10年

开启了AOF

10年头,redis挂了

  1. AOF多大:10T

    恢复:会不会溢出

  2. 恢复要多久:恢复5年

redis内存数据库

原点:redis是内存数据库

写操作会触发IO

NO

aways

每秒

graph TB
	id1[java] --> id2[kenel fd8 buffer]
	id2 --> id3[磁盘]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
############################## APPEND ONLY MODE ###############################
appendonly yes
appendfilename "appendonly.aof
# appendfsync always
appendfsync everysec
# appendfsync no 等内核buffer快满了.kernel自动flush

no-appendfsync-on-rewrite no
redis有一个正在进行rdb的子进程时,此值为true,就不会产生AOF写
???
# When rewriting the AOF file, Redis is able to use an RDB preamble in the
# AOF file for faster rewrites and recoveries. When this option is turned
# on the rewritten AOF file is composed of two different stanzas:
#
# [RDB file][AOF tail]
#
# When loading Redis recognizes that the AOF file starts with the "REDIS"
# string and loads the prefixed RDB file, and continues loading the AOF
# tail.
aof-use-rdb-preamble yes

AOF实操

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
daemonize no
#logfile /var/log/redis_6379
aof-use-rdb-preamble no
[lemcoden@hadoop01 6379]$cd /var/lib/redis/6379
[lemcoden@hadoop01 6379]$rm -f dump.rdb
[lemcoden@hadoop01 6379]$ systemctl stop redis_6379
[lemcoden@hadoop01 6379]$ sudo ./redis-server /etc/redis/6379.conf
[lemcoden@hadoop01 6379]$ redis-cli
127.0.0.1:6379> set k1 hello
OK
[lemcoden@hadoop01 6379]$ vim appendonly.aof
*2
$6
SELECT
$1
0
*3
$3
set
$2
k1
$5
hello
[lemcoden@hadoop01 6379]$ redis-cli
127.0.0.1:6379> bgsave
[lemcoden@hadoop01 6379]$ redis-check-rdb dump.rdb
[offset 0] Checking RDB file dump.rdb
[offset 26] AUX FIELD redis-ver = '5.0.5'
[offset 40] AUX FIELD redis-bits = '64'
[offset 52] AUX FIELD ctime = '1605840860'
[offset 67] AUX FIELD used-mem = '853472'
[offset 83] AUX FIELD aof-preamble = '0'
[offset 85] Selecting DB ID 0
[offset 107] Checksum OK
[offset 107] \o/ RDB looks OK! \o/
[info] 1 keys read
[info] 0 expires
[info] 0 already expired

重写AOF

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[lemcoden@hadoop01 6379]$ redis-cli 
127.0.0.1:6379> set k1 a
OK
127.0.0.1:6379> set k1 b
OK
127.0.0.1:6379> set k1 c
OK
127.0.0.1:6379> get k1
"c"
127.0.0.1:6379> BGREWRITEAOF
[lemcoden@hadoop01 6379]$ vim appendonly.aof
Backgrou*2
$6
SELECT
$1
0
*3
$3
SET
$2
k1
$1
c

AOF & RDB混合

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
aof-use-rdb-preamble yes
[lemcoden@hadoop01 6379]$ redis-cli
127.0.0.1:6379> set k1 a
OK
127.0.0.1:6379> set k1 b
OK
127.0.0.1:6379> set k1 c
OK
127.0.0.1:6379> get k1
"c"
127.0.0.1:6379> BGREWRITEAOF
[lemcoden@hadoop01 6379]$ vim appendonly.aof
REDIS0009ú redis-ver^E5.0.5ú
redis-bitsÀ@ú^EctimeÂK3·_ú^Hused-memÂÐ^E^M^@ú^Laof-preambleÀ^Aþ^@û^A^@^@^Bk1^Acÿ<84>^Q·<9e>^T"<96>]
~
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# Automatic rewrite of the append only file.
# Redis is able to automatically rewrite the log file implicitly calling
# BGREWRITEAOF when the AOF log size grows by the specified percentage.
#
# This is how it works: Redis remembers the size of the AOF file after the
# latest rewrite (if no rewrite has happened since the restart, the size of
# the AOF at startup is used).
#
# This base size is compared to the current size. If the current size is
# bigger than the specified percentage, the rewrite is triggered. Also
# you need to specify a minimal size for the AOF file to be rewritten, this
# is useful to avoid rewriting the AOF file even if the percentage increase
# is reached but it is still pretty small.
#
# Specify a percentage of zero in order to disable the automatic AOF
# rewrite feature.

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

邮箱收到github构建异常

三个月前,我的gmail收到一封关于hexo在github上构建异常的邮箱

邮箱的主要内容如下:

The page build failed for the master branch with the following error:

The symbolic link /blog_workspace targets a file which does not exist within your site’s repository.

Read more »

发展历程

数据库:表很大,性能下降?
如果表有索引,增删改变慢(需要维护索引)
查询速度会不会变慢:
1.一个或少量查询依然很快
2.并发大的时候会受硬盘带宽影响速度

Read more »

redis API 及其设计

graph LR
	id[value] --> id1[List 单向链表,双向链表,环形链表]
	id1 --> id2[list栈
同向命令] id1 --> id3[list队列
反向命令] id1 --> id4[数组] id1 --> id5[阻塞
单播队列
FIFO] id1 --> id6[Set] id1 --> id9[sorted Set] id6 --> id7[无序,去重] id100[成本思考,两次服务端通讯,keys*模式匹配成本高,mget] --> id101(hash
对field进行数值计算,场景:点赞,收藏,计算) id6 --> id8[随机事件]
Read more »

mysql四大排名函数

row_number: 连续 不重复

rank: 不连续 重复

dense_rank: 连续 重复

ntile:有参数 入参group_num, 将数据分成group_num个组排序编号

Read more »

1
2
3
4
5
6
7
$HIVE_SRC/build/dist/bin/hive 
--auxpath $HIVE_SRC/build/dist/lib/hive-hbase-handler-0.9.0.jar,
$HIVE_SRC/build/dist/lib/hbase-0.92.0.jar,
$HIVE_SRC/build/dist/lib/zookeeper-3.3.4.jar,
$HIVE_SRC/build/dist/lib/guava-r09.jar
--hiveconf
hbase.zookeeper.quorum=zk1.yoyodyne.com,zk2.yoyodyne.com,zk3.yoyodyne.com
Read more »

下载官方mysql源

1
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm

加载rpm源

1
rpm -ivh mysql-community-release-el7-5.noarch.rpm
Read more »