19c的 rac在oracle linux7.4上搭建总结

Posted 2022-01-29 ocp-100

准备：

1,ASM磁盘空间最低要求
OCR的磁盘占用需求有了明显增长。
为了方便操作，设置如下：
External: 1个卷x40G
Normal: 3个卷x30G
Hight: 5个卷x25G
Flex: 3个卷x30G
OCR+VOLTING+MGMT存储通常放到一个磁盘组，且选择Normal的冗余方式，也即最少3块asm磁盘80G空间。

2.操作系统检查
Red Hat Enterprise Linux 7: 3.10.0-123.el7.x86_64 or later
Red Hat Enterprise Linux 6.4: 2.6.32-358.el6.x86_64 or later

命令：
[[email protected] oracle]# lsb_release -id
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 6.8 (Santiago)

[[email protected] oracle]# uname -an
Linux B201I23Rs01 2.6.32-642.el6.x86_64 #1 SMP Wed Apr 13 00:51:26 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

3，服务器配置检查
内存 8G以上
/tmp 1g以上，建议2g
swap 4G以上
HugePages sga以上
oracle文件系统 100G(12G+9G)
/dev/shm

3.1 内存
[[email protected] oracle]# grep MemTotal /proc/meminfo
MemTotal: 264371924 kB

3.2 swap (大于16g)
[[email protected] oracle]# grep SwapTotal /proc/meminfo
SwapTotal: 1048572 kB

3.3 /tmp目录：
1 GB of space in the /tmp directory
[[email protected] oracle]# df -lh /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgroot-lv06
9.8G 23M 9.2G 1% /tmp
3.4 共享内存大小：
[[email protected] oracle]# df -h /dev/shm/
Filesystem Size Used Avail Use% Mounted on
tmpfs 127G 72K 127G 1% /dev/shm

4,操作系统包检查 (参考yum安装)

yum配置：
情景一：
盘已经挂载在某个路径下：
例如：挂载在/dev/sr1 下
umount /dev/sr1 umount原来目录下的安装介质
mount /dev/sr0 /mnt/cdrom mount到新的目录下
mount -t iso9660 -o,loop /dev/cdrom /mnt/cdrom
情景二：
iso镜像文件放在某个路径下
(例如：
mount -o loop OracleLinux-Release7-Update4-x86_64-dvd.iso /mnt/cdrom/
将某个路径下的iso文件以硬盘模式挂载到 /mnt/cdrom)

mount -o loop /tmp/V921569-01.iso /mnt

可以先
vi /etc/yum.conf --------gpgcheck=0 修改相关参数为0
cd /etc/yum.repos.d/ 复制一份.repo 出来 叫 abc.repo（修改里面 name，baseurl为你mount的路径 参考如下注意///个数，enable，gpgcheck，）其他repo可能会干扰，建议mv掉，或rm掉
vi /etc/yum.repos.d/abc.repo
[c6-media]
name=tOSia
baseurl=file:///mnt/cdrom
gpgcheck=0
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

yum clean all 清除yum缓存
yum makecache 缓存本地yum源
yum list 测试

yum -y install bc
yum -y install binutils
yum -y install compat-libcap1
yum -y install compat-libstdc++
yum -y install dtrace-modules
yum -y install dtrace-modules-headers*
yum -y install dtrace-modules-provider-headers*
yum -y install dtrace-utils*
yum -y install elfutils-libelf
yum -y install elfutils-libelf-devel
yum -y install fontconfig-devel
yum -y install glibc
yum -y install glibc-devel
yum -y install ksh
yum -y install libaio
yum -y install libaio-devel
yum -y install libdtrace-ctf-devel
yum -y install libXrender
yum -y install libXrender-devel
yum -y install libX11
yum -y install libXau
yum -y install libXi
yum -y install libXtst
yum -y install libgcc
yum -y install librdmacm-devel
yum -y install libstdc++
yum -y install libstdc++-devel
yum -y install libxcb
yum -y install make
yum -y install net-tools
yum -y install nfs-utils
yum -y install python
yum -y install python-configshell
yum -y install python-rtslib
yum -y install python-six
yum -y install targetcli
yum -y install smartmontools
yum -y install sysstat

5,创建用户和组、目录
# groupadd -g 54321 oinstall
# groupadd -g 54322 dba
# groupadd -g 54323 oper
# groupadd -g 54324 backupdba
# groupadd -g 54325 dgdba
# groupadd -g 54326 kmdba
# groupadd -g 54327 asmdba
# groupadd -g 54328 asmoper
# groupadd -g 54329 asmadmin
# groupadd -g 54330 racdba
# useradd -u 54340 -g oinstall -G asmadmin,asmdba,racdba,asmoper,racdba -m -d /home/grid -s /bin/bash grid
# usermod -u 54321 -g oinstall -G dba,backupdba,dgdba,kmdba,asmdba,racdba,asmoper,oper -m -d /home/oracle -s /bin/bash oracle
# mkdir -p /u01/app/19.0.0/grid
# mkdir -p /u01/app/grid
# mkdir -p /u01/app/oracle
# chown -R grid:oinstall /u01
# chmod -R 775 /u01/
# chown oracle:oinstall /u01/app/oracle

6,
DISABLE FIREWALL
# service iptables stop
# chkconfig iptables off
# chkconfig --list iptables

systemctl list-unit-files

systemctl stop firewalld

systemctl disable firewalld

service firewalld status
iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off



DISABLE SELINUX
# vi /etc/selinux/config
SELINUX=disable
# more /etc/selinux/config |grep SELINUX
# SELINUX= can take one of these three values:
SELINUX=disable
# SELINUXTYPE= can take one of these two values:
SELINUXTYPE=targeted

8.启用nfs文件系统的相关服务
chkconfig --list nscd
chkconfig --level 35 nscd on
service nscd start
service nscd restart

9.修改ssh时间等待为无限制

vi /etc/ssh/sshd_config

LoginGraceTime 0

10,禁用ntp
/sbin/service ntpd stop

chkconfig ntpd off

mv /etc/ntp.conf /etc/ntp.conf.org
rm /var/run/ntpd.pid

Complete these step on Oracle Linux 7 and Red Hat Linux 7:

systemctl status ntpd.service

Run the following commands as the root user:

# systemctl stop chronyd
# systemctl disable chronyd

mv /etc/chrony* /etc/chrony.conf.bak

13.修改 /etc/security/limits.conf 文件中的资源限制
oracle soft nproc 16384
oracle hard nproc 16384
oracle soft nofile 65536
oracle hard nofile 65536
oracle soft memlock 16777216
oracle hard memlock 16777216
oracle soft stack 10240
grid soft nproc 16384
grid hard nproc 16384
grid soft nofile 65536
grid hard nofile 65536
grid soft memlock 16777216
grid hard memlock 16777216
grid soft stack 10240

14.修改文件/etc/sysctl.conf 中的内核参数
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 26437192
kernel.shmmax = 135358425088 107374182400
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
#kernel.sem = 250 32000 100 128
kernel.sem = 5010 641280 5010 1280
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=1048586
net.ipv4.ip_local_range=9000 65500

net.ipv4.conf.ib2.rp_filter = 0
net.ipv4.conf.ib0.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0

****针对rp_filter部分，如果没有多张私有网卡，则不需要配置，文档 ID1286796.1有描述。

###注：这里shmmax设为120g，大约物理内存的一半
sysctl -p //立即生效
sysctl -a

To check resource limits:

Log in as an installation owner.
Check the soft and hard limits for the file descriptor setting. Ensure that the result is in the recommended range. For example:
$ ulimit -Sn
1024
$ ulimit -Hn
65536
Check the soft and hard limits for the number of processes available to a user. Ensure that the result is in the recommended range. For example:
$ ulimit -Su
2047
$ ulimit -Hu
16384
Check the soft limit for the stack setting. Ensure that the result is in the recommended range. For example:
$ ulimit -Ss
10240
$ ulimit -Hs
32768
Repeat this procedure for each Oracle software installation owner.

15.nfs参数注意

rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600

 

***
配置NOZEROCONF
vi /etc/sysconfig/network增加以下内容
NOZEROCONF=yes

16.修改 /etc/pam.d/login 文件
session required pam_limits.so

加载pam_limits.so模块
echo "session required pam_limits.so" >> /etc/pam.d/login

17.如果您启动了 selinux , 请 disable 掉。 它可能导致后续的 asmlib 无法创建磁盘。
edit file /etc/selinux/config.
SELINUX=disabled

18. 如果启用了 iptables，请 disable 掉.
# /etc/init.d/iptables save
# /etc/init.d/iptables stop
# chkconfig iptables off
----------------------------
3、重启防火墙
systemctl restart iptables.service #最后重启防火墙使配置生效
systemctl enable iptables.service #设置防火墙开机启动

4、其他相关命令
systemctl disable iptables #禁止iptables服务
systemctl stop iptables #暂停服务
systemctl enable iptables #解除禁止iptables
systemctl start iptables #开启服务

service iptables save #因为已经添加iptables服务，故该命令可以运行；若没有之前的添加服务则不能运行
service iptables status #查看iptables服务状态
-----------------------------------

19.网络参数调整

Check your TCP network buffer size to ensure that it is adequate for the speed of your servers.

By default, the network buffer size is set to 1 MB for TCP, and 2 MB for UDP. The TCP buffer size can set a limit on file transfers, which can negatively affect performance for Direct NFS Client users.
To check the current TCP buffer size, enter the following command:

# sysctl -a |grep -e net.ipv4.tcp_[rw]mem
The output of this command is similar to the following:

net.ipv4.tcp_rmem = 4096 87380 1048576
net.ipv4.tcp_wmem = 4096 16384 1048576
Oracle recommends that you set the value based on the link speed of your servers. For example, perform the following steps:

As root, use a text editor to open /etc/sysctl.conf, and add or change the following:

net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
Apply your changes by running the following command:

# sysctl -p
Restart the network:

# /etc/rc.d/init.d/network restart

20. 设置用户配置文件
# su - oracle
umask 0022
# Oracle Environment
export ORACLE_UNQNAME=cendb
export ORACLE_BASE=/oracle/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/18.0.0/dbhome_1
export GRID_HOME=/oracle/app/18.0.0/grid
export ORACLE_SID=cendb2
export PATH=.:$ORACLE_HOME/OPatch:$ORACLE_HOME/bin:$GRID_HOME/bin:/home/oracle/support/shell:$PATH
export ORA_NLS10=$GRID_HOME/nls/data
export LIBPATH=$ORACLE_HOME/lib
export AIXTHREAD_SCOPE=S
export ORACLE_TERM=vt100
export TMP=/tmp
export TMPDIR=/tmp
NLS_LANG=AMERICAN_AMERICA.zhs16gbk
export NLS_LANG
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib;
export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib;
export CLASSPATH
NLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS";
export NLS_DATE_FORMAT
stty erase ^H
echo ‘ ‘
echo ‘$ORACLE_SID: ‘$ORACLE_SID
echo ‘$ORACLE_HOME: ‘$ORACLE_HOME
echo ‘$GRID_HOME: ‘$GRID_HOME
echo ‘ ‘

# Set up the shell variables:
EDITOR=vi
export EDITOR
set -o vi
export PS1=`whoami`@`hostname`\[‘$ORACLE_SID‘\]‘:$PWD$ ‘

# ALIAS
alias psg="ps -ef | grep"
alias ll="ls -lrta"
alias ss="sqlplus ‘/as sysdba‘"
alias oh="cd $ORACLE_HOME"
alias ohbin="cd $ORACLE_HOME/bin"
alias crs="cd $GRID_HOME"
alias crsbin="cd $GRID_HOME/bin"
alias agent="cd $AGENT_HOME/bin"
alias wait="/home/oracle/support/shell/wait.sh"

export LANG=en_US
alias alert="cd /oracle/app/oracle/diag/rdbms/$ORACLE_UNQNAME/$ORACLE_SID/trace"

#su - grid

umask 022
# Oracle Environment
export ORACLE_BASE=/oracle/app/grid
export GRID_HOME=/oracle/app/18.0.0/grid
export ORACLE_HOME=/oracle/app/18.0.0/grid
export ORACLE_SID=+ASM1
export PATH=$GRID_HOME/bin:$ORACLE_HOME/OPatch:$ORACLE_HOME/bin:.:$PATH
export ORA_NLS10=$ORACLE_HOME/nls/data
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/precomp/lib:$LD_LIBRARY_PATH
export LIBPATH=$ORACLE_HOME/lib
export AIXTHREAD_SCOPE=S
export ORACLE_TERM=vt100
export TMP=/tmp
export TMPDIR=/tmp
NLS_LANG=AMERICAN_AMERICA.zhs16gbk
export NLS_LANG
echo ‘ ‘
echo ‘$ORACLE_SID: ‘$ORACLE_SID
echo ‘$ORACLE_HOME: ‘$ORACLE_HOME
echo ‘$GRID_HOME: ‘$GRID_HOME
echo ‘ ‘

# Set up the shell variables:
EDITOR=vi
export EDITOR
set -o vi
export PS1=`whoami`@`hostname`\[‘$ORACLE_SID‘\]‘:$PWD$ ‘
#ALIAS
alias psg="ps -ef | grep"
alias ll="ls -lrta"
alias oh="cd $ORACLE_HOME"
alias ohbin="cd $ORACLE_HOME/bin"
alias crs="cd $GRID_HOME"
alias crsbin="cd $GRID_HOME/bin"

export LANG=en_US

#su -

# vi .bash_profile
export CRS_HOME=/oracle/app/18.0.0/grid
PATH=$PATH:$CRS_HOME/bin

21,修改/etc/hosts

10.210.114.163 cendb1
10.210.114.164 cendb2
10.210.114.165 cendb1-vip
10.210.114.166 cendb2-vip
10.210.114.167 cendb-scan
192.168.10.163 cendb1_priv
192.168.10.164 cendb2_priv

 

10.0.0.130 cendb1
10.0.0.131 cendb2
200.0.0.130 cendb1-priv
200.0.0.131 cendb2-priv
10.0.0.132 cendb1-vip
10.0.0.133 cendb2-vip
10.0.0.134 cendb-scan

 

*******
解压zip文件

grid

unzip gird.zip -d /u01/app/19.0.0/grid

22, 安装cvuqdisk
cvuqdisk存于oracle GI安装介质的cv/rpm目录下
export CVUQDISK_GRP=oinstall
rpm –ivh cvuqdisk-1.0.10-1.rpm

22，重启主机后，配置互信

方法：使用sshUserSetup.sh快速创建互信
sshUserSetup.sh在GI安装介质解压缩后的oui/prov/resources/scripts目录下。

下面两条命令在一个节点上执行即可（可以在root用户下执行）：
#./sshUserSetup.sh -user grid -hosts "ylzserver1 ylzserver2" -advanced -exverify –confirm

#./sshUserSetup.sh -user oracle -hosts "ylzserver1 ylzserver2" -advanced -exverify -confirm

23，asm磁盘配置

24，grid安装
24.1 安装前检查
--全面检查：
./runcluvfy.sh stage -pre crsinst -n ylzserver1,ylzserver2 -fixup -verbose

--检查网络和等效性
./runcluvfy.sh comp nodecon -nrac1,rac2 -verbose

mkdir -p /u01/app/oracle/product/19.0.0/dbhome_1

*************安装遇到问题********
ora-12537
原因：二节点目录权限不对，修改以下权限（在dbca之前）
cd /oracle/app/oracle/

chmod 755 product
chmod 755 18.0.0
chmod 755 dbhome_1

rehat7.4和oracle linux 7.4需要升级以下两个包到对应的版本 ，可在http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/index.html寻找下载

kmod-20-21 (x86_64)    
kmod-libs-20-21 (x86_64)

多路径软件安装卸载：

3. udev配盘-参考 udevudev 配盘

scsi_id --whitelisted --replace-whitespace --device=/dev/sdc

xiv_devlist

[[email protected] soft]# scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/mpathc
200173800d320003b

scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/mpathbp1

scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/mpathb
scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/mpathc
scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/mpathd
scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/mpathe
scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/mpathf
scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/mpathg

/sbin/scsi_id -g -u -d /dev/mapper/mpathc

76 77
200173800d320003b /dev/mapper/mpathb b
200173800d320003a /dev/mapper/mpathc 9
200173800d3200039 /dev/mapper/mpathd c
200173800d320003c /dev/mapper/mpathe d
200173800d320003d /dev/mapper/mpathf a
200173800d320003e /dev/mapper/mpathg e

OceanStor_UltraPath_21.2.0_RHEL.zip sh install.sh
echo "- - -" > /sys/class/scsi_host/host6/scan
lspci
lspath
upadmin
vi 99-my-asmdevices.rules
/sbin/udevadm control --reload
udevadm control --reload
/sbin/udevadm trigger --type=devices --action=change

1. 编辑/etc/scsi_id.config文件，如果该文件不存在，则创建该文件，添加如下行：

options=-g
孙维亮 18/1/24 16:00:09
vi /etc/udev/rules.d/99-oracle-asmdevices.rules
孙维亮 18/1/24 16:00:15
爱信不信 2018/01/24 16:00:15
KERNEL=="mpth?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/mapper/$parent", RESULT=="200173800d320003b", NAME="asm-disk1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="mpth?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/mapper/$parent", RESULT=="200173800d320003a", NAME="asm-disk2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="mpth?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/mapper/$parent", RESULT=="200173800d3200039", NAME="asm-disk3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="mpth?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/mapper/$parent", RESULT=="200173800d320003c", NAME="asm-disk4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="mpth?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/mapper/$parent", RESULT=="200173800d320003d", NAME="asm-disk5", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="mpth?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/mapper/$parent", RESULT=="200173800d320003e", NAME="asm-disk6", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="200173800d320003b", NAME="asm-disk1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="200173800d320003a", NAME="asm-disk2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="200173800d3200039", NAME="asm-disk3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="200173800d320003c", NAME="asm-disk4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="200173800d320003d", NAME="asm-disk5", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="200173800d320003e", NAME="asm-disk6", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="mpath*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/mpathc", RESULT=="200173800d320003a", NAME="asm-disk2", OWNER="grid", GROUP="asmadmin", MODE="0660"

partprobe /dev/mapper/mpathb
udevadm control --reload-rules
/sbin/start_udev

KERNEL=="mpath*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/mapper/$name", RESULT=="200173800d320003b", NAME="asm-disk1", OWNER="grid", GROUP="asmadmin", MODE="0660"

KERNEL=="sd*[!0-9]", ENVDEVTYPE=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="363c7843100ba123b43600d8200000080", RUN+="/bin/sh -c ‘/usr/bin/mkdir /dev/asm; mknod /dev/asm/ocr01 b $major $minor; chown grid:asmadmin /dev/asm/ocr01; chmod 0660 /dev/asm/ocr01‘"
KERNEL=="sd*[!0-9]", ENVDEVTYPE=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="363c7843100ba123b0d93e8c100000039", RUN+="/bin/sh -c ‘/usr/bin/mkdir /dev/asm; mknod /dev/asm/ocr01 b 8 96; chown grid:asmadmin /dev/asm/ocr01; chmod 0660 /dev/asm/ocr01‘"