一次误操作引起的linux系统网络故障
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了一次误操作引起的linux系统网络故障相关的知识,希望对你有一定的参考价值。
1、故障描述
接到用户报障,生产某系统无法访问。同事接到报障后立即排查,经测试,系统确实无法访问,并且无法ping通服务器。
2、故障处理
由于客户端无法ping通服务器,需要进入机房查看。经查看,服务器硬件无报警,系统无重启。登录系统使用ifconfig命令查看,IP丢失(eth0不存在),紧接打开网卡配置目录/etc/sysconfig/network-scripts,发现网卡文件ifcfg-eth0丢失,只存在之前备份的ifcfg-eth0.bak文件和ifcfg-peth0文件。根据先抢通业务后处理故障原则,通过备份的文件复制一份进行修复,重启network服务,故障解决。
3、故障分析
3.1经了解,故障发生时,有一同事正在登录系统查询安全基线配置,但同事坚称并未进行rm或者mv网卡文件操作。通过history命令得知,该同事确实未执行rm或者mv操作,只执行了chkconfig --list命令,但却不小心把原本需要复制的内容误操作的当作命令去执行了,历史记录如下:
883 chkconfig --list 884 NetworkManager 0:off 1:off 2:off 3:off 4:off 5:off 6:off 885 PowerIscsi 0:off 1:off 2:off 3:on 4:off 5:on 6:off 886 PowerMig 0:off 1:off 2:off 3:on 4:off 5:on 6:off 887 PowerMigRecoverAll 0:off 1:off 2:off 3:on 4:off 5:on 6:off 888 acpid 0:off 1:off 2:on 3:on 4:on 5:on 6:off 889 anacron 0:off 1:off 2:on 3:on 4:on 5:on 6:off 890 atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off 891 auditd 0:off 1:off 2:on 3:on 4:on 5:on 6:off 892 autofs 0:off 1:off 2:off 3:on 4:on 5:on 6:off 893 avahi-daemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off 894 avahi-dnsconfd 0:off 1:off 2:off 3:off 4:off 5:off 6:off 895 bluetooth 0:off 1:off 2:on 3:on 4:on 5:on 6:off 896 capi 0:off 1:off 2:off 3:off 4:off 5:off 6:off 897 conman 0:off 1:off 2:off 3:off 4:off 5:off 6:off 898 coremail 0:off 1:off 2:on 3:on 4:on 5:on 6:off 899 cpuspeed 0:off 1:on 2:on 3:on 4:on 5:on 6:off 900 crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off 901 cups 0:off 1:off 2:on 3:on 4:on 5:on 6:off 902 dnsmasq 0:off 1:off 2:off 3:off 4:off 5:off 6:off 903 dund 0:off 1:off 2:off 3:off 4:off 5:off 6:off 904 ebtables 0:off 1:off 2:off 3:off 4:off 5:off 6:off 905 firstboot 0:off 1:off 2:off 3:on 4:off 5:on 6:off 906 gpm 0:off 1:off 2:on 3:on 4:on 5:on 6:off 907 haldaemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off 908 hidd 0:off 1:off 2:on 3:on 4:on 5:on 6:off 909 hplip 0:off 1:off 2:on 3:on 4:on 5:on 6:off 910 httpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off 911 ip6tables 0:off 1:off 2:on 3:on 4:on 5:on 6:off 912 ipmi 0:off 1:off 2:off 3:off 4:off 5:off 6:off 913 iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off 914 irda 0:off 1:off 2:off 3:off 4:off 5:off 6:off 915 irqbalance 0:off 1:off 2:on 3:on 4:on 5:on 6:off 916 iscsi 0:off 1:off 2:off 3:on 4:on 5:on 6:off 917 iscsid 0:off 1:off 2:off 3:on 4:on 5:on 6:off 918 isdn 0:off 1:off 2:on 3:on 4:on 5:on 6:off 919 kdump 0:off 1:off 2:off 3:off 4:off 5:off 6:off 920 kudzu 0:off 1:off 2:off 3:on 4:on 5:on 6:off 921 libvirt-guests 0:off 1:off 2:off 3:on 4:on 5:on 6:off 922 libvirtd 0:off 1:off 2:off 3:on 4:on 5:on 6:off 923 lvm2-monitor 0:off 1:on 2:on 3:on 4:on 5:on 6:off 924 mcstrans 0:off 1:off 2:on 3:on 4:on 5:on 6:off 925 mdmonitor 0:off 1:off 2:on 3:on 4:on 5:on 6:off 926 mdmpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off 927 messagebus 0:off 1:off 2:off 3:on 4:on 5:on 6:off 928 microcode_ctl 0:off 1:off 2:on 3:on 4:on 5:on 6:off 929 multipathd 0:off 1:off 2:off 3:off 4:off 5:off 6:off 930 named 0:off 1:off 2:off 3:off 4:off 5:off 6:off 931 netbackup 0:off 1:off 2:on 3:on 4:off 5:on 6:off 932 netconsole 0:off 1:off 2:off 3:off 4:off 5:off 6:off 933 netfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off 934 netplugd 0:off 1:off 2:off 3:off 4:off 5:off 6:off 935 network 0:off 1:off 2:on 3:on 4:on 5:on 6:off 936 nfs 0:off 1:off 2:off 3:off 4:off 5:off 6:off 937 nfslock 0:off 1:off 2:off 3:on 4:on 5:on 6:off 938 nscd 0:off 1:off 2:off 3:off 4:off 5:off 6:off 939 ntpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off 940 pand 0:off 1:off 2:off 3:off 4:off 5:off 6:off 941 pcscd 0:off 1:off 2:on 3:on 4:on 5:on 6:off 942 portmap 0:off 1:off 2:off 3:on 4:on 5:on 6:off 943 psacct 0:off 1:off 2:off 3:off 4:off 5:off 6:off 944 rawdevices 0:off 1:off 2:off 3:on 4:on 5:on 6:off 945 rdisc 0:off 1:off 2:off 3:off 4:off 5:off 6:off 946 readahead_early 0:off 1:off 2:on 3:on 4:on 5:on 6:off 947 readahead_later 0:off 1:off 2:off 3:off 4:off 5:on 6:off 948 restorecond 0:off 1:off 2:on 3:on 4:on 5:on 6:off 949 rhnsd 0:off 1:off 2:off 3:on 4:on 5:on 6:off 950 rpcgssd 0:off 1:off 2:off 3:on 4:on 5:on 6:off 951 rpcidmapd 0:off 1:off 2:off 3:on 4:on 5:on 6:off 952 rpcsvcgssd 0:off 1:off 2:off 3:off 4:off 5:off 6:off 953 saslauthd 0:off 1:off 2:off 3:off 4:off 5:off 6:off 954 sendmail 0:off 1:off 2:off 3:off 4:off 5:off 6:off
以上操作记录表面看起来,并无异常。
3.2通过查看系统日志messages,发现有“removed ifcfg-eth0”关键字,发生的时间与同事误操作的时间吻合,如下:
Mar 21 09:46:50 localhost nm-system-settings: ifcfg-rh: removed /etc/sysconfig/network-scripts/ifcfg-eth0. Mar 21 09:46:50 localhost nm-system-settings: ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-peth0 ... Mar 21 09:46:50 localhost nm-system-settings: ifcfg-rh: read connection ‘System peth0‘ Mar 21 09:46:50 localhost nm-system-settings: ifcfg-rh: updating /etc/sysconfig/network-scripts/ifcfg-peth0
同事既然没有误操作,那为什么会有remove网卡文件的日志呢?难道被黑了?还是有其它原因?
3.3查看日志secure和命令last,并未发现异常登录IP,先排除被黑可能性,着重排查同事误操作的命令中,哪一条才是引起网卡文件丢失的。
3.4再一次确认3.1的history操作记录,表面看上去确实没有什么异常,而且都是chkconfig --list的输出内容,百思不得其解。
3.5查问题,看日志。只能通过仔细的分析message日志查找一点蛛丝马迹。从3.2的日志来看,当看到
Mar 21 09:46:50 localhost nm-system-settings: ifcfg-rh: parsing /etc/sysconfig/network-scripts/ifcfg-peth0 ...
时,发现“ifcfg-peth0”这个网卡文件很可疑,该文件应该跟XEN虚拟化有关,但这个系统并未使用到XEN虚拟化。
3.6登录系统确认,系统虽未使用虚拟化,但前期安装时安装了XEN虚拟化,并且加载了kernel-xen内核和启动了xend服务:
1)[[email protected]~]# uname -r 2.6.18-238.el5xen 2)# /etc/init.d/xend status xend is running
3.7查看Ifcfg-peth0文件的创建修改时间,与同事误操作的时间吻合,再一次怀疑这个文件跟这次故障有一定的关系:
# find . -type f -mtime 2|xargs ls -l -rw-r--r-- 1 root root 303 Mar 21 09:46 ./etc/modprobe.conf -rw-r--r-- 1 root root 23116 Mar 21 09:46 ./etc/sysconfig/hwconf -rw-r--r-- 1 root root 122 Mar 21 09:46 ./etc/sysconfig/network-scripts/ifcfg-peth0
3.8为方便排查和重现故障,根据系统的环境,在测试环境搭建:安装了XEN虚拟化RHEL5.6。
3.8.1跟生产系统一样,同样的备份一份Ifcfg-eth0.bak文件;
3.8.2根据同事误操作的历史记录,逐条进行执行测试,当测试到“kudzu 0:off 1:off 2:off 3:on 4:on 5:on 6:off”,问题重现:ifcfg-eth0文件丢失,同时生成了ifcfg-peth0文件,并且断网。与生产系统故障的情况一致。如图:
3.9搭建另一个测试环境:并未安装XEN虚拟化的RHEL5.6。同样的执行3.8.2章节的命令,但问题未重现。如图:
4、故障原因
通过问题重现,得出结论:安装了XEN虚拟化环境的系统,同事误操作的时候执行了其中一条“kudzu 0:off 1:off 2:off 3:on 4:on 5:on 6:off”命令,两者条件满足情况下,从而导致删除了ifcfg-eth0文件,继而发生断网。
5、相关知识
根据网上信息了解,kudzu命令为什么会导致删除网卡配置文件,目前所了解的,应该是在特定情况下(安装了XEN虚拟化)触发的BUG或者本身的机制导致。
附:
1、kudzu介绍:http://blog.csdn.net/huyangg/article/details/7189743
2、kudzu相关BUG:https://bugzilla.redhat.com/show_bug.cgi?id=206910、https://bugzilla.redhat.com/show_bug.cgi?id=229579、http://linux.bigresource.com/Red-Hat-Prevent-kudzu-from-changing-ifcfg-ethX-file--wi29JYmpf.html
本文出自 “[[email protected]卢伟开~]#rm -rf /” 博客,转载请与作者联系!
以上是关于一次误操作引起的linux系统网络故障的主要内容,如果未能解决你的问题,请参考以下文章