一次11g rac由于gipc进程无法识别网卡状态导致集群无法启动的排查 |
您所在的位置：网站首页 › 苏南的地图 › 一次11g rac由于gipc进程无法识别网卡状态导致集群无法启动的排查 |
|
一次gipc进程网卡状态问题的排查 问题：客户联系我一个集群节点无法启动。连上后发现之前由于心跳网络问题，集群节点2 GRID自动重启，重启时依然报心跳网络异常，因此启动不成功。检查此时的心跳网络已经恢复，手动启动集群软件仍然未解决。最终解决后总结：这个问题的处理，主要需要对11G RAC的架构(几个AGENT管理不同资源)以及集群中不同进程管理的资源以及交互情况有了解，可以知道OCSSD进程认为心跳网络异常，心跳网络状态是GIPCD进程监控因此需要看GIPCD进程的日志。通过日志发现问题后，再尝试解决。常规解决方法无效后，要继续排查包括往BUG方面排查，这个问题就是低版本集群上的BUG(本次异常的集群是LINUX+11.2.0.3，无PSU)。在11.2.0.4版本集群上至今未遇到过此问题。 相关处理过程如下： 1.查看集群日志及ocssd.log日志 ===可以发现是典型的心跳网络异常导致的集群软件重启(oracle 11g 新特性Rebootless Restart特性默认是重启GRID不重启主机) [crsd(7657)]CRS-2772:Server 'test1' has been assigned to pool 'Generic'. 2020-12-04 10:34:10.572 [crsd(7657)]CRS-2772:Server 'test1' has been assigned to pool 'ora.ipcc'. 2021-01-16 23:34:20.127 [cssd(7161)]CRS-1612:Network communication with node test1 (1) missing for 50% of timeout interval. Removal of this node from cluster in 14.940 seconds 2021-01-16 23:34:28.128 [cssd(7161)]CRS-1611:Network communication with node test1 (1) missing for 75% of timeout interval. Removal of this node from cluster in 6.940 seconds 2021-01-16 23:34:32.129 [cssd(7161)]CRS-1610:Network communication with node test1 (1) missing for 90% of timeout interval. Removal of this node from cluster in 2.940 seconds 2021-01-16 23:34:35.071 [cssd(7161)]CRS-1609:This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00008:) in / opt/11.2.0/grid/log/test2/cssd/ocssd.log. 2021-01-16 23:34:35.071 [cssd(7161)]CRS-1656:The CSS daemon is terminating due to a fatal error;2.检查操作系统 日志 当时确实心中网络异常，但是目前已经恢复。 3.重启集群软件时报无网络心跳 日志中一直报has a disk HB, but no network HB 2021-01-20 18:18:43.082: [ CSSD][2730108672]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0 2021-01-20 18:18:43.236: [ CSSD][2734855936]clssnmvDHBValidateNCopy: node 1, test1, has a disk HB, but no network HB, DHB has rcfg 375467028, wrtcnt, 130988652, LATS 4294435040, lastSeqNo 130988651, uniqueness 1607049396, timestamp 1611137741/4086845544 2021-01-20 18:18:44.082: [ CSSD][2730108672]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0 2021-01-20 18:18:44.236: [ CSSD][2734855936]clssnmvDHBValidateNCopy: node 1, test1, has a disk HB, but no network HB, DHB has rcfg 375467028, wrtcnt, 130988653, LATS 4294436040, lastSeqNo 130988652, uniqueness 1607049396, timestamp 1611137742/4086846554 2021-01-20 18:18:45.082: [ CSSD][2730108672]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0 2021-01-20 18:18:45.236: [ CSSD][2734855936]clssnmvDHBValidateNCopy: node 1, test1, has a disk HB, but no network HB, DHB has rcfg 375467028, wrtcnt, 130988654, LATS 4294437040, lastSeqNo 130988653, uniqueness 1607049396, timestamp 1611137743/4086847554 2021-01-20 18:18:45.238: [ CSSD][2745972480]clssscSelect: cookie accept request 0xc39eb0 4.问题的分析： 心跳网络中断后已经恢复，集群始终认为心跳网络是异常。 因此我们基于11G RAC集群各个组件AGENT及启动流程进行分析梳理，GIPCD进程负责监控心跳网络的可用性。 因此查看GIPCD进程的日志，可以发现GIPCD进程一直认为网卡状态是异常的(rank值为0或－1说明网络异常，rank 99表示正常) 2021-01-20 18:16:52.489: [GIPCDMON][2970949376] gipcdMonitorSaveInfMetrics: inf[ 0] eth1 - rank 0, avgms 30000000000.000000 [ 32 / 0 / 0 ] 2021-01-20 18:16:52.634: [ CLSINET][2970949376] Returning NETDATA: 1 interfaces [grid@ccdb2 gipcd]$ tail -n 500 gipcd.log |grep rank 2021-01-20 18:16:22.480: [GIPCDMON][2970949376] gipcdMonitorSaveInfMetrics: inf[ 0] eth1 - rank 0, avgms 30000000000.000000 [ 33 / 0 / 0 ] 2021-01-20 18:16:52.489: [GIPCDMON][2970949376] gipcdMonitorSaveInfMetrics: inf[ 0] eth1 - rank 0, avgms 30000000000.000000 [ 32 / 0 / 0 ] 2021-01-20 18:17:22.498: [GIPCDMON][2970949376] gipcdMonitorSaveInfMetrics: inf[ 0] eth1 - rank 0, avgms 30000000000.000000 [ 32 / 0 / 0 ] 2021-01-20 18:17:52.505: [GIPCDMON][2970949376] gipcdMonitorSaveInfMetrics: inf[ 0] eth1 - rank 0, avgms 30000000000.000000 [ 32 / 0 / 0 ] 2021-01-20 18:18:22.515: [GIPCDMON][2970949376] gipcdMonitorSaveInfMetrics: inf[ 0] eth1 - rank 0, avgms 30000000000.000000 [ 33 / 0 / 0 ] 2021-01-20 18:18:52.524: [GIPCDMON][2970949376] gipcdMonitorSaveInfMetrics: inf[ 0] eth1 - rank 0, avgms 30000000000.000000 [ 32 / 0 / 0 ] 2021-01-20 18:19:22.532: [GIPCDMON][2970949376] gipcdMonitorSaveInfMetrics: inf[ 0] eth1 - rank 0, avgms 30000000000.000000 [ 32 / 0 / 0 ] 2021-01-20 18:19:52.541: [GIPCDMON][2970949376] gipcdMonitorSaveInfMetrics: inf[ 0] eth1 - rank 0, avgms 30000000000.000000 [ 33 / 0 / 0 ] 2021-01-20 18:20:22.541: [GIPCDMON][2970949376] gipcdMonitorSaveInfMetrics: inf[ 0] eth1 - rank 0, avgms 30000000000.000000 [ 32 / 0 / 0 ] 2021-01-20 18:20:52.551: [GIPCDMON][2970949376] gipcdMonitorSaveInfMetrics: inf[ 0] eth1 - rank 0, avgms 30000000000.000000 [ 33 / 0 / 0 ] 5.问题的处理根据11G RAC架构的AGENT资源管理情况，gipcd进程是由ohasd对应的代理进程 所管理的。可以KILL GIPCD进程之后ohasd会启动新的gipcd守护进程。 因此我们尝试了： 1.KILL此进程，之后问题未解决。 2.尝试重启故障节点2的集群软件，问题未解决。 3.重启了节点2主机，问题未解决。 查一下MOS上的文档： 11gR2 GI Node May not Join the Cluster After Private Network is Functional After Eviction due to Private Network Problem (Doc ID 1479380.1) Bug 13653178 - OCSSD from a rebooted node cannot rejoin the cluster (Doc ID 13653178.8) 解决方法就是非业务高峰KILL存活节点的GIPCD进程。本次按照此方法，解决了问题。
|
| CopyRight 2018-2019 办公设备维修网 版权所有 豫ICP备15022753号-3 |