Oracle 19.20未知BUG导致oraagent进程内存泄漏

故障现象

查询操作系统进程的使用排序，这里看到oraagent的物理内存达到16G，远远超过正常环境（正常环境在19.20大概就是100M多一点）

[root@orastd tmp]# ./hmem|more
PID      NAME                     VIRT(kB)   SHARED(kB)      RSS(kB)  PRIVATE(kB)
2102     oraagent.bin             18212328        47160     16215456     16215456

数据库版本

获得数据库的版本为19.20

[grid@orastd trace]$ $ORACLE_HOME/OPatch/opatch lspatches35332537;ACFS RELEASE UPDATE 19.20.0.0.0 (35332537)
35320081;Database Release Update : 19.20.0.0.230718 (35320081)
33575402;DBWLM RELEASE UPDATE 19.0.0.0.0 (33575402)

分析过程

获得内存使用详细信息

获得内存的详细信息

[root@orastd tmp]# ./hmem -l|more
PID      NAME                     VIRT(kB)   SHARED(kB)      RSS(kB)  PRIVATE(kB)    RssAnon    RssFile   RssShmem      VmRSS     VmData
2102     oraagent.bin             18212328        47160     16215456     16215456   16168296      47160          0   16215456   17850384

RssAnon标识进程的私有内存，常住在内存中，其实可以理解为DATA部分的内存，这里可以肯定oraagent存在内存泄漏了。

查看oraagent日志

oraagent部分一直不断的在重复显示下面内容，这里面有个奇怪的现象就是ora.LISTENER.lsnr，但是我集群中并没有这个资源

2025-07-24 23:04:11.272 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateEndPoints 2040 Listener ResName:ora.LISTENER.lsnr
2025-07-24 23:04:11.272 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateHostName entry lsnrResName:ora.LISTENER.lsnr
2025-07-24 23:04:11.272 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::getLsnrResNameAddrType &s_lsnrResNametoAddrTypeXLock:0x55555a209718get resname:ora.LISTENER.lsnr addressType
2025-07-24 23:04:11.272 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::getAddressTypeValue 000 entry { actx:(nil) lsnrName:ora.LISTENER.lsnr
2025-07-24 23:04:11.277 : USRTHRD:3858700032: [     INFO] {0:0:2} Agent::getValueFromNetwork 000 { get netParam:ADDRESS_TYPE from vipResName: defaultNetRes: tag:
2025-07-24 23:04:11.277 : USRTHRD:3858700032: [     INFO] {0:0:2} Agent::getValueFromNetwork 999 } netParam:ADDRESS_TYPE vipResName: defaultNetRes: networkResName: netParamValue:IPV4
2025-07-24 23:04:11.277 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::getAddressTypeValue 220 lsnrResNametoAddrType lsnrNetworkMap lsnrName:ora.LISTENER.lsnr networkResName:
2025-07-24 23:04:11.277 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::getLsnrResNameAddrType &s_lsnrResNametoAddrTypeXLock:0x55555a209718lsnrNetworkMap[ora.LISTENER.lsnr]: networkAddressTypeMap[]:IPV4
2025-07-24 23:04:11.277 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateHostName getNodeName 1
2025-07-24 23:04:11.278 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateHostName exit hostIP:192.168.1.223  hostIPV6:
2025-07-24 23:04:11.281 : USRTHRD:3858700032: [     INFO] {0:0:2} checkCrsRet clscrsret: 5
2025-07-24 23:04:11.281 : USRTHRD:3858700032: [     INFO] {0:0:2} CrsCmd::ClscrsCmdData:statGetAttr 040 error getting attribute:ENDPOINTS@SERVERNAME(orastd)
2025-07-24 23:04:11.281 : USRTHRD:3858700032: [     INFO] {0:0:2} CrsCmd::statGetAttr 120 error
2025-07-24 23:04:11.284 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateEndPoints 2311 lsnrResName:ora.LISTENER.lsnr endpAttr:TCP:1521
2025-07-24 23:04:11.284 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::ifFirewallParam entry
2025-07-24 23:04:11.284 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::ifFirewallParam exit
2025-07-24 23:04:11.284 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateEndPoints 2511 type:2 vipVector endpString:(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.223)(PORT=1521))
2025-07-24 23:04:11.284 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::removeDuplicates
2025-07-24 23:04:11.284 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateEndPoints 1999 exit }

因为是备库环境，监听用的1522端口，所以没有默认的ora.LISTENER.lsnr，难道跟这个有关系？

监控oraagent内存的变换

可以看到每个10分钟，大概新增加如下的物理内存的使用量。

[root@orastd tmp]# sh ./count_diff_rss.sh  /tmp/ps_2102.log
2025-07-24 14:13:14: 164
2025-07-24 14:23:15: 96
2025-07-24 14:33:15: 272
2025-07-24 14:43:15: 164
2025-07-24 14:53:15: 184
2025-07-24 15:03:15: 72
2025-07-24 15:13:15: 180
2025-07-24 15:23:15: 140
2025-07-24 15:33:15: 192
2025-07-24 15:43:15: 156
2025-07-24 15:53:16: 184
2025-07-24 16:03:16: 196
2025-07-24 16:13:16: 164
2025-07-24 16:23:16: 148
2025-07-24 16:33:16: 228
2025-07-24 16:43:16: 164
2025-07-24 16:53:16: 196
2025-07-24 17:03:16: 92
2025-07-24 17:13:16: 264
2025-07-24 17:23:16: 176
2025-07-24 17:33:16: 180
2025-07-24 17:43:16: 160
2025-07-24 17:53:16: 200
2025-07-24 18:03:16: 164
2025-07-24 18:13:16: 76
2025-07-24 18:23:17: 292
2025-07-24 18:33:17: 48
2025-07-24 18:43:17: 184
2025-07-24 18:53:17: 168
2025-07-24 19:03:17: 128
2025-07-24 19:13:17: 196
2025-07-24 19:23:17: 192
2025-07-24 19:33:17: 144
2025-07-24 19:43:17: 220
2025-07-24 19:53:17: 164
2025-07-24 20:03:18: 200
2025-07-24 20:13:18: 112
2025-07-24 20:23:18: 232
2025-07-24 20:33:18: 192
2025-07-24 20:43:18: 184
2025-07-24 20:53:18: 160
2025-07-24 21:03:18: 192
2025-07-24 21:13:18: 164
2025-07-24 21:23:18: 84

解决方案

从上面的日志中可以怀疑oraagent在不断判断对listener的资源时可能异常，导致申请的内存一直不释放，并且不断重复上述过程，所以使得内存不断增加。

新建监听

新建监听，并启动监听

[grid@orastd ~]$ srvctl add listener
[grid@orastd ~]$ srvctl start listener -l listener

观察内存使用量

新建监听后，内存的使用未有明显变化，所以这里大概就猜测是的原因就是oraagent内部判断监听资源时异常，导致内存不释放

手动释放oraagent的内存资源

oraagent进程负责grid用户下面资源的启停，是一个可以手动kill的进程，所以这里采用手动的方式。

kill oragent进程

[grid@orastd ~]$ ps -ef|grep oraagent
grid      2102     1 18  2024 ?        102-00:56:17 /oracle/app/19.3.0/grid/bin/oraagent.bin
grid     41122 40551  0 23:05 pts/0    00:00:00 tail -1000f ohasd_oraagent_grid.trc
grid     41317 40748  0 23:06 pts/1    00:00:00 grep --color=auto oraagent
[grid@orastd ~]$ kill -9 2102

查看启动日志信息

ohasd进程负责oraagent进程的启动，有如下的信息：

2025-07-24 23:06:38.834 : CRSCOMM:3643938560: [     INFO]  IpcL: connection to member 4 has been removed
2025-07-24 23:06:38.834 :CLSFRAME:3643938560: [     INFO]  Removing IPC Member:{Relative|Node:0|Process:4|Type:3}
2025-07-24 23:06:38.834 :CLSFRAME:3643938560: [     INFO]  Disconnected from AGENT process: {Relative|Node:0|Process:4|Type:3}
2025-07-24 23:06:38.835 :    AGFW:3637634816: [     INFO] {0:0:23496} Agfw Proxy Server received process disconnected notification, count=1
2025-07-24 23:06:38.835 :    AGFW:3637634816: [     INFO] {0:0:23496} /oracle/app/19.3.0/grid/bin/oraagent_grid disconnected.
2025-07-24 23:06:38.835 :    AGFW:3637634816: [     INFO] {0:0:23496} Agent /oracle/app/19.3.0/grid/bin/oraagent_grid[2102] stopped!
2025-07-24 23:06:38.835 : CRSCOMM:3637634816: [     INFO] {0:0:23496} IpcL: removeConnection: Member 4 does not exist in pending or formed connections.
2025-07-24 23:06:38.835 : CRSCOMM:3637634816: [    ERROR] [FFAIL]{0:0:23496} IpcL: Failed to disconnect member 4 rc = 4
2025-07-24 23:06:38.835 :   CRSPE:1879045888: [     INFO] {0:0:23497} Disconnected from server:
2025-07-24 23:06:38.835 :    AGFW:3637634816: [     INFO] {0:0:23496} Restarting the agent /oracle/app/19.3.0/grid/bin/oraagent_grid
2025-07-24 23:06:38.835 :    AGFW:3637634816: [     INFO] {0:0:23496} Starting the agent: /oracle/app/19.3.0/grid/bin/oraagent with user id: grid and incarnation:3
2025-07-24 23:06:38.844 :    AGFW:3637634816: [     INFO] {0:0:23496} Starting the HB [Interval =  30000, misscount = 6kill allowed=1] for agent: /oracle/app/19.3.0/grid/bin/oraagent_grid
2025-07-24 23:06:38.930 : CRSCOMM:3643938560: [     INFO]  IpcL: Accepted connection 16497 from user grid member number 7

oraagent进程自己的日志：

2025-07-24 23:06:30.319 :    AGFW:3846092544: [     INFO] {0:0:2} Agent received the message: AGENT_HB[Engine] ID 12293:687689824
Trace file /oracle/app/grid/diag/crs/orastd/crs/trace/ohasd_oraagent_grid.trc
Oracle Database 19c Clusterware Release 19.0.0.0.0 - Production
Version 19.20.0.0.0 Copyright 1996, 2023 Oracle. All rights reserved.CLSB:4160626432: [     INFO] Argument count (argc) for this daemon is 1CLSB:4160626432: [     INFO] Argument 0 is: /oracle/app/19.3.0/grid/bin/oraagent.bin
2025-07-24 23:06:38.927 : default:4160626432: clsu_load_ENV_levels: Failure to retrieve ORA_DAEMON_LOGGING_LEVELS environment variable [-1] [21104].
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: clsdadr  2
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: clsdstat  2
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: clsdnreg  0
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: clsdynam  0
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: allcomp  0
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: default  0
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: clsdimt  0
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: CLSCAL  0

crs的alet日志如下：

2024-01-06 20:37:59.966 [OCSSD(2281)]CRS-1720: Cluster Synchronization Services daemon (CSSD) is ready for operation.
2025-07-24 23:06:38.917 [ORAAGENT(41326)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 41326

总结

本案例展示了在Oracle 19.20环境下，oraagent进程出现内存泄漏的实际处理过程。通过对系统进程、内存使用、日志信息的细致排查，发现oraagent不断尝试处理一个并不存在的监听资源（ora.LISTENER.lsnr），导致内存持续增长，最终远超正常值。虽然尝试新建监听未能缓解问题，但通过手动重启oraagent进程，内存占用恢复正常，间接验证了进程本身存在异常分配和释放内存的问题。

提示：

即使在较新版本的Oracle 19C中，依然可能遇到未被官方收录的BUG，遇到异常现象时要善于通过日志和系统工具定位问题根源。
oraagent进程的内存异常增长，往往与资源配置或内部异常循环有关，及时关注日志中的异常提示（如不存在的监听资源）有助于快速定位问题。
对于oraagent等可安全重启的进程，适当的重启操作可以临时缓解内存泄漏带来的影响，但建议后续持续关注官方补丁和MOS文档，及时获取修复方案。
日常巡检中，建议定期监控关键进程的内存使用情况，发现异常及时处理，避免影响数据库集群的稳定性。
最重要的就是环境搭建的规范化，这次的内存泄漏，明显是由于在部署备库环境时，未完全按照Oracle环境的最佳实践来部署，最佳实践是一个永无休止的根据大量真实案例来不断的更新。
建议将数据库版本升级到最新的补丁包，不要想到当前没有出现故障，就不升级。当前没有出现故障，并不代表着环境中不存在隐患，甚至有可能是已经出现了隐患，但是没有被发现，就如这个案例一样，如果不是朋友的细心，估计备库环境中oraagent泄漏的问题不会提前发现，等生产系统容灾切换到备库后，到时出现内存不足，影响数据库稳定性和性能，此时估计就是重大故障了。最新的补丁包其实都是在修复BUG，并没有引入新的功能，修复的BUG越多，也就意味着环境越稳定。