OVM – Remove stale cluster enties
Intro
It has been ages since when Oracle released its own hypervisor (OVM). OVM technology is based on paravirtualization and uses Xen-based
hypervisor. OVM’s latest release version 3.4.6.3 is the latest one
available. Oracle announces extended support for OVM and the support period
is March 2021 and will end on March 31, 2024.
hypervisor. OVM’s latest release version 3.4.6.3 is the latest one
available. Oracle announces extended support for OVM and the support period
is March 2021 and will end on March 31, 2024.
If you need more information about OVM support read the below mentioned article:
https://blogs.oracle.com/virtualization/post/announcing-oracle-vm-3-extended-support/
This is going to be the end of the OVM tree, after this there will be no
release for OVM. The latest technology going to Oracle KVM. Oracle KVM
is much more stable than the OVM and gives you more flexibility in the
virtualization environment. If you are still planning on staying on on-prem. I
would say this is the right time to plan your journey to KVM.
release for OVM. The latest technology going to Oracle KVM. Oracle KVM
is much more stable than the OVM and gives you more flexibility in the
virtualization environment. If you are still planning on staying on on-prem. I
would say this is the right time to plan your journey to KVM.
In this article, I will cover the issue we faced recently in the OVM
environment.
Overview of the issue
We faced a new issue with the OVM cluster environment. This was caused due
to sudden data center power outage. Once everything was online we were not
able to start the OVM hypervisor. so we had to perform a complete
reinstallation of the node.
to sudden data center power outage. Once everything was online we were not
able to start the OVM hypervisor. so we had to perform a complete
reinstallation of the node.
When I try to add node backup to the cluster we faced an issue with mounting
the repositories. The next option was to remove the nodes from the cluster
again, This action was performed via GUI.
the repositories. The next option was to remove the nodes from the cluster
again, This action was performed via GUI.
I did validation and realized that cluster entries are still there in
node02. we found that there were some stale entries in the master node and
node02.
node02. we found that there were some stale entries in the master node and
node02.
Please find the Oracle meta link node that covers the issue for stale
entries :
entries :
OVM – How To Remove Stale Entry of the Oracle VM Server which was Removed
from The Pool (Doc ID 2418834.1)
How to identify there are stale entire?
First, validate the o2cb status. This is the cluster service which consists
of all the information about the cluster. I have highlighted node information in red color.
of all the information about the cluster. I have highlighted node information in red color.
[root@ovm-node02 ~]# service o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Stack glue driver: Loaded
Stack plugin "o2cb": Loaded
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster "f6f6b47b38e288e0": Online
Heartbeat dead threshold: 61
Network idle timeout: 60000
Network keepalive delay: 2000
Network reconnect delay: 2000
Heartbeat mode: Global
Checking O2CB heartbeat: Active
0004FB0000050000B705B4397850AAD6 /dev/dm-2
Nodes in O2CB cluster: 0 1
Debug file system at /sys/kernel/debug: mounted
Now let’s check entries from node02, if this is correctly removed from the
cluster you should see only one entry. But here there are two entries.
cluster you should see only one entry. But here there are two entries.
[root@ovm-node02 ovm-node02]# ls -lrth /sys/kernel/config/cluster/f6f6b47b38e288e0/node/
total 0
drwxr-xr-x 2 root root 0 Jun 23 09:28 ovm-node02
drwxr-xr-x 2 root root 0 Jun 23 09:33 ovm-node01
[root@ovm-node02 ovm-node02]#
The next step is to validate from the master node (ovm-node01)
database entries. This shows there are two pool_member_ip_list.
[root@ovm-node01]# ovs-agent-db dump_db server
{'cluster_state': 'DLM_Ready',
'clustered': True,
'fs_stat_uuid_list': ['0004fb000005000015c1fb14ef761f40',
'0004fb000005000079ae03177c3edc7e',
'0004fb000005000065985109f8834e8b'],
'is_master': True,
'manager_event_url': 'https://192.168.85.152:7002/ovm/core/wsapi/rest/internal/Server/08:00:20:ff:ff:ff:ff:ff:ff:ff:00:10:e0:ef:de:6a/Event',
'manager_ip': '192.168.85.152',
'manager_statistic_url': 'https://192.168.85.152:7002/ovm/core/wsapi/rest/internal/Server/08:00:20:ff:ff:ff:ff:ff:ff:ff:00:10:e0:ef:de:6a/Statistic',
'manager_uuid': '0004fb0000010000c8ecbd219dc6b1ee',
'node_number': 0,
'pool_alias': 'EclipsysOVM',
'pool_master_ip': '192.168.85.177',
'pool_member_ip_list': ['192.168.85.177', '192.168.85.178'],
'pool_uuid': '0004fb0000020000f6f6b47b38e288e0',
'poolfs_nfsbase_uuid': '',
'poolfs_target': '/dev/mapper/36861a6fddaa0481ec0dd3584514a8d62',
'poolfs_type': 'lun',
'poolfs_uuid': '0004fb0000050000b705b4397850aad6',
'registered_hostname': 'ovm-node01',
'registered_ip': '192.168.85.177',
'roles': set(['utility', 'xen'])}
[root@calavsovm01 ovm-node01]#
Remove node from cluster commands line
Now we can remove the oven-node02 from the second node.
[root@ovm-node01]# o2cb remove-node f6f6b47b38e288e0 ovm-node02
Validate node entries
After removing node02, we can see only one entry in the OVM database.
[root@ovm-node01]# ls /sys/kernel/config/cluster/f6f6b47b38e288e0/node/
ovm-node02
[root@ovm-node01]#
Validate using O2CB
First, restart the ovs-agent on both nodes and validate the o2cb cluster status from node01.
[root@ovm-node01]# service ovs-agent restart
Stopping Oracle VM Agent: [ OK ]
Starting Oracle VM Agent: [ OK ]
[root@ovm-node01 ~]# service ovs-agent status
log server (pid 32442) is running...
notificationserver server (pid 32458) is running...
remaster server (pid 32464) is running...
monitor server (pid 32466) is running...
ha server (pid 32468) is running...
stats server (pid 32470) is running...
xmlrpc server (pid 32474) is running...
fsstats server (pid 32476) is running...
apparentsize server (pid 32477) is running...
[root@ovm-node01 ~]#
Also I would recommend to restart the node02 after the node removal, Once the node is back online validate the /etc/ocfs2/cluster.conf
[root@ovm-node01 ~]# cat /etc/ocfs2/cluster.conf
cluster:
heartbeat_mode = global
node_count = 1
name = f6f6b47b38e288e0
node:
number = 0
cluster = f6f6b47b38e288e0
ip_port = 7777
ip_address = 10.110.110.101
name = ovm-node01
heartbeat:
cluster = f6f6b47b38e288e0
region = 0004FB0000050000B705B4397850AAD6
Note: ovs-agent restart won’t have any impact on running VMs.
[root@ovm-node01]# ovs-agent-db dump_db server
{'cluster_state': 'DLM_Ready',
'clustered': True,
'fs_stat_uuid_list': ['0004fb000005000015c1fb14ef761f40',
'0004fb000005000079ae03177c3edc7e',
'0004fb000005000065985109f8834e8b'],
'is_master': True,
'manager_event_url': 'https://192.168.85.152:7002/ovm/core/wsapi/rest/internal/Server/08:00:20:ff:ff:ff:ff:ff:ff:ff:00:10:e0:ef:de:6a/Event',
'manager_ip': '192.168.85.152',
'manager_statistic_url': 'https://192.168.85.152:7002/ovm/core/wsapi/rest/internal/Server/08:00:20:ff:ff:ff:ff:ff:ff:ff:00:10:e0:ef:de:6a/Statistic',
'manager_uuid': '0004fb0000010000c8ecbd219dc6b1ee',
'node_number': 0,
'pool_alias': 'EclipsysOVM',
'pool_master_ip': '192.168.85.177',
'pool_member_ip_list': ['192.168.85.177'],
'pool_uuid': '0004fb0000020000f6f6b47b38e288e0',
'poolfs_nfsbase_uuid': '',
'poolfs_target': '/dev/mapper/36861a6fddaa0481ec0dd3584514a8d62',
'poolfs_type': 'lun',
'poolfs_uuid': '0004fb0000050000b705b4397850aad6',
'registered_hostname': 'ovm-node01',
'registered_ip': '192.168.85.177',
'roles': set(['utility', 'xen'])}
[root@ovm-node01]#
Conclusion
There can be situations gui will not remove the entries from the OVM hypervisor. Always validate the OVM data entries before retying the node addition to the cluster. Make sure the cluster-shared repositories are mounting automatically.