!!! Please be warned that all information in this article is based on my (and my colleagues) experiences. You can proceed with all these steps on YOUR responsibility. In any case you should contact Oracle official support to help you with this task. Some steps described here may destroy your environment and system data. !!!
Task: change server pool storage. If storage for the server pool is implemented on SAN storage this task cannot be proceeded online. There is no procedure I could find (even with oracle support) to replace storage for the server pool with running virtual systems.
Replacing storage for a “Server Pool” can be extremely important in situation you are planning replace storage system or plan to do some maintenance in SAN infrastructure that will exclude storage prepared for pool storage.
Storage is configured during “create a server pool” wizard by adding storage location.
Having this special storage is a requirement to build a “Server Pool”. And this is a place where the “Server Pool” keeps all important needed to maintain.
Several problems identified to complete the task:
- There are active virtual systems running in this server pool
- Some of the vms has disks spanned between more than one repository
- Lots of constraints in OVM therefore for that reason operation cannot be proceeded with running virtual systems
- High risk of operation, especially if environment is big and there is no time for backup-recovery in requested time window
- Only one operation can be proceeding at the same time. This is due to OVM Server limitation. You can start with several operations but most of them will wait for others to finish.
- Some steps may be required to restart hosts. Plan this step in preparation as this could take more than 10 minutes to restart a single host.
One more time I want to underline that operation cannot be done online, with the “Server Pool” currently running virtual systems lots of dependencies takes place.
Preparation:
- Double check if you have valid backup, define and plan backup before you can start with procedure
Copy and backup all virtual system and templates configuration file; you can use the following:
# mkdir /home/REPO
# cp –parent /OVS/Repositories/*/VirtualMachines/*/mv.cfg /home/backup/
cp –parent /OVS/Repositories/*/Templates/*/mv.cfg /home/TEMPLATE/
- If possible, do offline snapshot of oracle vm manager; at least do mysql backup
- Restart all vms. In case something goes wrong and additionally vm won’t up, you won’t be able to determine what was the reason (long running vm without restart or your work)
- Double check if you know all passwords, especially passwords for all hosts root users and “Oracle VM Agent Password”.
- Make notes (screenshots) of the environment. In particular placement of vms (if this is important from resources or licenses point of view).
Procedure:
- You have to remove all virtual from the “Server Pool”. The easiest way is to move them to the “Unassigned Virtual Machines” folder. This can be done only for VM in powered off state.
It is possible to power off all vms at once by selecting the “Server Pool”, from perspective select virtual machines, select them with shift and stop them. - Similar way you can select all vms and move them using mouse to “Unassigned Virtual Machines”.
- Next step will be to edit all repositories ownership
If you run into a failure step like this:
This means something is residing on the repository and these constraints need to be resolved. Maybe there is ISO attached to some vm or system template. Another possibility, you have vm spanned between more than one repository. In this scenario check next point
- ** VM has disk spanned between the repository**
With such configuration I wasn’t able to release repository ownership. Task stuck on one or more hosts with no chance to succeed. Such vm needed to be deleted from inventory:
Make sure you have collected all vm configuration files as the manager is going to delete them in this step. Additionally, do not select any checkbox to delete.
- Unpresents all repositories from all hosts in the server pool
In case issues still exist you can try to restart a problematic host. There is no guarantee this helps, but in my scenario this helped with few hosts. Please plan this step during preparation as this could take more than 10 minutes to successfully reboot the server.
This point is extremely important. Without this you won’t be able to migrate hosts between the server pools.
Make sure that repositories ownership in the info perspective has changed to “not owned” and there are no servers presented to.
- Migrate all servers to “Unassigned Servers” tab by simply drag and drop them in “servers and VMs” tab.
- If you are stuck with the last server, you will have to proceed with the additional steps.
Login to host via SSH and (backup and) remove the following files:
Do not change anything in the manager. Host after restart should automatically migrate to “Unassigned Servers” without any intervention. This can take some minutes. - Add the host removed in the previous step to inventory by using the “discover servers” function.
Make sure you use the correct password to save a few minutes.
- Create a new “Server pool” and migrate all hosts to it. If some hosts won’t allow migration this means, there are some unresolved constraints. Try to check this, rescan host physical disks or restart host. Check if all previous steps were accomplished successfully.
- For all repositories select the new “Server pool” in the edit window. Click OK. Than edit it again and present repository to all hosts in “Server pool”
- Migrate all vms to the new “Server Pool”. Make sure they are placed on correct hosts.
- Deleted vm can be restored by simply copying their configuration files to the proper path, the same was before. After config files are placed in the correct place they should show up in inventory after a while. If not try to refresh the repository.
Conclusions:
When planning downtime for this operation, make sure you can shut down all virtual machines in the “Server Pool”. Plan your time carefully within a given time window and then double it. Make sure you schedule the hosts restart as well as the time it takes to restart all virtual machines (2x).
This procedure is not extremely complicated, but you may run into problems at every step without a simple solution. At some point, it may not be possible to return to the starting situation. Take the time to plan all steps in detail.
Good luck!
Useful links:
No Comments