Avamar System State Backup

NOTE: This applies to Avamar 5.0 with no service pack installed. Windows 2008 system state backups have changed in Avamar 5.0 Service Pack 2.

Avamar System State Backups:

There appears to be some confusion related to Windows systems state backups with Avamar. Most new comers to Avamar are used to the backup software handling the system state backup natively, that is without the need for scripting or storing the backups locally on the client. Avamar handles the system state backups differently as compared to other backup applications. For Windows 2000 and Windows 2003 clients, Avamar utilizes the NTBackup utility to create a system state backup which is stored locally on the client. Starting with Windows 2008, Avamar is capable of making a backup of the system state using the VSS plugin and storing it on the Avamar node(s) itself. However, this method is no longer supported by EMC, and EMC recommends to script the backup using the Windows Backup utility and direct it to a shared folder.

Following are the systems state backup requirements, observations and best practices for each Windows OS.

Windows 2000 and Windows 2003:

Windows 2000 is not officially supported by Avamar 5.0+, so the documentation may not cover it. However, it's system state backup procedure is similar to Windows 2003. System state backups on Windows 2000 are around 300 - 500 MB in size, while in Windows 2003, they can range from 800 MB to 3 GB. If there is enough space in the local C: drive, then use the option in the dataset to create a system state backup. When a client backup is initiated, the Avamar agent calls the NTBackup program and creates a system state backup locally. The files is called systemstate.bkf. On Windows 2003, this procedure uses VSS which is known to cause issues with SQL in certain situations, however in my experience this process has been mostly problem free.

Inside the Windows File System dataset options there is an additional setting called "Backup the System Profile" and this is set to disabled by default. I believe this is what is causing the most confusion. This option only works with Windows 2003 servers and requires that the Avamar Backup System State agent (refered to as AvamarBackupSystemState-windows-x86-5.0.100-409.msi) be installed on the client. Installing this, and enabling the system profile option creates a recovery profile for the client. This profile contains all the configuration information that would be necessary to perform a bare metal restore of the server. The profile is called a HomeBase profile and is often referred to as HomeBase Lite since it only works with Windows 2003 and only offers limited capabilities, such as it will not install the OS or the service pack as is common with other bare metal restore applications. This option is not necessary to restore the client to similar hardware.

If a complete restore of the system is necessary, then install the operating system and the service pack, restore the system state backup file to another client and follow the procedure outlined in the documentation to restore the client. During my testing I only had the Windows system state and the server data, so I had to follow the directions from the version 4.1 Administrator Guide. After following the instructions, and a couple of reboots later, the system came back up.

If the client does not enough space to create a system state backup locally on the C drive, then the backup can be redirected to another local drive, or be scripted to send it to a remote share. To send the system state to a different local drive, include this parameter in the dataset definition that is applied to the client: systemstatefile=d:\avamarfolder. The redirect location can be any directory on the drive. However, if you are doing this, then it is a good idea to document where the system state file resides so it can be located when it is needed.

If the backup needs to be scripted, then the Avamar dataset can be instructed to run a script before the backup begins. This option is found under advanced dataset options and is called pre-script. The script needs to be placed inside c:\program files\avs\etc\scripts and can only be a .bat, .vbs or .js file. Be sure to uncheck the option underneath which says "Abort backup if script fails". If the script fails to run, there is no sense in not backing up the data. When manually creating a system state backup, the system state and either the c:\windows\windows32 (for Windows 2003) or the c:\winnt\system32 (for Windows 2000) need to specified. The script I use is as follows:

Windows 2000 Script:

@echo off
ntbackup backup "c:\WINDOWS\system32" /m normal /f "\\servername\share\%COMPUTERNAME%.bkf"
ntbackup backup systemstate /M normal /f "\\servername\share\%COMPUTERNAME%.bkf" /a

Windows 2003 Script:

@echo off
ntbackup backup "c:\WINDOWS\system32" /m normal /f "\\servername\share\%COMPUTERNAME%.bkf"
ntbackup backup systemstate /M normal /f "\\servername\share\%COMPUTERNAME%.bkf" /a

These scripts require that a share already exist, and it should have share permissions set to allow Everyone to write. This is because Avamar agent runs as System user and this is the only option I have found to allow the System user to write to remote share. I haven't tried running the Avamar client agent as a specific user, so I don't know if that will allow specific share level permissions. I know this is a security hole, but I don't have a work around for this.

One important thing to do when redirecting system state backups to a remote share is to create a avtar.cmd file in c:\program files\avs\var and put the following parameter in it: --backupsystem=false. This is important because if by mistake a different dataset is selected for a client and it does not specify that the system state should be directed to a remote share, the system state backup will be created locally. This parameter blocks the system state backup to made using the Avamar agent. I do this because I only enable redirecting system state backups to a remote share for clients which have very small amount of free space available locally, and if by accident the drives fill up, the client could crash.

When the client backup runs, it calls the system state backup script which creates the backup to a remote share. This remote share can then be backed up by installing the Avamar agent on that client and backing up that client last.

One thing to note about the Windows dataset is that it includes a SystemState/ option under Source Data. This makes some people think that the system state will be backed up for Windows 2000 and 2003 servers. This is not true, and this option is there to backup the system state for Windows 2008 servers. Although, even then this option is not used.

Windows 2008:

Windows 2008 system state backups cannot be made with Avamar natively. EMC has a document on Powerlink called "Windows Server 2008 Offline System Recovery Using Windows Server Backup with Avamar" which describes how to configure system state backups for Windows 2008. Do not use the VSS Plugin to backup the system state, even though it appears to be the obvious choice, or the client logs might indicate so if you enabled the "Backup the System State" option. If you enable it, you will see that a successful system state backup was made, but this backup cannot be properly restored. If you try to restore from the VSS plugin system state backup, it will appear to restore data, but it will never complete, it will just get stuck at 99%. EMC says that this is a Microsoft issue due to recent changes they have made to Windows 2008. Thus, the process outlined in the document mentioned earler has to be followed in order to make a successful restorable system state backup.

To enable the system state backup, the Windows Backup utility has to be installed on Windows 2008. This utility is not installed by default, so go to Manage, Features and enable it. Once it is installed, system state backups can be made. I use a script to start the backup and redirect it to a remote share. The script is:

@echo off
wbadmin start backup -backupTarget:\\remoteserver\share$ -allCritical -quiet

Then, go to the dataset being used with this server and specify the script under pre-script option.

Windows 2008 system state backups take much longer than Windows 2003 backups and at least 10GB in size because they backup more data. Whatever server the system state backups are going to should be backed up last, otherwise system state backups will be at least a day old.

The script provided in the documentation has a line to delete the system state backup after it has been backed up, but I don't do that.