Now that you have successfully installed and configured your backup domain, you are ready to sit back and take it easy. But wait, someone knocking on your door wants to know the status of their backup or restore. I guess you will have to start monitoring the backup and restore processes. While we are at it, we might as well look at monitoring some of the other elements in the backup domain that might need our attention. In this article, we go through some of the tools available to monitor the activities of our example backup and restore application, VERITAS NetBackup. The most common tool for monitoring the backup and restore activity is the Activity Monitor. This is available on both the UNIX Java GUI and the Windows GUI. This screen lists all the jobs that are currently being tracked by NetBackup, including jobs that are queued, active, and finished. You can also see the summary information that includes all the jobs in each of these categories, plus the sum total of all jobs and the master server name. From the Activity Monitor, you can cancel individual jobs or cancel all running jobs. This is done from the Actions menu at the top of the screen.

There are additional fields that give information about each job, including the actual policy name and schedule for backups, start time, elapsed time, media server name, client name, end time, number of this attempt, and others. You can also highlight a job and select a detailed view that provides more information about the selected job. You can get to the Job Details screen by selecting a job from the Activity Monitor and double-clicking on it, by highlighting a job, right-clicking, and selecting Details from the menu, or by highlighting a job and then clicking on Actions at the top of the window. Doing so results in a pull-down menu with Details as a selection. The Jobs Details screen contains two tabs: Job Overview and Detailed Status. These screens can give you a very good idea of what is going on with a specific job. Also, if a job has a nonzero status, you can select the Troubleshooter button. This wizard provides an explanation of the specific status code and can be very helpful in discovering what might have caused a job to end with a nonzero status code.

The bperror command is a very useful tool that can be used in a variety of scripts for reporting on backup status, throughput statistics, and so on. You will see its value as we look into the reporting later in this article. For reference, the bperror command may be found in the following directories per your installed system for NT/2000 and UNIX, respectively. The Activity Monitor window also has two other tabs on the UNIX Java monitor and three other tabs on the Windows Activity Monitor. There is a Daemons tab on the Java GUI, a Services tab on Windows, a Processes tab on both system types, and a Drives tab on Windows. The Daemons/ Services tab allows you to see all the NetBackup and Media Manager daemons or services that are running on the system. It also gives you the ability to stop and start them. The Processes tab on both GUIs displays all the NetBackup and Media Manager related processes that are running. The Windows GUI has a third tab for drives that gives another view of the physical drives. The Windows GUI also has a topology view that is currently not available as part of the Activity Monitor with the Java GUI. At times, you may want to know the status of the backups or other jobs and you don't have access to a graphic terminal or are working remotely. You can still get information and monitor both the jobs activity and the device status. You can monitor the jobs by using the command-line interface (CLI) /usr/openv/netbackup/bin/admincmd/bpdbjobs on a UNIX system or C:\Program Files\VERITAS\NetBackup\bin\admincmd\bpdbjobs.exe on a Windows system. This command allows you to see what is going on in the job queue. You can either issue the CLI without options, or you can add options to further refine the output.

You can also find out what daemons are running on your UNIX master by issuing this command: /usr/openv/netbackup/bin/bpps -a. This returns a list of all the NetBackup and Media Manager daemons and processes that are running on the system. As you start to use your backup application, you will want to create reports that help you know how things are going and that better enable you to update the status of the backup domain to others. Reports are an important part of any backup application. NetBackup has reports that are available from the GUI. Most of these are actually created by executing a CLI with specific options. Many of these commands are documented, so you can use them to create custom reports if you do not find exactly what you want. If you are using a UNIX master and want to see the reports without using the CLI or GUI, you can use the character-based menu interface bpadm. This also gives you access to all the reports. This interface may be a bit difficult at first to navigate, especially if you are used to using only a GUI, but after a few times, it should become quite easy and much faster for remote management requirements.

The two interfaces are very similar in that they both allow you to run the exact same reports, but the character-based interface presents their selections a bit differently from the GUI. While this interface may not be your first choice, it is an excellent tool to help you understand the NetBackup architecture. The reports in the preceding list are purely NetBackup reports, relying completely on the NetBackup logs, /usr/openv/netbackup/db/error on UNIX or c:\Program Files\Veritas\NetBackup\db\error on Windows NT/2000. There are two general types of reports: backup reports and media reports. Backup reports report on the jobs and errors. Media reports give you the ability to look at the catalog information from the perspective of the media and from the perspective of the image. To provide you with a greater understanding of what NetBackup and Media Manager are doing, we have included the command-line equivalents for each of the reports below and a brief description of what processes are used and databases that are touched.

While the report may not be the most attractive from the command line, or for that matter, the GUI, it is rich with information about your images written to the media managed by your NetBackup servers. As one of our clients found out, this report helped to track down a questionable tape drive in their environment that was causing extended restore times because of this hardware failure.DWO stands for Drive Written On; this column indicates the drive index number of the physical drive used to write the image fragment you are looking at in this report on the host listed in the report. So the DWO was 0 for the first image and the host that wrote it was 'susoe09.' This particular client found that one of the drives had a calibration problem when writing its tracks to the tape. This problem would not manifest itself during backup, but rather during the restore when the original DWO was not used for this read process. When mounted in a properly calibrated drive, the tracks written by this original DWO were just far enough off to cause soft errors that are recoverable through the drive error correction algorithms; however, when you are error correcting the entire length of the tape, it adds considerable overhead and extends the normal duration of a restore exponentially. The information from this report was very helpful in identifying the root cause.

As we have seen, there are reports for jobs, media, and images. These reports can provide you with most, if not all, of the information you need to manage your backup domain. Unfortunately, there are other reporting needs. Sometimes it is nice to be able to produce graphical reports and to have reports on other parts of the overall backup domain. These advanced reports may be produced in various ways. VERITAS Software offers an option product, NetBackup Advanced Reporter (NBAR), with over 30 different reports. You can also write your own custom Perl scripts that can interface with the program interface in NBAR to produce your own reports. This is a Web-based tool. Many of these reports can be tailored to provide exactly the information you need. NBAR also includes an online version of the NetBackup Troubleshooting Guide. NBAR gives you a tool that allows you to use the information provided in the NetBackup debug logs without having to manually analyze the text log files. NBAR passively scans the VERITAS NetBackup logs for specific information about each client and copies that information to its own database. It then uses that data to generate reports on NetBackup backup and restore activities, catalog operations, and media usage. Maintaining a separate database allows Advanced Reporter to retain a record of NetBackup activities long after the NetBackup administrator has purged expired backup information from the NetBackup logs.

NBAR is installed on a NetBackup Data Center master server. The NBAR database server resides on this master server, as do the configuration utility and the client delivery component. NBAR gathers information about the NetBackup environment by running NetBackup commands. NBAR then loads the information gathered into its database. NBAR can be installed and configured on Sun, HP, and Windows master servers. It is also fully integrated with the NetBackup Global Data Manager (GDM), which we discuss later. By using NBAR with GDM, you can actually roll up reports from multiple masters to a single global master.

Several tools are available that allow you to monitor your backup domain. By using the different reports, you should be able to get a good idea of the status of the status of your backup domain, including the jobs and devices.

Now we will look at some of the other backup features and options that you might be interested in using. These can give you the capabilities to handle the backup and recovery requirements that go beyond the standard filesystems.