Well, what do we have here? Let's say, we created one backup Job in vSphere Data Protection Appliance, and we added 30 virtual machines to it. Next, we started the backup for this job, and you let it run overnight. Next morning when you login, you see that 29 VMs completed successfully and 1 VM failed for backup.
You login to the SSH of the VDP and browse to /usr/local/avamarclient/var and notice that there are 30 logs for the same backup Job with different IDs which makes no sense. You don't know which backup log this failed VM is contained in.
I have created 3 VMs: Backup-Test, Backup-Test-B and Backup-Test-C
I have a backup Job called Job-A
I initiated a manual backup and all 3 VMs completed successfully. If I go to the backup job logs folder, I see the following:
-rw-r----- 1 root root 1.7K Nov 18 04:33 Job-A-1479423789022-02ef0cfc8a738a3e98c44fdbef3354ae1ce86a4b-1016-vmimagel-xmlstats.log
-rw-r----- 1 root root 15K Nov 18 04:34 Job-A-1479423789022-02ef0cfc8a738a3e98c44fdbef3354ae1ce86a4b-1016-vmimagel.alg
-rw-r----- 1 root root 40K Nov 18 04:34 Job-A-1479423789022-02ef0cfc8a738a3e98c44fdbef3354ae1ce86a4b-1016-vmimagel.log
-rw-r----- 1 root root 18K Nov 18 04:33 Job-A-1479423789022-02ef0cfc8a738a3e98c44fdbef3354ae1ce86a4b-1016-vmimagel_avtar.log
-rw-r----- 1 root root 383 Nov 18 04:33 Job-A-1479423789022-3628808a321ddac3a6e0d1eaae3446ad996d9d43-3016-vmimagew-xmlstats.log
-rw-r----- 1 root root 15K Nov 18 04:33 Job-A-1479423789022-3628808a321ddac3a6e0d1eaae3446ad996d9d43-3016-vmimagew.alg
-rw-r----- 1 root root 40K Nov 18 04:33 Job-A-1479423789022-3628808a321ddac3a6e0d1eaae3446ad996d9d43-3016-vmimagew.log
-rw-r----- 1 root root 19K Nov 18 04:33 Job-A-1479423789022-3628808a321ddac3a6e0d1eaae3446ad996d9d43-3016-vmimagew_avtar.log
-rw-r----- 1 root root 1.7K Nov 18 04:33 Job-A-1479423789022-a573f30f993e9a58420c69564cf2e16135540c49-1016-vmimagel-xmlstats.log
-rw-r----- 1 root root 15K Nov 18 04:33 Job-A-1479423789022-a573f30f993e9a58420c69564cf2e16135540c49-1016-vmimagel.alg
-rw-r----- 1 root root 41K Nov 18 04:33 Job-A-1479423789022-a573f30f993e9a58420c69564cf2e16135540c49-1016-vmimagel.log
-rw-r----- 1 root root 18K Nov 18 04:33 Job-A-1479423789022-a573f30f993e9a58420c69564cf2e16135540c49-1016-vmimagel_avtar.log
Looking at this, neither you or I have any idea on which log here contains logging for Backup-Test-C.
Step 1 of the sneaky trick:
Run the following command:
# mccli activity show --verbose=true
The output:
0,23000,CLI command completed successfully.
ID Status Error Code Start Time Elapsed End Time Type Progress Bytes New Bytes Client Domain OS Client Release Sched. Start Time Sched. End Time Elapsed Wait Group Plug-In Retention Policy Retention Schedule Dataset WID Server Container
---------------- ------------------------ ---------- -------------------- ----------- -------------------- ---------------- -------------- --------- ------------- ------------------------------------ ------------- -------------- -------------------- -------------------- ------------ ------------------------------------------ -------------------- ---------------- --------- ------------------------ ------------------------------------------ ------------------- ------ ---------
9147942378903409 Completed w/Exception(s) 10020 2016-11-18 04:33 IST 00h:00m:22s 2016-11-18 04:33 IST On-Demand Backup 64.0 MB <0.05% Backup-Test /vc65.happycow.local/VirtualMachines windows9Guest 7.2.180-118 2016-11-18 04:33 IST 2016-11-19 04:33 IST 00h:00m:11s /vc65.happycow.local/VirtualMachines/Job-A Windows VMware Image Job-A D Admin On-Demand Schedule /vc65.happycow.local/VirtualMachines/Job-A Job-A-1479423789022 Avamar N/A
9147942378902609 Completed 0 2016-11-18 04:33 IST 00h:00m:21s 2016-11-18 04:33 IST On-Demand Backup 64.0 MB <0.05% Backup-Test-C /vc65.happycow.local/VirtualMachines rhel7_64Guest 7.2.180-118 2016-11-18 04:33 IST 2016-11-19 04:33 IST 00h:00m:21s /vc65.happycow.local/VirtualMachines/Job-A Linux VMware Image Job-A DWMY Admin On-Demand Schedule /vc65.happycow.local/VirtualMachines/Job-A Job-A-1479423789022 Avamar N/A
9147942357262909 Completed w/Exception(s) 10020 2016-11-18 04:29 IST 00h:00m:43s 2016-11-18 04:30 IST On-Demand Backup 64.0 MB <0.05% Backup-Test /vc65.happycow.local/VirtualMachines windows9Guest 7.2.180-118 2016-11-18 04:29 IST 2016-11-19 04:29 IST 00h:00m:12s /vc65.happycow.local/VirtualMachines/Job-A Windows VMware Image Job-A DWMY Admin On-Demand Schedule /vc65.happycow.local/VirtualMachines/Job-A Job-A-1479423572598 Avamar N/A
9147942378903009Completed 0 2016-11-18 04:33 IST 00h:00m:22s 2016-11-18 04:34 IST On-Demand Backup 64.0 MB <0.05% Backup-Test-B /vc65.happycow.local/VirtualMachines centosGuest 7.2.180-118 2016-11-18 04:33 IST 2016-11-19 04:33 IST 00h:00m:31s /vc65.happycow.local/VirtualMachines/Job-A Linux VMware Image Job-A DWMY Admin On-Demand Schedule /vc65.happycow.local/VirtualMachines/Job-A Job-A-1479423789022 Avamar N/A
Again, all the confusing? Well, we need to look at two fields here. The Job ID field and the Work Order ID Field.
The Job ID filed is the one highlighted in Red and the Work Order ID is the one highlighted in Orange.
The Work Order ID will match the first Name-ID in the log directory, but still this will not be helpful if there are too many VMs in the same backup Job as they will all have the same Work Order ID.
Step 2 of the sneaky trick:
We will use the Job ID to view the logs. The command would be:
# mccli activity get-log --id=<Job-ID> | less
The first thing you will see as the output is:
0,23000,CLI command completed successfully.
Attribute Value
Followed by tons of blank spaces and dashes.
Step 3 of the sneaky trick
The first Event ID of any backup is performing the logging and this event ID is 5008.
So as soon as you run the get-log command, search for this Event ID and you will be directly taken to the start of the client's backup logs. You will see:
2016-11-18T04:33:29.903-05:-30 avvcbimage Info <5008>: Logging to /usr/local/avamarclient/var/Job-A-1479423789022-a573f30f993e9a58420c69564cf2e16135540c49-1016-vmimagel.log
Not only you can view the logs from here, you also have the complete log file name too.
That's it. Sneaky tricks in place.