virtuallyPeculiar

In Part 1 we saw how to deploy and setup the basic Avamar 7.2 machine. In this article, we will have the appliance configured for usage.

Open a browser and type in:

https://avamar-server-ip/fqdn:7543/avi/avigui.html

This should bring up the below screen and you can login with root as the user and changeme as the password.

Post the login you will see the below page. Here on the top left corner you will see a lock icon. Click this and enter the password as Supp0rtHarV1. Once unlocked, click Install to begin the package installation.

The initialization will take a few minutes to complete and once done you will be presented with the below window. Here, you will need to fill all the fields with the red exclamation marks.

Under Server Settings, the Avamar Server Address should be the hostname and select an appropriate time zone.

The number of storage node I will be using is 0, just like how my VDP is setup. A single node server. This is the node you see when you run status.dpn

Fill in the remaining fields and click Continue on the bottom right. This should begin the package installation like seen below:

Once the configuration is complete you will see the below task. At this point, your AVE setup is complete.

Now you can SSH into the AVE machine with admin credentials and run the below command to verify system and service status.

admin@ave:~/>: dpnctl status
Identity added: /home/admin/.ssh/dpnid (/home/admin/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: Maintenance windows scheduler status: suspended.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: avinstaller status: up.

admin@ave:~/>: status.dpn

Tue Apr 18 00:49:01 IST 2017 [AVE] Mon Apr 17 19:19:01 2017 UTC (Initialized Mon Apr 17 17:51:04 2017 UTC)

Node IP Address Version State Runlevel Srvr+Root+User Dis Suspend Load UsedMB Errlen %Full Percent Full and Stripe Status by Disk

0.0 10.109.10.169 7.2.1-32 ONLINE fullaccess mhpu+0hpu+0hpu 1 false 0.49 5209 33104 0.0% 0%(onl:8 ) 0%(onl:8 ) 0%(onl:8 )

Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: 1492451464@00:50:56:9A:52:F6

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)

System-Status: ok

Access-Status: full

Last checkpoint: cp.20170417180550 finished Mon Apr 17 23:36:09 2017 after 00m 19s (OK)

No GC yet

No hfscheck yet

Maintenance windows scheduler capacity profile is active.

WARNING: Scheduler is WAITING TO START until Wed Apr 19 20:30:00 2017 IST.

Next backup window start time: Thu Apr 20 20:00:00 2017 IST

Next maintenance window start time: Thu Apr 20 08:00:00 2017 IST

That should be it with the configuration part. Next, we will look into how to Configure AVE to vCenter Server.

VDP - Avmar Migration: Part-2: Configuring Avamar Virtual Edition 7.2

Previously we configured AVE services. Next we need to download the Avamar Client, install it and then configure a vCenter and deploy a proxy. Let's look at this step by step.

Let's start off by downloading the Avamar Client. To get this, go to the below URL:

http://avamar-fqdn

You should see the below screen:

Click Downloads and you should see the below screen. First download the jre.exe and have it installed on the machine where you plan to run the Avamar Client. Then download AvamarConsoleMultipleWindows.exe

Run the installer, it's a pretty straightforward wizard to follow.

Once the client installation is done, open the Avamar client and you will be presented with the below login screen. The Server would be FQDN or IP of your AVE. The Username would be MCUser and the Password for the same with / Domain.

Once the login is successful, we will see the below Avamar management UI.

Notice there are no backup jobs, hence a red X for Backup Jobs tab. Now, we will have to configure vCenter to the AVE. Click Administration from the management UI and you will see a new window:

The vCenter client, should always be added to the / domain. So select the ave and then click Actions > Account Management and then click New Client.

You will see the following section. Select the Client Type as VMware vCenter. Provide the vCenter FQDN or IP for Client name. For username provide an admin user defined on the vCenter level. Disable vCenter Certificate parameter before performing this. You can find this step in this link here.
Click OK and the vCenter should be registered successfully to the AVE machine.

You will notice the vcenter-prod (my vCenter) under the ave domain added as a client.

Next, you will have to deploy a proxy, without this your migration pre-check will fail. In the same window, click VMware and Proxy Deployment Manager.

Here you can input your required setting and create a recommendation.

Click Apply and then click New Proxy and you will be presented with the below window. Enter a name for the proxy. The domain would be your /vcenter-fqdn/VirtualMachines. Provide an IP for the proxy, datastore and a network. Please note the proxy should be resolvable before performing the deployment. Click Save.

The proxy deployment will begin and you can monitor the progress here as well as the vCenter Recent Tasks pane.

This should be it. Until now, you have deployed AVE ovf, configured it, downloaded the client and configured it to a vCenter and added a proxy.

Next we will look at the migration of data and jobs from VDP to AVE.

VDP - Avmar Migration: Part-4: Migrating Data And Jobs From VDP To Avamar

VDP - Avmar Migration: Part-3: Avamar Client And Configuring To vCenter Server

In this part, we will replicate VDP backup data to Avamar and then migrate the jobs. First, to replicate the data from VDP to Avamar, we create a regular replication job. Let's have a look at this.

Connect to the VDP appliance from the web client and click the Restore tab. To give you a brief overview, here is the restore points for my VM-A

Click the Replication tab and click Replication job actions. and click Create New Replication Job.

You will be presented with the below wizard. I want to replicate my current Image backups, so the first option and click Next.

I will replicate the individual client, and select VM-A and then click Next.

Select which backup schedule you would like to replicate. This is as per your need and then click Next.

Enter the AVE FQDN/IP as the hostname, the port for replication is 29000. The username to be used is repluser and the password for repluser is the one you setup during the Avamar configuration. Click Verify Authentication. Once the connection test is completed successfully, click Next.

Select how frequently you need to replicate. This does not really matter here, because it is going to be one time replication, because post the migration source VDP will be in a unusable state. Click Next.

Select a retention policy for the replicated backup as per your requirement and click Next.

Provide a name to this replication job and then click Next and then Finish the wizard.

Once the job is created, select the job and click Replicate Now to begin the task.

Once the replication is completed, you can see the replicated data in the Avamar end.

You will still notice there is a red X for Backup Jobs as the jobs are not transferred. It is only the VDP backup data that is replicated to Avamar.

Next we will begin migrating the jobs. To do this, open a browser and go to:

https://vdp-ip/fqdn:8543/vdp-configure

Once on this page, click Avamar Migration and click Migrate.

Check the Terms and conditions to proceed further and click Next.

Provide the AVE FQDN, the root credentials would be for Avamar root and not the OS root password. Click Verify Authentication.

Once the authentication succeeds, click OK and then click Next.

You can select the backup jobs you need to migrate here.

You can switch the tab to Backup Verification and select the required verification jobs and then click Next.

A validation will be performed and if everything is successful, click Next.

You will be prompted for migration. Click Yes to begin the migration.

You can then monitor the migration progress.

Then you can review the migration status. Click Close.

Now the status is, Jobs have been migrated and the appliance is disabled.

The same is seen in the VDP page in web client as well.

On the Avamar end, you can see the jobs have now been migrated successfully.

So, in the end, the VDP would be used only for performing restore operations with the existing data. And once new valid backups are performed by Avamar, you can discard VDP completely.

Hope this series was helpful. Let me know your thoughts in the comments.

There are scenarios where you will have to reinstall a vCenter site. When a vCenter site is reinstalled the SRM solution connected to it has to be reinstalled. The reinstall is usually done with an existing database is because you would like to save your protection groups and recovery plans without the need to recreate them all. However, this reinstall with existing database would fail with the following error:

The selected vCenter server does not match the one used in the previous installation

This is because, every vCenter has a vCenter UUID with it. In this case, the vCenter being used was 6.0 and SRM was 6.1.

If you look at the instance.cfg file. It is located under:
Appliance:
/etc/vmware-vpx

Windows:
Installation directory/VMware/vCenterServer/vmware-vpx/cfg

In this instance.cfg you will have the vCenter UUID.
instanceUuid=c4c2202e-4acd-4b35-a2da-0947a3429658

When SRM is registered to vCenter, this instanceUUid is saved in the SRM database. Now, when the vCenter is reinstalled, you will have a new instanceUuid which does not match the one stored in SRM Database and the installation fails.

So, you will have to manually update the SRM database with this instanceUuid prior to installation.

The table you need to look into SRM is pd_localsite. If you connect to your DB and view this table, you will see something as below. I am using Embedded Postgres for SRM, so the query would be:

select * from pd_localsite;

The output:

path

-------+-------------+-----------+-----------------------------+-----------+--------+------------------+--------------------------------------+-------------------------------------------------------+--------------------------------------+--------------+-----------------------------------------------

---------------------------------------

covery Manager\bin\10.109.10.164su.p12

(1 row)

So the vcinstanceuuid needs to be matching your instanceUuid of instance.cfg

Once this is done, proceed with the SRM installation and it will work just fine.

When you try connecting to a specific VDP appliance from the web client, you will run into the generic Web Client Flash Error.

This error is generally encountered when there is an issue with VMware Tools on the VDP appliance. If you select the VDP VM and go to summary tab, you will notice that there are no tools running and there is no IP address for this VM.

The fix would be to uninstall VMware tools from VDP and re-install it back. To uninstall the VMware tools, run the vmware-uninstall-tools.pl script which is located under /usr/bin/
Once this script completes, the tools no longer exist on VDP. So next, you will have to re-install it back.

Right click the VM > Guest > Install or Upgrade VMtools. Then SSH into the appliance and run the below set of commands:

# mkdir /mnt/cdrom

# mount /dev/cdro /mnt/cdrom

# mount /dev/cdrom /mnt/cdrom

# /mnt/cdrom/VMwareTools-10.0.6-3560309.tar.gz /tmp
(This might vary depending on version of tools)

# cd /tmp

# tar -zxvf VMwareTools-10.0.6-3560309.tar.gz

# cd vmware-tools-distrib/

The VMtools install script has a ton of options if you go with manual install. The best is to choose default install by running the below command:

# ./vmware-install.pl -d default

This will consider all the default settings and complete the tools install and you should see the VM tools status in the VM summary page as running and the appliance should connect successfully to the web client.

Hope this helps.

Earlier in this article here we saw the issue where post an upgrade of VDP the ignore_vc_certificate parameter is flipped to false because of which the proxy registration fails. If you would like to edit the files manually you can follow that article. Else, you can use the attached script in this one.
The script basically changes the parameter, restarts the MCS and re-registers the internal proxy to vCenter.

Download the script, place it in your /home/directory of VDP appliance and run the script while logged in as admin. Provide the script a+x permissions.

You can download the script from github here:
https://github.com/happycow92/shellscripts/
https://github.com/happycow92/shellscripts/blob/master/proxy-fix.sh

The script:

#!/bin/bash
echo -e "***************************************"
echo -e "* This script is written by Suhas G  *"
echo -e "*          gsuhas@vmware.com          *"
echo -e "***************************************"
echo -e "This script fixes vCenter certificate parameter for proxy\n"
echo -e "The current value for ignore_vc_cert is\n"
value=$(cat /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml | grep ignore_vc_cert | cut -d = -f 3 | cut -d "" -f 1)
echo $value
read -p "Proceed with fix?: " choice

case "$choice" in
y|Y )
sed -i -e 's/entry key="ignore_vc_cert" value="false"/entry key="ignore_vc_cert" value="true"/g' /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml
echo -e "\nThe new value for ignore_vc_cert is"
value=$(cat /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml | grep ignore_vc_cert | cut -d = -f 3 | cut -d "" -f 1)
echo $value
echo -e "\nProceeding to restart MCS"
mcserver.sh --restart --verbose
echo -e "\nProceeding to re-register internal proxy"
sudo /usr/local/avamarclient/etc/initproxy.sh --start
echo -e "\nAll done!";;
n|N )
echo -e "\nExiting script";;
esac

If there are any issues reach out to me. Hope this helps.

Deploying vSphere Data Protection is always easy from the GUI. But with the 6.5 vCenter release there are a handful of issues with ova deployment. Hence, we use either ovf tool to deploy the VDP or we use govc CLI. If you use govc CLI, you can refer this article here.

Instead of that, you can now use the below script to deploy VDP. Download the script and the Read Me guide from my github repo here:
https://github.com/happycow92/vdp-deploy

The script:

#!/bin/bash
echo -e "*****************************************"
echo -e "*   This script is written by Suhas G   *"
echo -e "*           gsuhas@vmware.com           *"
echo -e "*****************************************"
echo -e "This Script Deploys vSphere Data Protection"
echo -e "\nDownloading govc for Linux"
curl -L https://github.com/vmware/govmomi/releases/download/v0.5.0/govc_linux_386.gz | gunzip -d > /usr/local/bin/govc
chmod +x /usr/local/bin/govc
echo -e "\nThe version of govc is"
govc version
echo -e "\nEnter the requested details"
read -p "Specify vCenter FQDN or IP Address: " GOVC_URL_a
read -p "Specify administrator username: " GOVC_USERNAME_a
read -p "Specify the password: " GOVC_PASSWORD_a
read -p "Specify the Datastore name: " GOVC_DATASTORE_a
read -p "Specify Network Portgroup: " GOVC_NETWORK_a
read -p "If needed Specify resource pool name: " GOVC_RESOURCE_POOL_a

export GOVC_INSECURE=1
export GOVC_URL=$GOVC_URL_a
export GOVC_USERNAME=$GOVC_USERNAME_a
export GOVC_PASSWORD=$GOVC_PASSWORD_a
export GOVC_DATASTORE=$GOVC_DATASTORE_a
export GOVC_NETWORK=$GOVC_NETWORK_a
export GOVC_RESOURCE_POOL=$GOVC_RESOURCE_POOL_a

echo -e "\n\nThe JSON Specification file is as below"
echo -e "Edit the file to enter your deployment details\n"
sleep 5
govc import.spec /vSphereDataProtection-6.1.3.ova | python -m json.tool
govc import.spec /vSphereDataProtection-6.1.3.ova | python -m json.tool > /vdp.json
vi /vdp.json
echo -e "Deploying VDP\n"
govc import.ova -options=/vdp.json /vSphereDataProtection-6.1.3.ova
echo "\n\nDeployment Complete. All Done!"

Hope this helps!

If you have issues with your VDP service / system functionality and would like a quick overview of how your maintenance services are doing, then look no further, for here is a script.

This script does the following task:

1. Check the core, maintenance, scheduler and management services of VDP
2. Check for VDP System Status
3. Provide an overview of VDP server information
4. Perform checks for Checkpoint, Garbage Collection and HFS check
5. Display storage information
6. Provide client information
7. Perform vCenter port and connectivity tests
8. (Optional) Generate log bundle for VMware Support
9. (Optional) email the report output for VMware Support.

The script zipped file can be downloaded from here along with read me. Please go through the read me file before running the script. Do not run the script as root, for you shall not pass!!

Script download:
https://github.com/happycow92/vdp-health-script

Enjoy!

By default when you run a command, the output is reflected onto stdout. Consider this simple example,

# echo "Hello World"

The output would be:
Hello World

The output is displayed on stdout which is my terminal.

In some cases, we will have a need to run the command, view the output and simultaneously save the output to a log file. This is where tee comes in. Consider tee as a fork in a road. The output goes both ways, one to stdout and the other to a file

Syntax:

# some command | tee <file_1> <file_2>.....<file_n>

The output of some command is displayed on stdout and stored in one or more files.

tee comes with a couple of switches. The most simple tee command is:

# echo "Hello World" | tee file.txt

The output is Hello World on stdout and if you cat file.txt you will see the output Hello World
If you run the same command with a different echo text to the same file, then the existing content is replaced.

So if you want to save the existing content and append new output, then use the -a switch

# echo "Hello World" | tee file.txt

# echo "Today is a good day" | tee -a file.txt

When you run the first command, the output on stdout is Hello World and the same output is seen when you cat file.txt

When you run the second command, the output on stdout is Today is a good day and the output when you cat the file.txt is:
Hello World
Today is a good day

Next is tee with -i or --ignore-interrupts which ignores the SIGINT which would be your Ctrl+C
Sigint sends an interrupt signal to terminate a foreground process, generally when you press Ctrl+C, but you want tee to be terminated gracefully. In short, -i ignores Ctrl+C SIGINT

When you upgrade your VDP from 5.8.x to/ot 6.0.x to/or 6.1.x you will need to uninstall the old backup agent, re download the new agent from the 6.1.x Configuration Tab and install it. However, this installation will fail with the below error:

This is because the client was not deactivated once the agent was uninstalled. Until you deactivate the client associated with the old plugin you will not be able to install the new one.

On the machine where you are installing the client, if you browse to C:\Program Files\avp\var you will have a log file called avagent.log and if you notice the error here, you will see:

2017-05-19 14:09:37 avagent Error <6210>: Unable to register '/clients/VDPApps\sqlvm' with MCS at 192.x.x.x:28001

2017-05-19 14:09:37 avagent Error <6163>: Registration error: MCS returned error code 110, '* 110 * Re-registration of account /clients/VDPApps/sqlvm.happycow.local is not permitted'

2017-05-19 14:09:37 avagent Error <7531>: Unable to register /clients/VDPApps/sqlvm with Administrator 192.x.x.x:28001

So, to deactivate this client, from the SSH of the VDP run the below command:

# mccli client edit --activated=false --name=/clients/VDPApps/<client-fqdn>

Then retry the install of the agent and it should complete successfully.

Hope this helps.

In this article, we will discuss about FLR issue Specifically, how the FLR functionality breaks when a VDP IP/hostname reconfiguration is done. Thanks to one of our customer's who logged in a ticket with VMware for this issue. This would definitely be one of the most awaited workarounds.

So, in the vdp-configure page, under proxy you will have the proxy name as the hostname of the VDP appliance (If Internal proxy is being used) When a hostname change of VDP is done, this value does not get updated in addition to one of the proxy configuration files on which FLR is based. This causes FLR to give the error:

Failed to get disks: The VMDK filename is not valid or present. Verify that the proxy was correctly registered using the supported method in the documentation.

In the command line, you can run the below command to extract the name of the internal proxy:

# mccli client show --recursive=true | grep -i /clients | awk '{print $1}'

In my case, I had a VDP with name vdp-dest.happycow.local with an IP. I changed this to vdp-new-dest.happycow.local. Then when I checked "hostname" and the proxy name using the above command, both of them reflected the updated address.

Due to this, your backups and full image restores continue to work without issues. FLR uses axionfs.cmd file to pull server parameters.

This file is located under:

# cd /usr/local/avamar/var

If you vi this file, you will notice the --server parameter is still having the old FQDN.

--server=vdp-dest.happycow.local
--id=restoreonly
--password=1aa34f26dc161939f7452aa3302133075449750f01c02148814a4563bb0965f96a61cdf9bfc96114
--fuseoptions='-s -r -f -o allow_other,use_ino'
--primarybackupdir=by-number
--inactivity_timeout=172800
--encrypt=tls
--encrypt-strength=high

So you will have to make sure the right FQDN is reflected under the --server parameter. Once this is done restart the axionfs service using:

# service axionfs restart

Post this, you should be able to perform FLR. The internal proxy would not reflect with the new hostname even if a re-register of proxy is done, however this should not be an issue.

Hope this helps.

When you create a backup job, you get an option to specify how long should the backed up data be retained in the VDP appliance. Once the client is backed up, changing this expiry date is quite a challenge. You can use the below available script to perform this task.

Note that retention can be only increased using this script and cannot be decreased.

1. Place the script in /root folder of VDP and run the script while the Run User is "root"
2. Provide execute permissions to this script

# chmod +x retention-extend.sh

3. Run the script as ./retention-extend

Test the script on a few test VMs and when comfortable you can move on to the remaining VMs. Please note, the script is not from VMware and is self developed. Run this at your own risk. (Should not affect anything on the appliance, but still. always a disclaimer!)

Download the script from:
https://github.com/happycow92/re

View the source code:
https://github.com/happycow92/re/blob/master/retention-extend.sh

Hope this helps.

This issue is applicable to 6.1.4 only. Here, you will be able to send a Test email, however, the scheduled email reports are not sent. In the vdr-server.log, you notice the following NullPointerException.

vdr-server.log:

ERROR [http-nio-8543-exec-6]-services.EmailConfigServiceWS: Unable to trigger test email java.lang.NullPointerException
at com.emc.vdp2.common.vi.ViJavaAccess.getVmInstanceUuidByIpAddress(ViJavaAccess.java:558)
at com.emc.vdp2.email.VirtualMachineSummary.addDiskRequirements(VirtualMachineSummary.java:240)
at com.emc.vdp2.email.VirtualMachineSummary.getReportNode(VirtualMachineSummary.java:222)
at com.emc.vdp2.email.VirtualMachineSummary.getVirtualMachinesReport(VirtualMachineSummary.java:131)
at com.emc.vdp2.email.EmailSummaryReport.createReport(EmailSummaryReport.java:347)
at com.emc.vdp2.email.EmailSummaryReport.createReport(EmailSummaryReport.java:260)
at com.emc.vdp2.email.EmailSummaryReport.createAndSendReport(EmailSummaryReport.java:142)
at com.emc.vdp2.services.EmailConfigServiceWS.sendEmailReport(EmailConfigServiceWS.java:111)
at com.emc.vdp2.rest.ReportService.sendEmailReport(ReportService.java:281)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)

To fix this:

1. Download the zip file from this link here.
2. The md5 hash for this is: 90a57acb1404cdff68181b6768d3b539 Verify if this is the same on the downloaded zip
3. Use WinScp and copy this file to the /root folder of VDP
4. From SSH of the VDP appliance stop the tomcat service using:

# emwebapp.sh --stop

5. Backup the original vdr-server.war file using this command:

# cp -p /usr/local/avamar/lib/vdr-server.war /root/vdr-server.war_ORIG

6. Copy the patched binary using this command:

# cp /root/vdr-server.war /usr/local/avamar/lib/vdr-server.war

Make sure the permission for this patched binary is same as the original file.

7. Remove the vdr-server* files from tomcat using:

# rm -R /usr/local/avamar-tomcat/webapps/vdr-server*

There will be two files under this webapps directory for vdr-server. Verify before removing and these will be recreated once tomcat is restarted.

8. Start the tomcat service using:

# emwebapp.sh --start

9. The vdr server initialization takes a while and then you should be able to connect to VDP in web client and test your email reporting.

If this still does not work, open a case with VMware support. Hope this helps!

Once you configure email settings in VDP, the first thing we do is send a test email to verify the configuration is good. The test email content should be something like:

This email has been generated by VMware vSphere Data Protection to test your email report settings.

However, you might come across an issue, where the test email does not trigger and you will see the following error in the web client:

In the vdr-server.log you can notice the following:

2017-06-07 01:46:27,636 ERROR [http-nio-8543-exec-9]-email.EmailBase: On try 0to send the email, recieved a connection error from the external email server:Could not connect to SMTP host: email.vmware.com, port:
587
2017-06-07 01:46:27,636 ERROR [http-nio-8543-exec-9]-services.EmailConfigServiceWS: Unable to trigger test email
javax.mail.MessagingException: Could not connect to SMTP host: email.vmware.com, port: 587;
nested exception is:
java.net.ConnectException: Connection timed out (Connection timed out)
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1282)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:370)
at javax.mail.Service.connect(Service.java:297)
at javax.mail.Service.connect(Service.java:156)
at javax.mail.Service.connect(Service.java:105)
at javax.mail.Transport.send0(Transport.java:168)
at javax.mail.Transport.send(Transport.java:98)
at com.emc.vdp2.common.email.EmailBase.sendReport(EmailBase.java:174)
at com.emc.vdp2.common.email.EmailBase.sendReport(EmailBase.java:63)
at com.emc.vdp2.email.EmailSummaryReport.sendTestEmail(EmailSummaryReport.java:129)
at com.emc.vdp2.services.EmailConfigServiceWS.runEmailTest(EmailConfigServiceWS.java:93)
at com.emc.vdp2.rest.ReportService.runEmailTest(ReportService.java:241)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)

Use SMTP port 25 to fix this issue.
So in the configuration of mail settings, in the field for Outgoing mail server, specify
<your mail sever > : 25

Note that there is <space> before and after the :
Save the settings and now the test email should work.

I believe port 587 is block on Firewall.base for VDP and explicitly mentioning 25 resolves this, as SMTP 25 is open on VDP.
You can netstat -ano | grep 25 and netstat -ano | grep 587

Hope this helps!

Some of you might have tried to backup VDP using a VDP appliance (Or self backup a VDP) and would have failed to get this VM added to a backup job. The admin guide states that you cannot backup VDP using a VDP appliance. This is mainly because the data0? partitions of your data protection appliance are Independent persistent XFS drives which do not support snapshot feature. Yet, backing up the OS drive is some of us look forward to, right? Your OS drives contains your backup job, replication job and yeah, "logs", which no one needs.

Though not an officially supported way, here is a sneaky way to work around this.

Why does the VDP VM not list in the VM section when you create a backup job?

Good question. If you select the VDP VM in the vSphere Client inventory and then hit the summary tab, under Annotations section, you will notice you have "VMware vSphere Data Protection Appliance 6.1" under Notes

If any VM has this text in the Notes section, then it will not be listed in the inventory list while creating a backup Job. Go ahead, spin up a test VM, and then try creating a backup Job, you will notice your test VM. Then add this Notes section and you will notice this VM no longer shows up in the Inventory list during Job creation.

So you guessed it right, Go ahead and remove this Notes section from your VDP appliance and you will now see the VDP listed under the job create wizard. Make sure the job you create is a disk level job and not a complete VM level backup.

Hope this helps. In some way. Heh!

If you ever run into vSphere Replication Appliance deployment issues from web client (Specially in Pre U1 release of vCenter 6.5) you can use the OVF tool to deploy this appliance. Here is a quick walk-through of how to get this done.

1. Download the OVF tool using the below link
Download: https://my.vmware.com/web/vmware/details?downloadGroup=OVFTOOL400&productId=353

2. Install the OVF tool by running the exe file. The installation is pretty straight forward. As simple as calculating 5x5

3. Open the Command Prompt in administrator mode and change your directory to the install location of OVF tool. By default, this should be:

C:\Program Files\VMware\VMware OVF Tool

4. Below is the syntax for OVF tool deployment:

ovftool.exe --acceptAllEulas -ds="DATASTORE_NAME" -n="SPECIFY VRMS NAME" --net:"Management Network"="PORT GROUP NAME" --prop:"password"="VRMS ROOT PASSWORD" --prop:"ntpserver"="NTP SERVER IP OR FQDN" --prop:"vami.ip0.vSphere_Replication_Appliance"="SPECIFY VRMS SERVER IP" --vService:installation=com.vmware.vim.vsm:extension_vservice <PATH>\vSphere_Replication_OVF10.ovf vi://"administrator@vsphere.local":<SSO-PASSWORD>@VCENTER IP/?ip=HOST IP

Below is the sample command from my environment:

ovftool.exe --acceptAllEulas -ds="Local_Storage_1_39" -n="VR-OVF" --net:"Management Network"="VM Network 2" --prop:"password"="HappyCow123!" --prop:"ntpserver"="10.x.x.x" --prop:"vami.ip0.vSphere_Replication_Appliance"="10.x.x.x" --vService:installation=com.vmware.vim.vsm:extension_vservice E:\bin\vSphere_Replication_OVF10.ovf vi://"administrator@vmware.local":HappyCow123!@10.x.x.x/?ip=10.x.x.x

The switches:
-ds is for datastore
-n is to specify display name for the vR appliance (Not the host name, but the VM inventory name)
--net is to specify the port group for vR
--prop:"password" is vR root password
--prop:"ntpserver" is for your time server

The path would be the location of the OVF10.ovf file. If you downloaded like say 6.1.1 ISO and mounted it, then it would be the <CD-ROM-Drive>\bin\vSphere_Replication_OVF10.ovf

Always use SSO username for deployment and make sure it is within the double quotes, else you may run into issue where the '@' is not recognized

You should see a similar output once you execute the command:

Opening OVF source: //:@jump.happycow.local:80\bin\vSphere_Replication_O
The manifest validates
Source is signed and the certificate validates
Opening VI target: vi://administrator%40vmware.local@10.x.x.x:443/
Deploying to VI: vi://administrator%40vmware.local@10.x.x.x:443/
Transfer Completed
Completed successfully

You will also see a progress percentage along with a recent task for OVF deployment in the vSphere Client to monitor.

Hope this was helpful.

When you backup a VM residing on ESXi 5.1 which does not share the storage with the ESXi hosting the VDP appliance, the backup will use nbd / nbdssl protocol. With VDP 6.1.4 these backups will fail.

In the backup log you will notice the following:

2017-06-14T09:21:07.869+05:00 avvcbimage Info <14700>: submitting pax container in-use block extents:
2017-06-14T09:21:07.899+05:00 avvcbimage Warning <16041>: VDDK:[NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 2 from server: Illegal message during fssrvr session, id = 46
2017-06-14T09:21:07.900+05:00 avvcbimage Info <16041>: VDDK:DISKLIB-LIB : RWv failed ioId: #1 (290) (34) .
2017-06-14T09:21:07.900+05:00 avvcbimage Info <16041>: VDDK:VixDiskLib: Detected DiskLib error 290 (NBD_ERR_GENERIC).
2017-06-14T09:21:07.900+05:00 avvcbimage Info <16041>: VDDK:VixDiskLib: VixDiskLib_Read: Read 128 sectors at 0 failed. Error 1 (Unknown error) (DiskLib error 290: NBD_ERR_GENERIC) at 5178.

2017-06-14T09:21:07.900+05:00 avvcbimage Error <0000>: [IMG0008] VixDiskLib_Read() at offset 0 length 128 returned 1 (1) Unknown error
2017-06-14T09:21:07.900+05:00 avvcbimage Error <0000>: [IMG0008] VixDiskLib_Read() ([Prod-Datastore] VM-Prod/VM-Prod.vmdk) at offset 0 length 128 sectors returned (1) (1) Unknown error
2017-06-14T09:21:07.900+05:00 avvcbimage Info <9772>: Starting graceful (staged) termination, VixDiskLib_Read returned an error (wrap-up stage)
2017-06-14T09:21:07.901+05:00 avvcbimage Info <40654>: isExitOK()=157
2017-06-14T09:21:07.901+05:00 avvcbimage Info <16022>: Cancel detected(miscellaneous error), isExitOK(0).
2017-06-14T09:21:07.901+05:00 avvcbimage Info <9746>: bytes submitted was 0

You might also notice a back trace:

2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] INTERNAL ERROR: <0001> assert error (uapp::staging().isExitCancel() || list == __null || skipped_ntfs_chunks), /local/jenkins/workspace/client_CuttyS
ark_SLES11-64/abs2/work/src/avclient/filepipe.cpp line 101
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00000000006b93b1
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00000000006ba127
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00000000006ba292
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00000000006ba3fe
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00000000006ba87d
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 000000000060cb74
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 000000000060d67d
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00000000006d9379
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00000000006dc9a1
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00000000006dc200
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00000000006d9379
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00000000005e5af1
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 000000000056455a
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 0000000000442c07
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00000000004844d5
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00000000004844f9
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 000000000049a0bd
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 000000000070ac4c
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00007f8a8d282806
2017-06-14T09:21:08.322+05:00 [VcbImageBackupAssistThread] | 00007f8a8bec79bd

This is because, in the newer release of VDP for backup over network, they have added couple of new compression algorithms. Zlib, FastLZ, SkipZ in the VDDK module. And ESXi 5.1 does not support this.

You could try making the VMs storage on ESXi 5.1 available to the VDP's ESXi host so a hotadd protocol can be used.

But the bottom line is, Please! Upgrade your ESXi from 5.1 (This is officially unsupported for 6.1.3 and 6.1.4 VDP)

Hope this helps.

If you ever run into issues with vSphere Replication deployment in 6.5 or any other versions, you can use ovf tool to perform this deployment. Ovf tool might look a little complex to perform the deployment as the commands are quite big.

For windows version refer the below link:
http://www.virtuallypeculiar.com/2017/06/deploying-vsphere-replication-using-ovf.html

If you have a Linux based environment you can use this shell script I wrote to automate the process. (User intervention needed to enter environment details, duh!)

To download the script and the ReadMe (Before running anything) refer:
https://github.com/happycow92/vsphere-replication-deploy/

I will be making a few changes to this in the coming days, but the base functionality will remain more or less the same.

Hope this helps.

I am going to cover two things in this post rather than splitting it up because it is fairly simple. Before we get started, we had seen deploying VDP from a tool called govc CLI (Thanks to William Lam for a good article on this tool)

Govc CLI VDP deployment:
www.virtuallypeculiar.com/2017/03/deploying-vsphere-data-protection-from.html

Govc CLI:
http://www.virtuallyghetto.com/2014/09/govmomi-vsphere-sdk-for-go-govc-cli-kubernetes-on-vsphere-part-1.html

Another way to deploy this is using our good old OVF tool, which is available for Windows and for Linux (For which we will be bash scripting)

Deploying using OVF tool for Windows:

1. Download OVF tool 4.2 from the below link:
https://my.vmware.com/web/vmware/details?productId=614&downloadGroup=OVFTOOL420

2. Install the tool on a windows box

3. Open a command prompt and change your directory to install location of OVF Tool.

The syntax for deploying VDP would be:

ovftool.exe --acceptAllEulas -ds="<Datastore-name>" --net:"Isolated Network"="<vm-portgroup-name>" --prop:"vami.gateway.vSphere_Data_Protection_6.1"="<gateway-address>" --prop:"vami.DNS.vSphere_Data_Protection_6.1"="<dns-address>" --prop:"vami.ip0.vSphere_Data_Protection_6.1"="<ip-address>" --prop:"vami.netmask0.vSphere_Data_Protection_6.1"="<subnet-mask>"<Location-of-the-file>\vSphereDataProtection-6.1.3.ova vi://<vcenter-ip/fqdn>/<data-center-name>/host/<cluster-name>

So a sample command would look like:

ovftool.exe --acceptAllEulas -ds="Local_Storage_1_39" --net:"Isolated Network"="VM Network 2" --prop:"vami.gateway.vSphere_Data_Protection_6.1"="10.x.x.x" --prop:"vami.DNS.vSphere_Data_Protection_6.1"="10.x.x.x" --prop:"vami.ip0.vSphere_Data_Protection_6.1"="10.x.x.x" --prop:"vami.netmask0.vSphere_Data_Protection_6.1"="255.x.x.x" C:\Users\Administrator\Downloads\vSphereDataProtection-6.1.3.ova vi://vcenter-prod.happycow.local/HappyCow-DC/host/HappyCow-Cluster

This will then prompt you to enter SSO username and SSO password and once authenticated it will proceed with the deployment.

Deploying Using Bash Script with Linux.

If you have a Linux based environment, then I have automated this process using bash and the script can be downloaded from the below link:
https://github.com/happycow92/vdp-deploy-ovftool

Please go through the ReadMe Before performing any deployments.

A copy of ReadMe:

1. Place the VDP OVF in the /root folder of the Linux appliance
2. Download the script and place it in the /root folder
3. If the Linux VM has internet access, then the script will download and install ovf tool If not, it will exit the script and you will have to manually install ovf tool on linux and then run the script
4. If the script detects the ovftool is already installed it will proceed further with the deployment of VDP where user inputs are requested.

Note the script and deployment is only for VDP 6.1. Hope this helps!

There might be situations where you have like a handful of VMs with multiple restore data, and you want to know how many restore data is available for each of these clients. Perhaps, you would like to determine if you could perform some maintenance and get rid of few of them that are having a large number of backups to free up some space.

If you connect to VDP from the regular Web Client plugin, you will have to select each of the VM and then scroll to count, note that the GUI does not include a section to number to list. This would be a tedious task if you have 20+ VMs with backups in a varying range of 10-20+

You can use this below simple bash script to get this done. This will basically list out all the clients protected by VDP and number of backups present within each client.

#!/bin/bash

clear

# Print purpose of script
echo -e "\nAvailable number of backups for each client protected by VDP\n"

# Save vCenter hostname to a variable
vcenter_name=$(cat /usr/local/vdr/etc/vcenterinfo.cfg | grep vcenter-hostname | cut -d '=' -f 2)

# List clients in GSAN and save to variable
client_list=$(avmgr getl --path=/$vcenter_name/VirtualMachines | awk '{print $2}' | tail -n+2)

# List Backups for each of the registered client
count=1 # For Sl.No increment

for i in $client_list
# Begin For
do
        number_of_backup=$(avmgr getb --path=/$vcenter_name/VirtualMachines/$i | tail -n+2 | wc -l)
        printf "\n$count. For $(echo $i | cut -d '_' -f 1) the number of backups available are: $number_of_backup"
        ((count++)) #Increment Sl.No
done

# Done with For

echo
echo

You should see an output similar to:

Available number of backups for each client protected by VDP

1. For VM-A the number of backups available are: 3
2. For VM-B the number of backups available are: 3
3. For VM-C the number of backups available are: 3
4. For window the number of backups available are: 8

If you think, you would like some more information along with this, then leave a comment. I will further develop this script as needed. Hope this helps.

VDP - Avmar Migration: Part-2: Configuring Avamar Virtual Edition 7.2

VDP - Avmar Migration: Part-3: Avamar Client And Configuring To vCenter Server

VDP - Avmar Migration: Part-4: Migrating Data And Jobs From VDP To Avamar

Unable To Reinstall SRM With Existing Database "The selected vCenter server does not match the one used in the previous installation"

VDP Error 1009 When Connecting To Web Client

Script For VDP Proxy Registration Issue

Shell Script To Deploy vSphere Data Protection

Shell Script To Monitor vSphere Data Protection 6.x Health Status

Shell Command: Understanding tee

Unable To Install SQL/Exchange Backup Agent on vSphere Data Protection

VDP 6.x FLR - Failed To Get Disks: The VMDK Filename Is Not Valid Or Present

Bash Script To Extend Retention/Expiry Date Of A Restore Point In vSphere Data Protection 6.x

Unable To Send Email Report In VDP 6.1.4

Unable To Send Test Email In vSphere Data Protection 6.x

Backup vSphere Data Protection Using vSphere Data Protection? Really!

Deploying vSphere Replication Using OVF Tool

VDP 6.1.4 Backup Failures For ESXi 5.1

Automating vSphere Replication Deployment Using Bash Script

Deploy vSphere Data Protection 6.1 Using OVF Tool + Automating The Deployment With Bash

Bash Script To List Number Of Backups In VDP For Each Client