WinSCP with a 6.x vCenter Appliance

March 2, 2016, 1:11 pm

Written by Suhas Savkoor

With 6.x you have two shells. Appliance shell and the bash shell. When you take a SSH session to the appliance, you will enter the appliance shell and you will have to enable the bash shell manually by running two commands that is displayed during the session.

Now, with this setup, if you try to use the WinSCP then the login will fail, as it will not be able to open the bash shell.
So here, we need to set the appliance to open with bash shell as default. To do this, login to the appliance with bash shell access and run the below command:

# chsh -s "/bin/bash" root

You will see the output

Changing login shell for root.
Shell changed.

Now you can WinSCP into your appliance machine.

↧

Customize Web Client Login Page

March 2, 2016, 5:16 pm

≫ Next: Changing Logging Location For vCenter 6.0

≪ Previous: WinSCP with a 6.x vCenter Appliance

Written by Suhas Savkoor

Bored of the same blue screen login for vSphere Web Client? Well, don't be bored any more. If you are running 6.0 VMware, then there's a bunch of good news. You can have your own welcome screen for Web Client. Yes, you heard me right!

Web Client login page for 6.0, is designed using HTML and CSS, and when you have this, you can tweak around a little to customize your own page. With the help of William Lam's article, I was able to tweak my Web Client Appearance.

This is how it looks now!

So, here is the bigger story of how to get this done:

>> The first thing is, you need to be on 6.x vCenter, this is not going to work with any other versions of vCenter.
>> There are three files to be altered: the img file, the login.css file and the unpentry.jsp file

Location of these files:

Windows vCenter:

C:\ProgramData\VMware\vCenterServer\runtime\VMwareSTSService\webapps\websso\WEB-INF\views\unpentry.jsp
C:\ProgramData\VMware\vCenterServer\runtime\VMwareSTSService\webapps\websso\resources\css\login.css
C:\ProgramData\VMware\vCenterServer\runtime\VMwareSTSService\webapps\websso\resources\img

VCSA 6.0

/usr/lib/vmware-sso/vmware-sts/webapps/websso/WEB-INF/views/unpentry.jsp
/usr/lib/vmware-sso/vmware-sts/webapps/websso/resources/css/login.css
/usr/lib/vmware-sso/vmware-sts/webapps/websso/resources/img

>> This has to be modified on the machine where your Platform Services Controller is installed. If it is an embedded installation, then you can find it all in one machine. If it is an external PSC deployment, then all these file locations are on the PSC machine.

>> I am using a 6.0 vCenter Appliance with External PSC, so I used WinSCP to transfer files from my desktop to the appliance.

>> If you are using appliance with WinSCP, then you need to set the bash shell to be opened as default. To do this, you can use this article here

**Have a backup of all these three above files before you make any changes**

The procedure:

>> Download a required image and save it as "AppBgPattern.png" (PNG format), as this naming convention would eliminate editing certain lines in the CSS/JSP file.

>> Login to WinSCP and browse to the "img" directory as mentioned above and paste this copied image

This takes care of the background image for the Web Client.

Now, you will have to edit the unpentry.jsp file to reflect the text message.

>> Once the backup of unpentry.jsp file is made from it's location, open the file and copy paste the below text into it:

<%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>

<%@ page session="false" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"

"http://www.w3.org/TR/html4/strict.dtd">

<!--[if lte IE 8]>

<![endif]-->

<head>

<title>Login</title>

// copying JSP variables to JS

var tenant_brandname="${tenant_brandname}";

var searchString ="${searchstring}";

var replaceString ='${replacestring}';

var error = '${error}';

var errorSSPI = '${errorSSPI}'

</script>

</head>

<body>

// regex to check for internet explorer 11 and below

var isInternetExplorer = /MSIE (\d+\.\d+);/.test(navigator.userAgent) || /Trident\/(\d+\.\d+);.*rv:(\d+\.\d+)/.test(navigator.userAgent);

if (!isVCLogin()) {

document.write('<link rel="stylesheet" type="text/css" href="../../resources/css/login_generic.css">');

}

else {

document.write('<link rel="stylesheet" type="text/css" href="../../resources/css/login.css">');

}

</script>

//createProbes();

//createVmrcPluginObject();

//var _cspId = createCspPluginObject();

if (isVCLogin()) {

//document.write("<img id=\"topSplash\" src=\"../../resources/img/AppBgPattern.png\"/>");

document.write("<img id=\"brand\" src=\"../../resources/img/vmwareLogoBigger.png\" />");

}

else {

document.write("<p id=\"tenantBrand\">"+tenant_brandname+"</p>");

}

</script>

<span class="loginLabel">${username}:</span>

</p>

<span class="loginLabel">${password}:</span>

</p>

<label id="checkboxLabel"><input id="sspiCheckbox" disabled="true" type="checkbox" onchange='enableSspi(this);'/>${winSession}</label>

</p>

</p>

</div>

Welcome To South Park</br></br>

if (isVCLogin()) {

document.write("<img id=\"VCSSO-Title\" src=\"../../resources/img/VCSSO-title.png\" />");

}

</script>

</br></br>

</div>

<a id="downloadCIPlink" target="_blank">${downloadCIP}</a>

</span>

</div>

<span class="validation-message-text">${unsupportedBrowserWarning}</span>

</div>

if (isVCLogin() && !isBrowserSupportedVC()) {

$(".browser-validation-banner").css("visibility","visible");

}

</script>

</body>

</html>

The text highlighted in "Red" can be edited to display the required message.

>> Save this file

Next, we will have to edit the CSS file to apply colors and font size to this field or any other field as per requirement:

>> Go to the directory where the CSS file is and simply paste the below code:

@charset "utf-8";

/* CSS Document Login.css */

.base-app-style {

min-width: 1024px;

min-height: 612px;

position: relative;

height: 100%;

width: 100%;

}

body {

background: #3075ab; /* Old browsers */

background: -moz-linear-gradient(top, #3a8dc8 0%, #183a62 100%); /* FF3.6+ */

background: -webkit-gradient(linear, left top, left bottom, color-stop(0%,#3a8dc8), color-stop(100%,#183a62)); /* Chrome,Safari4+ */

background: -webkit-linear-gradient(top, #3a8dc8 0%,#183a62 100%); /* Chrome10+,Safari5.1+ */

background: -o-linear-gradient(top, #3a8dc8 0%,#183a62 100%); /* Opera 11.10+ */

background: -ms-linear-gradient(top, #3a8dc8 0%,#183a62 100%); /* IE10+ */

background: linear-gradient(to bottom, #3a8dc8 0%,#183a62 100%); /* W3C */

filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#3a8dc8', endColorstr='#183a62',GradientType=0 ); /* IE6-9 */

background-image: url(../../resources/img/AppBgPattern.png);

background-repeat:no-repeat;

margin: 0;

font-size:12px;

font-family:Arial, Helvetica, sans-serif;

color: #87ceff;

}

.browser-validation-banner {

padding: 6px 12px 6px 10px;

margin: 0px;

position: absolute;

line-height: 20px;

top: 0px;

left: 0px;

right: 0px;

z-index: 100;

background-color: #ffffcc;

border: 1px solid #efe69b;

border-top-style: none;

border-radius: 0 0 5px 5px;

color: #333333;

box-shadow: 0 3px 3px rgba(0, 0, 0, 0.15);

}

.validation-message-text {

margin-left: 25px;

display: block;

font-weight: bold;

font-size: 11px;

}

#topSplash {

position: absolute;

top:0;

left:0;

z-index:1;

}

#brand {

position: absolute;

top: 55px;

left: 120px;

z-index:2;

}

#loginForm {

position: absolute;

top: 320px;

width: 405px;

padding-left: 44px;

z-index:3;

background-image:url(../img/divider.png);

background-position:right;

background-repeat:repeat-y;

}

.margeTextInput {

float:right;

margin-right: 44px;

width:200px;

}

.loginLabel {

font-weight:bold;

color: #FFFFFF;

text-align:left;

display:block;

padding-top:4px;

float:left;

}

.loginRow {

display:block;

padding-top:20px;

}

#sessionID {

clear:both;

float:right;

width:248px;

text-align:left;

margin-top:8px;

}

#checkboxLabel {

display: block;

padding-left: 20px;

text-indent: -20px;

}

#sspiCheckbox {

width: 13px;

height: 13px;

padding: 0;

margin:0;

margin-right:10px;

vertical-align: bottom;

position: relative;

top: -1px;

left: 5px;

*overflow: hidden;

}

#loginButtonRow {

clear: both;

float:right;

margin-right:40px;

margin-top:0px;

}

.button {

display: inline-block;

zoom: 1; /* zoom and *display = ie7 hack for display:inline-block */

*display: inline;

vertical-align: baseline;

margin: 0 2px;

outline: none;

cursor: pointer;

text-align: center;

text-decoration: none;

padding: 0px 30px;

height:24px;

font-size:12px;

}

.button:hover {

text-decoration: none;

}

.button:active {

position: relative;

top: 1px;

}

.blue {

color: #fff;

border: solid 2px #1d4772;

background: #0095cd;

background: -webkit-gradient(linear, left top, left bottom, from(#0c6fb9), to(#014781));

background: -moz-linear-gradient(top, #0c6fb9, #014781);

filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#0c6fb9', endColorstr='#014781');

}

.blue:hover {

background: #007ead;

background: -webkit-gradient(linear, left top, left bottom, from(#0d84dc), to(#004d88));

background: -moz-linear-gradient(top, #0d84dc, #004d88);

filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#0d84dc', endColorstr='#004d88');

}

.blue:active {

color: #fff;

background: -webkit-gradient(linear, left top, left bottom, from(#035a9d), to(#086fba));

background: -moz-linear-gradient(top, #035a9d, #086fba);

filter: progid:DXImageTransform.Microsoft.gradient(startColorstr='#035a9d', endColorstr='#086fba');

}

#username:disabled,

#password:disabled,

#submit:disabled,

#sspiCheckbox:disabled {

opacity:0.3;

filter:alpha(opacity=30); /* For IE8 and earlier */

}

#productName {

position: absolute;

top: 345px;

left:499px;

color:#F88017;

font-size:23px;

}

#response {

background:url(../../resources/img/error.png) left center no-repeat;

margin-top: 20px;

font-size: 12.5px;

padding-left: 20px;

color: #87CEFF;

}

#rolo {

font-weight: bold;

margin-top: 20px;

font-size: 16px;

padding-left: 10px;

color: #FF3333;

}

#progressBar {

display: none;

}

.footer {

background-color: black;

position: absolute;

left: 0px;

right: 0px;

bottom: 0px;

font-weight: bold;

padding: 10px;

color: #7B7E81;

padding-left: 44px

}

.footer a {

color : #7B7E81;

text-decoration: none;

}

.footer a:hover{

text-decoration: underline;

}

.info-icon {

display: inline-block;

width: 16px;

height: 16px;

line-height: 16px;

vertical-align: middle;

background-repeat: no-repeat;

margin: 0px 4px 0;

position: relative;

background-image: url(../../resources/img/infoOver.png);

cursor: pointer;

}

The text highlighted in red, is the part to Edit the Product area text. if you want to change the colors of the remaining sections, find the appropriate ones in the CSS file and change them.

>> Save this file as well.

>> Reload your Web Client

And that's it, you have your own custom Web Client Login Page.

You can find more samples on GitHub, click here

Enjoy!

↧

Changing Logging Location For vCenter 6.0

March 11, 2016, 7:23 am

≫ Next: Client Integration Plugin 6.0 Not Working With IE 11 / Chrome 45 and above

≪ Previous: Customize Web Client Login Page

Written by Suhas Savkoor

By default your vCenter 6.0 logs are in the install directory.

C:\ProgramData\VMware\vCenterServer\logs\vmware-vpx

Now, if you do not want to have this location for logging, but a custom location, then you will have to make a few changes.

Perform the below steps:

1. Navigate to the below directory:

C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpx

2. Make a copy of the vpxd.cfg file so that you can quickly restore if something goes wrong.
3. Stop VMware Virtual Center service from services.msc
4. Open the vpxd.cfg file using a notepad/wordpad

5. There are two "<log> </log>" tags that are available. Add the below line between <log> and </log> [<log> represents the start of the xml code of log function and </log> determines the end]

<directory>[Directory:\Name\configured]</directory>

**Make sure that this is added under both <log></log> tags**

6.Save the vpxd.cfg file
7. Start the VMware Virtual Center service.

Your new logging will take place in the custom directory that was defined.

↧

Client Integration Plugin 6.0 Not Working With IE 11 / Chrome 45 and above

March 15, 2016, 2:46 pm

≫ Next: Deploying External Proxy for VDP

≪ Previous: Changing Logging Location For vCenter 6.0

Written by Suhas Savkoor

There are always issues regarding Client Integration Plugin not working as expected is logged with VMware. At least, I handle one on a weekly basis.

The main thing what happens here is, you download and install CIP from the download option you see at the bottom when you open a Web Client session. Once the installation wizard starts, you close out all the browser sessions and complete the install. After the installation, you open up a browser and you still see the download CIP message and the Use Windows Session Credentials is grayed out, always!

There are couple of steps that we take into account before we proceed with the troubleshooting. Go through and validate all the steps. (If I find any more workarounds, I will update this article)

Since, CIP is based on each workstation where web client is accessed from, this has to be checked on each of the machines

1. If there is UAC enabled on the Windows machine from where you are accessing web client, disable it

2. Check if the below entry is there in host file on the same machine.

127.0.0.1 vmware-localhost

::1 vmware-localhost

3. Open the task manager, if you find vmware-csd.exe, kill this process and continue with Step 4

4. Uninstall Client Integration Plugin from Control Panel.

5. Re-install by using the download link from this KB article here

Once the plugin is reinstalled, follow the below steps:

1. On the required browser, clear the cache/cookies

2. In Chrome / IE 11 add the https://vcenter_FQDN:9443 (Web Client URL) in the trusted site list.

3. Refresh the browser and open web client again.

If the Client integration plugin still does not work, then:

1. In IE 11 change the trusted site exception to Local Intranet (For the same vCenter FQDN)

2. In Chrome add the exception to the Plugin as well (All options are in settings and advanced settings of chrome)

3. Refresh the browser and load the web client again

If it still fails, then implement the above 2 sections (adding exception and clearing browser cache) with vCenter IP as well along with FQDN.

Even after all this, if it still fails, please contact VMware Support, and maybe me or someone else will get to further troubleshooting.

Cheers!

↧

Deploying External Proxy for VDP

March 28, 2016, 3:01 pm

≫ Next: Re-deploy VDP with existing storage

≪ Previous: Client Integration Plugin 6.0 Not Working With IE 11 / Chrome 45 and above

Written by Suhas Savkoor

With VDP, you get 8 internal proxy by default. Using this, you can backup up to 8 VMs concurrently.

The moment you configure an external proxy for the VDP, the internal proxy is disabled. So with external proxies you can deploy up to 8 external proxy VM. And each external proxy again supports up to 8 concurrent backups. However, with for example 2 external proxies, you can have 16 concurrent backups. This does not mean that we go ahead and deploy all 8 external proxies and have 64 backups running concurrently. This will have a huge performance impact on your environment. Hence, choose the external proxies as required based on your environment.

To deploy an external proxy:

1. Login to your VDP management page.

https://vdp-IP:8543/vdp-configure

2. Click the Gear icon on the Proxy row and select Add External Proxy

2. Provide the host where this proxy virtual machine should reside, the storage and the network. Please provide a Standard vSwitch Port Group and make sure the underlying ESXi host has 4 CPU cores. If it is deployed on a DVS or on a host less than 4 CPUs, then you will run into the error "VDP: Failed to find CIM service on VM Image Proxy"

3. Enter the network configuration details for the proxy virtual machine

4. Enter the name for the proxy virtual machine

5. Click Finish and wait for the deployment to be completed. You will come across the following message for a successful deployment.

Click Close and now under the proxy list, you will see the external proxy in a working status.

To delete an external proxy:

1. Click the gear icon again and select Manage Proxies. Check Review information and proceed to remove the proxy.

Now, once the external proxy is removed the internal proxy is not enabled automatically. We will have to perform this step manually. Click the gear icon again and select Enable internal proxy

Check box the Enable internal proxy and click Finish.

The proxy status will be in warning state. Refresh the proxy using the refresh button next to it in about 5-10 minutes and we should be good to go.

↧

Re-deploy VDP with existing storage

April 1, 2016, 1:09 pm

≫ Next: Migrate Networking From Standard Switch To Distributed Switch

≪ Previous: Deploying External Proxy for VDP

Written by Suhas Savkoor

You will come across instances where your VDP appliance has gone corrupt or the appliance is not booting at all no matter what fix you have implemented, in this case the simple and easy resolution would be to redeploy the appliance. When you redeploy the appliance in VDP, you will have an option to attach an existing storage to the appliance.

So your VDP appliance comprises of X number of disks, with the hard disk 1 always being your OS disk, and the remaining ones created to store backup data. Now, when you re-deploy the appliance, the OS disk is replaced completely. This means, your backup jobs or email configuration for the VDP appliance is lost. So, once the appliance is redeployed you will have to recreate your backup jobs. However, the backup data will be intact as it resides on the storage disk.

What you need to do:

So, here before discarding my old appliance, you can see the backed up virtual machine available under Restore of VDP.

Make a note of the data disks for the VDP appliance from the Edit Settings of the virtual machine.

Here the Hard Disk 1 is the OS disk hosting the backup jobs and the SMTP settings for VDP and DIsk 2/3/4 are the storage disk where the backup data will be stored.

Power OFF the old VDP appliance and remove it from the inventory.

Follow the OVF deployment procedure to deploy the new VDP appliance. Once the virtual machine is deployed, power it on and wait for the boot to be complete. Then browse to the vdp-configure page:

https://<VDP_IP>:8543/vdp-configure

Go through the initial configuration step, until you come across Create Storage. In this wizard, select Attach Existing VDP Storage and click Next.

Browse the location where the existing data disk for VDP resides. Select each disk one by one and mount them. There is no option to select all VMDKs at a time and perform the mount. This has to be done for each disks. It's going to validate each disk.

Once all the disks are mounted, then proceed Next and it's going to validate the set of disks.

Proceed ahead to complete, and each disk is imported and attached to the new VDP appliance, and a mount point is created for each disk. Reboot the machine once completed, and it is going to run a VDP: Configure task for about 15-20 minutes.

Once the appliance is configured, login to Web Client and connect to the appliance. Go to the Restore tab and you can see the backed up data still available.

You will have to just reconfigure your backup jobs for the virtual machines.

↧

Migrate Networking From Standard Switch To Distributed Switch

April 2, 2016, 2:55 am

≫ Next: Migrate Networking From Distributed Switch To Standard Switch

≪ Previous: Re-deploy VDP with existing storage

Written by Suhas Savkoor

In this article, let's see how to migrate your ESXi host networking from vSphere Standard Switch to vSphere Distributed Switch.

Step 1:
Here we see the current configuration of my standard switch for one of my ESXi host.

I have one standard switch, with two portgroups; one for virtual machines and other one is the management network. I have simplified the networking by eliminating additional vmkernel portgroups for vMotion, FT, iSCSI as the process to move them would be the same. I have one uplink given to this standard switch, vmnic1.

Step 2:

Let's go ahead and create a distributed switch. Go to Home and select Networking. Right click the required Datacenter and click New vSphere Distributed Switch.

Step 3:

Select the version of the distributed switch that you are going to create

Step 4:

Provide a name to this distributed switch and if you want to alter the number of uplink ports to this switch, you can do the same in the Number of Uplink Ports section.

Step 5:

I am going to add hosts later as I like to review and make sure I got the setup right before I start moving anything off my standard switch.

Review your settings and Finish in the Ready to Complete section.

Step 6:

Navigate back to the Networking section, now you can see your distributed switch created under the specified Datacenter. Right click this switch and select Add Host

Step 7:

Select the host that you want to add. Now, you can see that I have two uplinks in the menu for this host. vmnic0 and vmnic1. You need to make sure that you have one free uplink when you add the host to the distributed switch. This is because, when you are migrating your portgroups off standard switches and you do not have any uplinks on the vDS, your networks are going to be disconnected.

Here, I will choose the free unused adapter, vmnic0, to be added to the vDS.

Step 8:

As seen in the standard switch configuration, I had one vmkernel port-group, vmk0. I am not going to migrate this port group right now. You can do it at this stage by simply using the drop-down under Destination Port Group and selecting the required port-group on the vDS as to where your management network must migrate to.

Step 9:

I am neither moving any virtual machine networking as well because I will be doing both of these steps later. Review your settings and complete the host add to vDS.

Step 10:

Now, we will migrate the VMkernel from standard switch to the vDS. Select the Host and click the Configuration tab and browse to Networking > vSphere Distributed Switch. Click Manage Virtual Adapters.

Step 11:

Click Add to check the required vmk.

Step 12:

Select Migrate existing virtual adapter as we already have the vmk in the standard switch.

Step 13:

Select the required port group and the destination port-group in vDS under the section "Port Group"

Review the settings and complete the migrate. It will take a couple of seconds to finish the migrate. You can also do a continuous ping to the host to check the network connectivity. Once migrated, you can review your vDS diagram.

Step 14:

Next, we will migrate the virtual machine networking from standard switch to the vDS. Go back to Home and select Networking. Right click the respective distributed switch and select Migrate Virtual Machine Networking.

Step 15:

The source network is your standard switch networking and from the drop-down select the port-group on the standard switch where the virtual machines reside. In my case, the port-group on the standard switch is called, VM Network. The destination network port-group is on the vDS and I want to migrate the VMs to a port-group called dvPortGroup on the vDS.

Step 16:

Select the virtual machine you want to migrate on this port-group in the next section.

Review changes and finish the migration. Once Migrated, go back to your distributed switch under the ESXi host and cluster section and review the final configuration.

That's pretty much it. If you have additional portgroups you will have to repeat the process. If your port-group have VLAN IDs, then you will have to create a port-group on the vDS with the same VLAN ID, else the migration will fail.
If you are migrating iSCSI with port binding, then you will have to remove the port binding and then migrate the iSCSI and then configure port binding post migration.

Click this link here to see how to migrate from vDS to standard switch.

↧

Migrate Networking From Distributed Switch To Standard Switch

April 2, 2016, 3:48 am

≫ Next: Unable To Register VDP to vCenter in the vdp-configure Page

≪ Previous: Migrate Networking From Standard Switch To Distributed Switch

Written by Suhas Savkoor

In the previous article here, we saw how to migrate ESXi networking from Standard Switch to Distributed Switch. In this one, we will perform the reverse of this.

Step 1:
This is the setup that I have for my vDS after I had it migrated.

Here I have 2 portgroups, one for my virtual machine and one for my vmk management port-group. And both of these are connected to two uplinks, vmnic0 and vmnic1

Step 2:
Before creating a standard switch, I will be removing one of the vmnic (Physical Adapter) from the vDS as I do not have any free uplinks to add to the standard switch. Select Manage Physical Adapters and Remove the required uplink.

Step 3:
Now let's go ahead and create a new Standard Switch. Select the vSphere Standard Switch and click Add Networking

Step 4:
Choose Virtual Machine as the port-group type.

Step 5:
Select the available uplink that needs to be connected to this standard switch and click Next

Step 6:
Provide a Network Label to the virtual machine port-group on the standard switch.

Review the settings and complete the create and now you will have one Standard Switch with one virtual machine port-group connected to an uplink. It's now time to begin the migration.

Step 7:
Go back to distributed switches section and select Manage Virtual Adapters

Step 8:
Select the required vmk and click Migrate

Step 9:
Select the required vSwitch as to where you want to migrate this port-group to.

Step 10:
Provide a Network label for this vmk port-group on the standard switch. If you are using any VLAN for the vDS port-group for this vmk, specify the same in the VLAN section to replicate this on the standard switch. Else the migration fails.

Review and complete and you have the management vmk migrated off the distributed switch to the standard switch.

Step 11:
To migrate virtual machine's networking, go to Home > Networking > Right click the vDS and select Migrate VirtualMachine Networking

Step 12:
The source would be the VM port-group on the vDS in my case is, dvPortGroup ad the destination is the standard switch port-group which we created recently, VSS VM Portgroup

Step 13:
Select the virtual machines that you want to migrate.

Review and finish and once the migrate completes, you can now check the standard switch configuration to verify everything is migrated successfully.

Well, that's it!

↧

Unable To Register VDP to vCenter in the vdp-configure Page

April 4, 2016, 10:20 am

≫ Next: Cannot Open vdp-configure Page Or Check Status of VDP Services

≪ Previous: Migrate Networking From Distributed Switch To Standard Switch

Written by Suhas Savkoor

After deploying the OVF template for a new VDP appliance, we will have to go to the vdp-configure page to get the appliance configured to the vCenter.

Here, in the vCenter registration page, after entering the username, vCenter details and try to test connection, you run into the below error.

"Unable to verify vCenter listening on specified HTTP Port. Please re-check values and try again"

So, here I was trying to configure VDP appliance with a port number of 80 for http and 443 for https.

However, the vCenter is running on a custom port of 81 and 444.

You can login to your vCenter, 6.0, Select the Administration tab > vCenter Server Settings > Advanced Settings.

Here there are two parameters which talks about your vCenter ports. They are:

config.vpxd.rhttpproxy.httpsport 443
config.vpxd.rhttpproxy.httpport 80

443 and 80 are the default ports. If they are different, then we are using custom ports and we need top open this port on the VDP appliance firewall.

You can use telnet to check the connection between the appliance and vCenter.
Run the command from the SSH of the appliance

telnet <server_IP> <port_number>

To perform this:
1. Open a SSH to the VDP appliance.
2. Change your directory to:

#: cd /etc/

3. Open the file "firewall.base" in a vi editor
4. Locate the line:

exec_rule -A OUTPUT -p tcp -m multiport --dport 53,80

5. Add your custom http and https port value here and save the file.
6. Restart the firewall service using the following commands:

#: service avfirewall stop

#: service avfirewall start

Register the appliance again and make sure you give the custom ports in the http and https field during configuration.

That's it!

↧

Cannot Open vdp-configure Page Or Check Status of VDP Services

April 5, 2016, 2:22 pm

≫ Next: Re-register VDP 6.1 to vCenter Server

≪ Previous: Unable To Register VDP to vCenter in the vdp-configure Page

Written by Suhas Savkoor

So when you try to open the :8543/vdp-configure page you will receive the message:

"This site can't be reached. ERR_CONNECTION_REFUSED"

When you open SSH to this VDP appliance and check the status of the services, this will also throw up an error. Run the below command to check the VDP service status:

root@vdp:~/#: dpnctl status

The error you will receive is:

mkdtemp: private socket dir: No space left on device.

I tried to run the command to start the webservices, which is:

root@vdp:~/#: emwebapp.sh --start

Which also failed with an error:

Waiting for postmaster to start ...........Failed to connect DBI:Pg:dbname=postgres;port=5558.ERROR: Failed to start the database.

Interesting!
I ran the below command to check the space on the VDP appliance partition.

root@vdp:~/#: df -h

Here the output I noticed the partition, /dev/sda2 was at 100 percent used.

Run the below command to list directories within each other with largest used space:

root@vdp:~/#: du -h --max-depth=1 <directory>

Upon performing this, I found the below directory occupying nearly 40 percent of space on sda2

root@vdp:~/#: /usr/local/avamar-tomcat-7.0.42/logs

Removed all the old log files from this directory.

Also, if the space does not change even when the logs from the above directory is removed, then you need to check the following directory:

root@vdp:/usr/local/avamar-tomcat-<your_version>/webapps/ROOT

You will see a logbundle.zip file which is a manually generated log file. You can go ahead and remove this log.zip file. Do not remove any other file in this directory.

Prior to removing the logs, stop the VDP services using the below command:

root@vdp:~/#: dpnctl stop

However, this command also failed due to unavailable space. If this occurs, go ahead and remove the files without stopping the service. I risked this, however, the log files cleared out and space was freed and I was able to start the web services for VDP and login to the GUI of the appliance.

Cheers!

↧

Re-register VDP 6.1 to vCenter Server

April 7, 2016, 7:13 am

≫ Next: VDP CLI Commands

≪ Previous: Cannot Open vdp-configure Page Or Check Status of VDP Services

Written by Suhas Savkoor

Sometimes, there might be a need to re-register your VDP appliance to your vCenter server, maybe to use a different user account for registration or some issues with the vCenter. The registration process is quite easy and will not affect any of your backup jobs or the backup data present in your deduplication store.

To Re-register your VDP appliance to vCenter, follow the below steps:

From the below screenshot you can see the backup job that is already present on my appliance prior to the re-registration.

Next, you need to go to your vdp-configure page, which is available at the below link.

https://VDP_IP/vdp-configure

Login to your appliance with your root credentials and you will come across the below page. Click the gear icon and select vCenter registration.

Please read the below message. Do not make any changes to vCenter with regards to vCenter host-name, IP, port number. This will cause your backup jobs to be lost.

However, re-registering with a different user should not cause any issues.

Provide the new user details and keep your vCenter details the same. Click Next, review the changes and click Finish.

The below task will be started during the re-registration process. Once the task is done, it will reconnect to web client and log you out of your vdp-configure session.

Now, login back to your web client, go to vSphere Data Protection. Connect the required appliance to web client and go to the backup tab and you will notice your backup job is still retained.

That's it!

↧

VDP CLI Commands

April 7, 2016, 11:45 am

≫ Next: Cannot Connect VDP To Web Client. SSO Server Could Not Be Found

≪ Previous: Re-register VDP 6.1 to vCenter Server

Written by Suhas Savkoor

Here is a set of commands for various operations that you can perform from the SSH of the VDP appliance. I will update this article as I come across new commands.

dpnctl status all

//Shows output of all the services of the VDP appliance.

//Sample output:

dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: emt status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: axionfs status: down.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
dpnctl: INFO: avinstaller status: up.
dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

status.dpn

//Shows appliance access, status, last checkpoint, last hfs check and deduplication space reports.

//Sample output

Thu Apr 7 20:02:17 IST 2016 [vdp.vcloud.local] Thu Apr 7 14:32:17 2016 UTC (Initialized Mon Apr 4 18:47:03 2016 UTC)
Node IP Address Version State Runlevel Srvr+Root+User Dis Suspend Load UsedMB Errlen %Full Percent Full and Stripe Status by Disk
0.0 192.168.1.206 7.2.80-94 ONLINE fullaccess mhpu+0hpu+0hpu 1 false 0.38 3420 102774 0.6% 0%(onl:13 ) 0%(onl:12 ) 0%(onl:12 )
Srvr+Root+User Modes = migrate + hfswriteable + persistwriteable + useraccntwriteable

System ID: 1459795623@00:50:56:B9:6B:3C

All reported states=(ONLINE), runlevels=(fullaccess), modes=(mhpu+0hpu+0hpu)
System-Status: ok
Access-Status: full

Last checkpoint: cp.20160407033314 finished Thu Apr 7 09:03:33 2016 after 00m 19s (OK)
Last GC: finished Thu Apr 7 08:00:39 2016 after 00m 16s >> recovered 64.05 MB (OK)
Last hfscheck: finished Thu Apr 7 09:03:05 2016 after 02m 01s >> checked 32 of 32 stripes (OK)

Maintenance windows scheduler capacity profile is active.
The backup window is currently running.
Next backup window start time: Fri Apr 8 20:00:00 2016 IST
Next maintenance window start time: Fri Apr 8 08:00:00 2016 IST

capacity.sh

//Shows change of data for the list of backed up VMs and a list of top 3 high change VMs in backup jobs.

//Sample output

Date New Data #BU Removed #GC Net Change
---------- ---------- ----- ---------- ----- ----------
2016-04-04 1047 mb 1 0 mb 1047 mb
2016-04-05 3311 mb 4 0 mb 1 3311 mb
---------- ---------- ----- ---------- ----- ----------
Average 2179 mb 0 mb 2179 mb

Top 3 High Change Clients:
--------------------------
Total for all clients 4358 mb 100.0%
Windows_2008_UDmaxWmHw0pC0PB5bXgAKg 3306 mb 75.8%
CentOS7_UDlmW0iZnAAWleQvAdTtog 1052 mb 24.2%

mccli activity show

//Shows status of running backup jobs.

//Sample output

ID Status Error Code Start Time Elapsed End Time Type Progress Bytes New Bytes Client Domain
---------------- ------- ---------- -------------------- ----------- -------------------- ---------------- -------------- --------- ------------ ----------------------------
9146003975305509 Running 0 2016-04-07 20:05 IST 00h:00m:16s 2016-04-08 20:05 IST On-Demand Backup 0 bytes 0% Windows 2008 /192.168.1.1/VirtualMachines

mccli activity get-log --id=XXXXXXXXXX

//Shows the logs for the backup job for the backup ID. Backup ID can be found from the mccli activity show

mccli activity cancel --id=XXXXXXXXXX

//To cancel the actively running backup job.

//Sample output

0,22205,Backup cancelled via console
Attribute Value
----------- ----------------
activity-id 9146003975305509

mccli client show --recursive=true

//Show the registered clients with the VDP.

//Sample output
Client Domain Client Type
---------------- ---------------------------- ------------------------------------
Windows 2008 /192.168.1.1/VirtualMachines Virtual Machine
192.168.1.1 /192.168.1.1 vCenter
vdp.vcloud.local /clients VMware Image Proxy with Guest Backup

mccli backup show --name=/vCenter/VirtualMachines/VirtualMachine_Name --recursive=true

//Shows the successful backup and the restore point created for the same.

//Sample Command

 mccli backup show --name=/192.168.1.1/VirtualMachines/CentOS7 --recursive=true

//Sample Output
Created LabelNum Size Retention Hostname Location
----------------------- -------- ------- --------- ---------------- --------
2016-04-07 20:31:34 IST 1 16.0 GB DWMY vdp.vcloud.local Local

mccli client backup-dataset --name=VM_NAME --domain=/VM/DOMAIN

//To start backup of a VM from command line.

//Sample Command:

mccli client backup-dataset --name="Windows 2008" --domain=/192.168.1.1/VirtualMachines

//Sample output

Attribute Value
----------- -----------------------------------------
client /192.168.1.1/VirtualMachines/Windows 2008
activity-id 9146004043408409
dataset /VMware Image Dataset

cplist

//Shows the checkpoint for the appliance. Valid and Not validated checkpoint

//Sample output

cp.20160407033040 Thu Apr 7 09:00:40 2016 valid rol --- nodes 1/1 stripes 37
cp.20160407033314 Thu Apr 7 09:03:14 2016 valid --- --- nodes 1/1 stripes 37

mccli server show-prop

//Shows VDP details. Useful if you cannot connect to VDP from the web client

//Sample output

Attribute Value
-------------------------------- ----------------------------
State Full Access
Active sessions 1
Total capacity 575.9 GB
Capacity used 5.3 GB
Server utilization 0.9%
Bytes protected 0 bytes
Bytes protected quota Not configured
License expiration Never
Time since Server initialization 2 days 20h:04m
Last checkpoint 2016-04-07 09:03:14 IST
Last validated checkpoint 2016-04-07 09:00:40 IST
System Name vdp.vcloud.local

mccli server show-services

//Another command to check service status

//Sample output

Name Status
---------------------------------- ---------------------------
Hostname vdp.vcloud.local
IP Address 192.168.1.206
Load Average 0.31
Last Administrator Datastore Flush 2016-04-07 19:45:00 IST
PostgreSQL database Running
Web Services Error
Web Restore Disk Space Available 266,182,888K
Login Manager Running
snmp sub-agent Disabled
ConnectEMC Disabled
snmp daemon Disabled
ssh daemon Running
Data Domain SNMP Manager Not Running
Remote Backup Manager Service Running
RabbitMQ Not Running
Replication cron job Not Running
/192.168.1.1 All vCenter connections OK.

emwebapp.sh --restart / --stop / --start

//Command to restart tomcat service for VDP. If this service is stopped you cannot connect VDP to Web Client or access VDP GUI page on 8543.

//Sample successful start

INFO: Copying CST libs to tomcat
Administrator Server is not running.
Starting the database server.
Waiting for postmaster to start ...Started
No script specified

Database server is still running...
Starting tomcat
/usr/local/avamar-tomcat/bin/startup.sh
Using CATALINA_BASE: /usr/local/avamar-tomcat
Using CATALINA_HOME: /usr/local/avamar-tomcat
Using CATALINA_TMPDIR: /usr/local/avamar-tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/avamar-tomcat/bin/bootstrap.jar:/usr/local/avamar-tomcat/bin/tomcat-juli.jar
Using CATALINA_PID: /usr/local/avamar-tomcat/avamar-tomcat.pid
Tomcat started.

dpnctl start gsan / mcs / axionfs

//Start each of the services individually

avmaint hfscheck --ava --full

//To run a full integrity check on your VDP appliance

//Sample successful output

<hfscheck
checkpoint="cp.20160407033314"
status="waitcgsan"
type="full"
checks="full"
elapsed-time="70"
start-time="1460064369"
end-time="0"
check-start-time="0"
check-end-time="0"
generation-time="1460064439"
percent-complete="0.00">
<hfscheckerrors/>
</hfscheck>

avmaint hfscheckstatus --ava

//To check hfscheck status. This tells the last cp that was created during the integrity check and the status.

//Sample output

checkpoint="cp.20160407033314"
status="completed"

↧

Cannot Connect VDP To Web Client. SSO Server Could Not Be Found

April 8, 2016, 1:15 am

≫ Next: Backup VDP Data To An External Drive

≪ Previous: VDP CLI Commands

Written by Suhas Savkoor

You have setup a new VDP appliance in your environment or have an existing one. And suddenly you run into issues regarding connecting this appliance to the Web Client. Every time you choose the appliance from the drop-down and hit connect, you run into the error:

"Could not connect to the requested VDP appliance. The SSO Server could not be found. Please make sure the SSO and DNS servers are available on the network and all settings are configured properly"

Pre-checks and solutions:

1. DNS should be working good. Check if you are able to resolve the IP and FQDN of the VDP appliance along with your vCenter server from the machine where you are trying to connect. If DNS resolution is running into issues, then have the DNS issue fixed, and then try the reconnect.

2. Time on the VDP appliance to ESXi host to your SSO server must be synced. If time is not synced then your will run into NTP error message during the connect. However, please make sure that the time is synced across all the three components. Use the command "date" in VDP to list the current time.

3. You can try adding the VDP appliance and vCenter IP/FQDN entry in the /etc/hosts file of the VDP appliance and then try a reconnect.

For me, all of the above pointers were working good, and yet I was unable to connect the appliance to the Web Client.

Restarted the tomcat service on the VDP appliance and I was able to successfully connect the appliance to the web client. To restart this:

#: emwebapp.sh --restart

That's pretty much it!

You can refer this blog here for more information and this KB here for further troubleshooting if these steps do not help.

↧

Backup VDP Data To An External Drive

April 9, 2016, 5:56 am

≫ Next: VDP 6.1.2 Plugin Not Available After A Fresh Deployment

≪ Previous: Cannot Connect VDP To Web Client. SSO Server Could Not Be Found

Written by Suhas Savkoor

Well, you have got your new vSphere Data Protection set up and have kicked off a few successful backups. The backup data is stored in your VDP datastore safely and you can restore it any time you want to. Let's say that your VMFS volume hosting your VDP data drives went corrupt, this means that all your precious backed up data is lost. So most of the enterprises look for means to move those backup disk of VDP to an external drive.

Now, there are two ways of achieving this.

1. Using a script to make a copy of your data disks of VDP to a tape drive or your local machine. The disadvantage of this is that you will have to power off your backup appliance in order for the disks to be copied over. So, if you have one VDP appliance in your environment, your backup is down until the copy completes.

2. Deploy another VDP appliance and configure replication between the primary and the secondary appliance. The primary is the one that performs your scheduled backups and then replicates this data to the secondary appliance. So, daily backups occur on the primary machine. Replication occurs on the weekends to the secondary appliance. The script runs on the secondary appliance for the rest of the week, as the secondary is not receiving any replication traffic and neither it is running any backup jobs, it can afford a downtime.

The script:

#Connects to the vCenter Server hosting the backup appliance

Connect-VIServer -Server <vCenter_IP> -User administrator -Password <Password>

#Shuts down the VDP appliance

Shutdown-VMGuest -VM "<VDP_Appliance_Name>" -Confirm:$false

#Sleep time after which the copy script kicks in. Value is in seconds

Start-Sleep -s 600

#Copy script

Copy-DataStoreItem vmstore:\<Data_centername>\<VDP_Datastore_Name>\"<VDP_Appliance_Name> 6.1"\*.vmdk D:\VDP\vdp01 -Force

#Starts the appliance after the copy is done

Start-VM -VM "vSphere Data Protection 6.1" -Confirm:$false

Here, I am copying the VDP vmdk only files to D drive on my local computer. If you want to copy the remaining files, then simply replicate the #Copy script lines and change *.vmdk to the other file extensions (*.vmx, *.nvram etc)

Sample output during the Power Off:

Sample output during the copy:

Once the copy completes, you can verify the contents of the VM in the local drive/tape connected to your machine.

**Try at your own risk if you are running a production environment. All the tests for the script was done in my lab**

↧

VDP 6.1.2 Plugin Not Available After A Fresh Deployment

April 13, 2016, 1:15 pm

≫ Next: Changing DNS for VDP 6.1.2 post deployment.

≪ Previous: Backup VDP Data To An External Drive

Written by Suhas Savkoor

So with 6.1.2, there is an issue seen with most of the deployments. Once the new VDP appliance is deployed we do not see the respective plugin in the web client. These are the observations that I came across during troubleshooting:

Couple of things to note:

1. vCenter is an appliance: Version 6.0 U2

2. VDP is 6.1.2

3. Tested with embedded and external PSC deployment

4. Tested with standalone and linked mode deployment.

Web Client logs:

Location: /var/log/vmware/vsphere-client/logs

Entry:

[2016-04-06T19:27:13.272Z] [ERROR] -extensionmanager-pool-98735 70049419 100449 200340 com.vmware.vise.vim.extension.VcExtensionManager Package com.vmware.vdp2 was not installed! Error expanding zip file found at https://10.22.132.174:8543/vdp-plugin-package.zip. Make sure it is a valid zip file then logout/login to force another download. java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(Unknown Source) at java.util.zip.ZipFile.<init>(Unknown Source) at java.util.zip.ZipFile.<init>(Unknown Source) at com.vmware.vise.vim.extension.VcExtensionManager.downloadPackage(VcExtensionManager.java:856) at com.vmware.vise.vim.extension.VcExtensionManager$1.call(VcExtensionManager.java:671) at com.vmware.vise.vim.extension.VcExtensionManager$1.call(VcExtensionManager.java:663) at java.util.concurrent.FutureTask.run(Unknown Source) at com.vmware.vise.util.concurrent.QueuingCachedThreadPool$QueueProcessor.run(QueuingCachedThreadPool.java:866) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

So here it is referencing that the com.vmware.vdp2 package was not installed as it was failed during the unzip operation.

MOB entry for VDP plugins:

Location: https://vCenter_IP/mob

Plugin report:

com.vmware.vdp2

com.vmware.vdp2.config

Both the required plugin for 6.x are present. There is no requirement to invoke them.

vCenter Serenity Report:

Location: /etc/vmware/vsphere-client/vc-packages/vsphere-client-serenity

This folder was empty and there was no reference of the VDP plugin here.

What has to be done next:

1. Go to the following URL

https://<VDP_IP>:8543/vdp-plugin-package.zip

2. This is going to download a plugin.zip folder. Extract this and you will get two files.

3. In the vsphere-client-serenity, create a folder by the name com.vmware.vdp2-<vdp_version> and pasted these two files there.

**Now I have seen a case where the "vc-packages" and "vsphere-client-serenity" is not available at all. In this case, you can manually create these two directories**

4. Restarted web client service

# service-control --stop vsphere-client

# service-control --start vsphere-client

5. Logged into web client.

The plugin was available and connected successfully.

I have also come across a situation where once the plugin is available and clicking the Connect option, the screen forever stays grayed out. I am currently testing this out more on my 6.0 U2 and will update a new article soon.

Update 15-April-2016

If your VDP appliance is residing on a distributed switch, then the web client screen always stays grayed out when you click the "Connect" button.

Migrate the VDP appliance to standard switch and the appliance connects instantaneously.

Update 16-April-2016

If the VMs are residing on distributed switch, then the web client screen grays out when submitting a new backup job.

Workaround: Migrate VMs to standard switch until a fix is released or use a lower version of VDP.

Update 26-April-2016

Hot fix is available in this KB article here.

↧

Changing DNS for VDP 6.1.2 post deployment.

April 19, 2016, 12:43 pm

≫ Next: Cannot Perform A Backup With VDP 5.1

≪ Previous: VDP 6.1.2 Plugin Not Available After A Fresh Deployment

Written by Suhas Savkoor

Short article for a quick reference to update the DNS name for VDP appliance. While working on a case today I had to change the DNS name from vdp.vapp.local to vdpnew.vapp.local

From the below screenshots, you can see that the "hostname" from VDP appliance console shows the original DNS for the appliance.

The ping to the DNS resolves to the current IP of the appliance.

Now, to update this record:

1. Update the DNS record from your DNS manager.

2. Now login to the vdp configure page which can be found at the below location:

https://vdp-ip:8543/vdp-configure

3. Here under the host-name section we will see the old VDP appliance DNS entry. This has to be updated. Click the gear icon and select Network Settings.

4. You will come across the above screen and update the host-name section with the new host-name that was given for the appliance and click Next and Finish.

5. Restart the Guest for VDP appliance and we should be good to go from there.

You can now see the updated DNS for VDP.

That's it!

↧

Cannot Perform A Backup With VDP 5.1

April 22, 2016, 12:14 am

≫ Next: VDP Deduplication Process

≪ Previous: Changing DNS for VDP 6.1.2 post deployment.

Written by Suhas Savkoor

So while performing a backup of any virtual machines in my environment on VDP, the backup task used to fail. So, I created one backup job, with one virtual machine in it and ran the backup task. The create snapshot task completes successfully and you can see the VDP snapshot in the snapshot manager of the virtual machine. Once the snapshot is taken, the next task initiated will be the VDP backup task, and this fails immediately (no percentage completed). It states it failed with miscellaneous errors.

When you open a SSH to the VDP appliance and browse the backup-job log, from the directory, /usr/local/avamarclient/var-proxy-N, I see the following logging:

2016-04-20 08:28:50 avvcbimage FATAL <5889>: Fatal signal 6 in pid 553

2016/04/20-15:28:50.11148 [] FATAL ERROR: <0001> Fatal signal 6

2016/04/20-15:28:50.11161 [] | 00000000005f39f1

2016/04/20-15:28:50.11162 [] | 00000000005f46b7

2016/04/20-15:28:50.11163 [] | 00000000005f5bdb

2016/04/20-15:28:50.11164 [] | 00000000005f5cce

2016/04/20-15:28:50.11168 [] | 000000000058b800

2016/04/20-15:28:50.11169 [] | 00007ff23c7c16a0

2016/04/20-15:28:50.11170 [] | 00007ff23b0797a9

2016/04/20-15:28:50.11170 [] | 00007ff2308a8900

2016/04/20-15:28:50.11171 [] | 00007ff23087b9ac

2016/04/20-15:28:50.11172 [] | 00007ff2306bf442

2016/04/20-15:28:50.11173 [] | 00007ff2362c472e

2016/04/20-15:28:50.11174 [] | 00007ff2362cff4c

2016/04/20-15:28:50.11174 [] | 00007ff230898d17

2016/04/20-15:28:50.11175 [] | 00007ff230894883

2016/04/20-15:28:50.11176 [] | 00007ff23c7b9696

2016/04/20-15:28:50.11177 [] | 00007ff23b07cd7d

2016/04/20-15:28:50.11184 [] ERROR: <0316> handlefatal exiting thread pid=553, sig=6

2016-04-20 08:28:50 avvcbimage Error <5891>: handlefatal: exiting thread pid=553, sig=6

2016-04-20 08:28:50 avvcbimage Info <16041>: VDDK:2016-04-20T08:28:50.113-07:00 [7FF22EE0D700 panic 'Default']

2016-04-20 08:28:50 avvcbimage Info <16041>: VDDK:-->

2016-04-20 08:28:50 avvcbimage Info <16041>: VDDK:--> Panic: Assert Failed: "_lockToken != __null" @ /build/mts/release/bora-774844/bora/lib/vcbLib/hotAdd.cpp:638

Here the appliance is failing to lock the VMDK which it has to backup and entering a panic state causing the backup to fail.

The 5.1 VDP has a VDDK version of 5.1. VDDK is a set of C++ libraries that is used to access and create virtual disk. The version of VDDK can be found out from the backup job log (Just search for VDDK and the first few lines will define the version)

You will see the entry for VDDK version as follow (This was on my 6.1.2 VDP)

2016-04-18T20:00:15.978+07:00 avvcbimage Info <16041>: VDDK:VMware VixDiskLib (6.0) Release build-2498720

Now, back to the failed backup. This is a known issue with 5.1 VDDK version and this can be verified from the 5.5.1 (5.5 Update 1) VDDK release notes. (Search for the section: Assert failure causes exit when HotAdd cannot acquire lock)

Upgrade your VDP to 5.8 and we should be good to go as it updates the underlying VDDK version. Re-run a backup after the upgrade and check the backup logs again and you can see the new version of VDDK release and it's corresponding build number.

↧

VDP Deduplication Process

April 30, 2016, 4:19 am

≫ Next: VDP Hot-Add vs NBD

≪ Previous: Cannot Perform A Backup With VDP 5.1

Written by Suhas Savkoor

Saving the same data after every iteration of backup is not ideal because the space consumed on you storage increases rapidly. To provide a better storage for backups deduplication technology is used. What this does is, during the first initial full back, the entire contents of the virtual machine is backed up. However, the subsequent backups only save the new data or the changes that has occurred when compared to the previous iteration of backup. This is called as incremental backup. The changed data will be processed by VDP and saved where as pointer files are created to the same/unchanged data that was present in the previous backup. This saves the storage space and also increases the backup efficiency.

Before we jump deeper into deduplication, let us have a look at the two types of deduplication we have at hand; Fixed Length (Also called as Fixed block) and Variable length (Variable block) deduplication.

This is the raw data that I have at hand right now:

"Welcome to virtuallypeculiar read abot vmware technology"

Fixed length deduplication: I am going to segment this raw data into a data-set defined by a block length of 8. Which is, 8 characters per data-set. The output will look something as:

Variable length deduplication: In this, we do not have a constant set of deduplication block length. The algorithm is going to look at the data set and set logical boundaries for deduplication length. The output will something as:

Now, on a high level basis, my backup software took a backup of the raw data which is saved on my notepad file. Since this is a first backup, the entire text data is saved on the storage using a deduplication technology.

Next, in the raw data I have a spelling error for the word abot (about), upon noticing this, I will re-open the notepad make the necessary changes and save the file again. When the next iteration of backup runs, it is going to scan for changes in blocks.

How fixed length deduplication deals with this?

When the new character is added, the data bits are shifted towards the right by 1. The output would be something as:

Now, in these cases there are scenarios where the shifting of data bits causes the shifted data to enter a new 8-character data-set which creates a new storage block to be occupied, just for one character. This reduces the storage efficiency when compared to variable length deduplication.

How variable length deduplication deals with this?

When the changed data is detected the variable length deduplication makes sure that the outcome of the changed data set matches the chunk size or data set size of the previous backup iteration. The output is something as:

Here the red box shows the changed data, and it is seen that it is limited to the same block whereas in fixed length it was seen till the end of the data set.
VDP is based on variable length deduplication, and using an algorithm the logical boundaries are set for the raw data.

Final note, variable length deduplication provides better storage density than fixed length as the changes in data-set is not vast.

How does VDP deduplication work?

Now, since you have a fair understanding of deduplication, we can look into how VDP handles deduplication. Please note, throughout the process, VDP uses only variable block deduplication.

Have a look at the flow chart below for the basic flow of deduplication process:

Before we get to the working of the flow chart let's have a little understanding regarding the various daemons or processes involved in this backup.

MCS (Management Console Server) This is responsible for the management of all your backup request, VDRDB database.

There are 8 internal proxies on the VDP appliance. Each proxy runs a process called avAgent. These query the MCS every 15 seconds for incoming job requests. Once the avAgent receives the backup request it in turn calls the avVcbImage

The avVcbImage is responsible for enabling, browsing backing up and restoring the disks.

The avVvcbImage in turn calls the avTar which is the primary process for backup and restore.

The entire deduplication process occurs inside the VDP appliance. When the backup request comes in, the first check is done on the client side, where the appliance determines if this virtual machine has been backed up or not. The .ctk file created due to CBT feature when a backup is taken records all the changed sector information since the previous backup. When the appliance scans for this, and if the ctk file determines the changes, only those changed data is sent further to the Sticky Byte Factoring. If older data is present, it is going to create pointer files and will be excluded from Sticky byte factoring.

In Sticky Byte Factoring:

The avTar running here is responsible for breaking down the raw data input that we received earlier into data chunks. The data set that is an outcome of this will be anywhere from 1 KB to 64 KB and will average out on a 24 KB set.

The earlier example we considered for variable block deduplication, let's use that data set, represent that in terms of KB of data and re-review the deduplication process.

So here, in the first full backup, the raw data is divided into variable length blocks using VDP algorithm. It produces a set of data chunks anywhere between 1 and 64K. Now, the data in the first two blocks have changed after the backup was performed. Now, in the next iteration of backup, the sticky byte factoring re-syncs the block so the output of new dataset matches the chunks of the previous dataset. So, no matter where the data has changed the avTar creates chunks to match the previous chunk size.

Compression:

Once the sticky byte factoring divides the raw data into chunks, these will be compressed. The compression ratio will be anywhere between 30 and 50 percent and data that is not favourable for compression will be omitted to prevent performance impact.

Hashing:

The compressed data is then hashed using SHA-1 algorithm. And the hashed data will always output a 20 byte data string. This hash data is unique to each block and serves as a reference for comparison to check if the previous backup has a similar has. If yes, then the similar hashed data are excluded from backup. Hashing does not convert data into hashes, it rather creates hashes for each data block. So at the end of Hashing, you will have your data chunks and hashes corresponding to it. If the hashes are not found in the hash cache, then the cache is updated with the new hash.

The above hashes are called as atomic hashes, further the atomic hashes are combined to form composites. The composites are further combined to form composite hashes. This process is continued until one single root hash is created.
So in the end, we have the actual data stripes. the atomic hashes, the composite hashes and the root hash all stored in their own locations on the VDP storage disks.

Hope this was helpful. If you have any questions, please feel free to reply.

All images are copyrights of virtuallyPeculiar.com

↧

VDP Hot-Add vs NBD

May 2, 2016, 1:22 pm

≫ Next: VDP Backup With External Proxy

≪ Previous: VDP Deduplication Process

Written by Suhas Savkoor

When a backup job is initiated for a virtual machine, there are two methods or protocols by which VDP performs the backup.

The first one we have is the Hot-Add protocol, wherein the virtual disks of the virtual machine being backed up is going to be mounted on to the VDP appliance and then the backup will be performed.

The second one is using Network Block Device (NBD) protocol, where in the virtual disk backup is done via the management network.

In the below screenshot, you can see the VM, New Virtual Machine, resides on Recovery_LUN datastore.

From the next screenshot you can see that the VDP appliance virtual machine is residing on VDP-Datastore.

For the ESXi host where the VDP appliance resides, the storage configuration page shows that it does not have access to the Recovery_LUN datastore. This means the ESXi host hosting the VDP appliance does not see the virtual machine's datastore.

In this scenario, where the ESXi host where the VDP appliance does not see the VMs datastore, Hot-Add protocol is not going to work and it falls back to the NBD protocol.

Here are a few test results while using the NBD protocol:

I have a backup job called "ble" (Please ignore the name) and this contains the virtual machine named, New Virtual Machine.

The next thing I did was, I started a manual backup job for this virtual machine. The backup job as usual initiates two tasks, create a virtual machine snapshot and a VDP: Backup Job.

When you go to Snapshot Manager for the virtual machine being backed up, you can see the snapshot created by the VDP, goes by VDP-EPOCH of backup.

And if I check the Edit Settings of the VDP appliance, I do not see the disk of the New Virtual Machine Hot-Added to the VDP appliance.

So now, the backup traffic will be flowing via the management network. So I performed a little esxtop test to check for this information.

From the SSH session for the host which is hosting the virtual machine being backed up, I saw the following:

The virtual machine and the management network resides on the vSwitch0 with a vmnic2 uplink and we can see that the PKTTX/s is high and rising and dropping accordingly throughout the backup process.

Next, if I SSH to the destination ESXi host where the VDP appliance is running, I see the following:

The VDP appliance is on vSwitch0 and is connected to an uplink vmnic0 and we can see the PKTRX/s is similarly high and is almost at the same rate of PKTTX/s of the virtual machine being backed up.

Next, the observations of Hot-Add:

I am going to migrate this VM to a datastore which is seen by the ESXi host hosting the VDP appliance.

So I migrated the VM to be backed up to the same host as my VDP appliance (As this host does not have any shared datastore) and put it on a different datastore, Suhas-Local-2. From my above screenshots you can see the Suhas-Local-2 is a datastore on my VDP hosting ESXi host.

I then ran the backup job again, and this time, if I come to the Edit Settings of the VDP appliance I see that there is a disk added to the VDP appliance and this corresponds to the virtual disk of the VM being backed up.

Additional information:

If you have virtual flash read cache disks (VFRC) in your environment, then those virtual machines have to be on virtual hardware 10 and above. When your VDP is on a lower version than the VMs with VFRC disks, then the backup protocol is always going to be NBD regardless if the ESXi host can see the datastore or not.

Well, that's pretty much it!

↧

VDP Backup With External Proxy

May 3, 2016, 2:45 pm

≫ Next: Error While Binding iSCSI To VMKernel Adapters: IscsiManager.QueryBoundVnics

≪ Previous: VDP Hot-Add vs NBD

Written by Suhas Savkoor

So in the previous article we saw the types of backup protocol used by VDP. We saw that when the ESXi host which is hosting the VDP appliance does not have the access to the datastore of the VM it is backing up, then the backup protocol used is Network Block Device (NBD). One way to avoid NBD is to bring the VM on to a storage which is seen by the ESXi host hosting the VDP appliance. The other method is to deploy an external proxy for the VDP appliance onto the host which is hosting the client that needs to be backed up. Please note, that once you deploy external proxy, the internal proxy will be disabled automatically.

To add external proxy in detail, you can visit this link here.

From the below screenshot you can see that the external proxy is deployed on the host where the client that requires to be backed up resides.

This will create a virtual machine for the proxy on the specified host.

Now, ideally, when I run the backup for this client without external proxy, it should work with the NBD protocol. However, now the proxy where the avAgent runs, can see the datastore of the client virtual machine, it will proceed with the Hot-Add protocol.

Since avAgent runs on the proxy which talks to the MCS, the disk will not be mounted on the VDP appliance, instead it will be mounted on the external proxy virtual machine. You can see this from the below screenshot.

vdp_ba_vma41 is the proxy virtual machine, and the client being backed up is CentOS7 and we can see the virtual disk of client is mounted on the external proxy virtual machine.

Well, that's one of the use case of deploying external proxy.

↧