Last few weeks I have spent considerable time writing integration test suite for one of our product installers. The goal of this test suite is to automatically test functionality of the product on various operating systems. Currently, we are running our automated test suite on Windows 2008, Windows 7, and Windows 8 systems. In this blog, I will first talk about the technology stack that we used to build our automation framework and then I explain how you can write such test suite using Java installer as an example.
Automation Framework Technology Stack
1. VirtualBox: VirtualBox is a cross platform powerful virtualization tool.
2. Veewee: Veewee is a ruby gem that helps you to easily build repeatable Vagrant base boxes from the command-line. It provides templates for various operating system that you can modify to configure your base box.
3. Vagrant: Vagrant is another Ruby tool that allows you to easily create and manage virtual machines from command-line. You can provision virtual machines using configuration management tools like Chef or Puppet.
4. Ovethere: Overthere is a Java library from XebiaLabs that allows you manipulate files and execute processes on remote hosts. It supports various protocols(local, SSH, CIFS) to communicate with remote machines. Refer to README.md https://github.com/xebialabs/overthere/blob/master/README.md for more information.
5. Overcast: Overcast is another great library from XebiaLabs that allows you test against various hosts. It supports Vagrant hosts, KVM hosts, Amazon EC2, etc. For more information refer to README.md https://github.com/xebialabs/overcast/blob/master/README.markdown.
6. JUnit: JUnit is a Java library that we used to write test cases.
Installing Veewee and Vagrant
Our test automation framework needs veewee and vagrant to work so lets install them before we can start working on our test cases. Both veewee and vagrant requires Ruby programming language. So, before installing them you need to have Ruby installed on your machine. Please install Ruby by following instructions mentioned in official documentation https://www.ruby-lang.org/en/documentation/installation/.
Installing Veewee
To install veewee run the following command.
$ gem install veewee
You might also need to install couple more dependencies depending on your operating system.
$ gem install em-winrm log4r
Installing Vagrant
Refer to the official Vagrant documentation for installation instructions http://www.vagrantup.com/downloads.
Please download package for your operating system.
Creating Base box configuration
Veewee provides a predefined set of templates for various operating systems that you can use to create your base box. To view list of all templates run the following command.
$ veewee vbox templates
It will list down all the available template as shown below. I am only showing part of the output for brevity.
veewee vbox define ‘<box_name>’ ‘windows-2008R1-serverstandard-amd64’ –workdir=/Users/shekhargulati/dev/blog/automation-framework
veewee vbox define ‘<box_name>’ ‘windows-2008R2-amd64’ –workdir=/Users/shekhargulati/dev/blog/automation-framework
veewee vbox define ‘<box_name>’ ‘windows-2008R2-serverstandard-amd64-winrm’ –workdir=/Users/shekhargulati/dev/blog/automation-framework
veewee vbox define ‘<box_name>’ ‘windows-2012-serverstandard-amd64’ –workdir=/Users/shekhargulati/dev/blog/automation-framework
veewee vbox define ‘<box_name>’ ‘windows-2012R2-serverdatacenter-amd64’ –workdir=/Users/shekhargulati/dev/blog/automation-framework
veewee vbox define ‘<box_name>’ ‘windows-7-enterprise-amd64-winrm’ –workdir=/Users/shekhargulati/dev/blog/automation-framework
veewee vbox define ‘<box_name>’ ‘windows-8-amd64’ –workdir=/Users/shekhargulati/dev/blog/automation-framework
veewee vbox define ‘<box_name>’ ‘windows-8-i386’ –workdir=/Users/shekhargulati/dev/blog/automation-framework
veewee vbox define ‘<box_name>’ ‘windows-8-preview-amd64’ –workdir=/Users/shekhargulati/dev/blog/automation-framework
Next choose the basebox definition that you need for your test cases. For this blog, I will go with “`windows-2008R2-serverstandard-amd64-winrm“` configuration. Now navigate to a convinient location where you want to create your base box configuration and run the following command.
$ mkdir windows2008R2 && veewee vbox define ‘windows2008R2’ ‘windows-2008R2-serverstandard-amd64-winrm’ –cwd=windows2008R2
The command shown above will create a new directory **windows2008R2** and place base box configuration in that directory. You can view the base box configuration by opening the **windows2008R2** directory in your favorite editor.
Lets look at the files generated by veewee.
— definitions
— windows2008R2
|– Autounattend.xml
|– README.md
|– definition.rb
|– install-chef.bat
|– install-vbox.bat
|– oracle-cert.cer
— postinstall.sh
2 directories, 7 files
We will discuss two files — definition.rb and Autounattended.xml.
The **definition.rb** is a ruby file that defines from where Veewee should download ISO images. It is also used to configure disk space, RAM, Number of CPUs that this VM should use. This also specifies the winrm user and other configurations for the base box.
**Tip**: If you already have an ISO image then create a directory named **iso** next to **definitions** directory and place your ISO image in that directory. You also have to tell Veewee about your ISO image. In the **definition.rb** file comment out “`iso_md5“` and “`iso_src“` as shown file change the iso_file value to your image name.
:iso_file => “my_image.iso”,
#:iso_md5 => “4263be2cf3c59177c45085c0a7bc6ca5”,
The Autounattended.xml allows you to fully automate Windows installation. It can be used to:
1. Create a single partition for the boot, system, crash dump and primary partition
2. Enter the product key
3. Configure WinRm
4. Open ports so that you can access them
5. Enable the ‘Administrator’ account
6. Set the ‘Administrator’ password to ‘password’
7. Create a local administrator account called ‘UserName’ with the password of ‘password’
In our test cases, we updated WinRm configuration so we modified this file to meet our needs. For example, we updated WinRm “`MaxMemoryPerShellMB“` property from “`300MB“` to “`1024MB“` so we updated “`Autounattend.xml“` file.
<SynchronousCommand wcm:action=”add”>
<CommandLine>cmd.exe /c winrm set winrm/config/winrs @{MaxMemoryPerShellMB=”1024″}</CommandLine>
<Description>Win RM MaxMemoryPerShellMB</Description>
<Order>4</Order>
<RequiresUserInput>true</RequiresUserInput>
</SynchronousCommand>
You can also open ports using this file as shown below.
<SynchronousCommand wcm:action=”add”>
<CommandLine>cmd.exe /c winrm set winrm/config/listener?Address=*+Transport=HTTP @{Port=”8080″} </CommandLine>
<Description>My application port</Description>
<Order>9</Order>
<RequiresUserInput>true</RequiresUserInput>
</SynchronousCommand>
Once you are done with configuration changes, you can create a VirtualBox VM build by executing the command shown below.
$ veewee vbox build ‘windows2008R2’
This will download the ISO image and then create a VirtualBox image for Windows 2008. Once installation is complete, you can export the Vagrant box by running the following command.
$ veewee vbox export ‘windows2008R2’
This will create a new Vagrant box in windows2008R2 directory. The name of the box will be windows2008R2.box.
Importing the Vagrant base box
Now that you have created the Vagrant box, you can add it Vagrant local registry by running the following command.
$ vagrant add windows2008R2 windows2008R2.box
Now you can test your Vagrant box by creating a Vagrantfile and launching the newly created Vagrant box. Go to a convenient location and create a new file with name Vagrantfile as shown below.
$ mkdir vagrant-windows2008R2
$ cd vagrant-windows2008R2
$ touch Vagrantfile
Put the following contents in the Vagrantfile as shown below.
Vagrant.configure(“2”) do |config|
config.vm.box = “windows2008R2”
config.vm.guest = :windows
config.vm.communicator = “winrm”
config.winrm.username = “vagrant”
config.winrm.password = “vagrant”
config.vm.provider “virtualbox” do |v|
v.gui = true
v.cpus = 2
v.memory = 2048
end
end
Fire the new Vagrant box by running the following command.
$ vagrant up
This will launch the Windows 2008 R2 virtual machine and you can use it for your work. To destroy the newly created box, run the following command.
$ vagrant destroy –force
Creating a Java project
Now that all prerequisites are take care off, lets start writing our test cases. Create a new Java Maven project using your favorite IDE or Maven archetype. Replace your application pom.xml with the one shown below:
<?xml version=”1.0″ encoding=”UTF-8″?>
<project xmlns=”http://maven.apache.org/POM/4.0.0″
xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
xsi:schemaLocation=”http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd”>
<modelVersion>4.0.0</modelVersion>
<groupId>org.shekhar</groupId>
<artifactId>installer-tests</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>com.xebialabs.overthere</groupId>
<artifactId>overthere</artifactId>
<version>2.4.5</version>
</dependency>
<dependency>
<groupId>com.xebialabs.cloud</groupId>
<artifactId>overcast</artifactId>
<version>2.4.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Create configuration files
Create a new file with name “`overcast.conf“` in your Java project root directory and populate with following contents.
vagrantHost {
vagrantDir = “vagrant”
vagrantIp = “10.10.200.200”
vagrantVm = “mybox”
}
Also, create a directory called vagrant in the project root directory and inside that create Vagrantfile configuration file as shown below.
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure(“2”) do |config|
config.vm.provider :virtualbox do |vb|
vb.customize [“modifyvm”, :id, “–name”, “mybox”]
end
config.vm.box = “windows2008R2”
config.vm.guest = :windows
config.vm.define “mybox” do |oi|
oi.vm.network :private_network, ip: “10.10.200.200”
end
config.vm.communicator = “winrm”
config.winrm.username = “vagrant”
config.winrm.password = “vagrant”
config.vm.provider “virtualbox” do |v|
v.gui = true
v.cpus = 2
v.memory = 2048
end
end
Write JUnit test case for VM initialization
Create a new JUnit test case in the test directory as shown below.
package org.shekhar.installer.tests;
import com.xebialabs.overcast.host.CloudHost;
import com.xebialabs.overcast.host.CloudHostFactory;
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Test;
import static org.hamcrest.core.Is.is;
import static org.hamcrest.core.IsNull.notNullValue;
import static org.junit.Assert.assertThat;
public class JavaInstallerTest {
private static CloudHost vagrantHost;
@BeforeClass
public static void setup() {
vagrantHost = CloudHostFactory.getCloudHost(“vagrantHost”);
vagrantHost.setup();
}
@AfterClass
public static void teardown() {
vagrantHost.teardown();
}
@Test
public void shouldLaunchVagrantBox() throws Exception {
assertThat(vagrantHost, is(notNullValue()));
}
}
Write Java installation test case
Now we will write a test case that would first copy the Java installer from local machine to the Vagrant box, installs it, and then checks if Java is installed.
package org.shekhar.installer.tests;
import com.xebialabs.overcast.OverthereUtil;
import com.xebialabs.overcast.host.CloudHost;
import com.xebialabs.overcast.host.CloudHostFactory;
import com.xebialabs.overthere.CmdLine;
import com.xebialabs.overthere.OverthereConnection;
import com.xebialabs.overthere.OverthereExecutionOutputHandler;
import com.xebialabs.overthere.local.LocalConnection;
import com.xebialabs.overthere.util.CapturingOverthereExecutionOutputHandler;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.text.MessageFormat;
import java.util.Arrays;
import static com.xebialabs.overcast.OvercastProperties.getOvercastProperty;
import static com.xebialabs.overcast.OverthereUtil.overthereConnectionFromURI;
import static com.xebialabs.overthere.util.CapturingOverthereExecutionOutputHandler.capturingHandler;
import static org.hamcrest.core.Is.is;
import static org.hamcrest.core.IsEqual.equalTo;
import static org.hamcrest.core.IsNull.notNullValue;
import static org.junit.Assert.assertThat;
public class JavaInstallerTest {
private static final String JDK_FILE_SOURCE = “jdk”;
private static final String JDK_DESTINATION_PATH = “c:\\”;
private static final String JDK_EXE_PATH = JDK_DESTINATION_PATH + “jdk-7u75-windows-x64.exe”;
public static final String VAGRANT_HOST = “vagrantHost”;
private static CloudHost vagrantHost;
private static final Logger logger = LoggerFactory.getLogger(JavaInstallerTest.class);
@BeforeClass
public static void setup() {
vagrantHost = CloudHostFactory.getCloudHost(VAGRANT_HOST);
vagrantHost.setup();
}
@AfterClass
public static void teardown() {
vagrantHost.teardown();
}
@Test
public void shouldInstallJDK7() throws Exception {
assertThat(vagrantHost, is(notNullValue()));
String url = makeWinrmConnectionUrl(VAGRANT_HOST);
String hostName = vagrantHost.getHostName();
OverthereConnection dest = getOverthereConnection(url, hostName);
copyFiles(LocalConnection.getLocalConnection(), dest);
CmdLine jdkInstallCommand = new CmdLine().addRaw(String.format(“%s /quiet”, JDK_EXE_PATH));
CapturingOverthereExecutionOutputHandler stdOutCapture = capturingHandler();
CapturingOverthereExecutionOutputHandler stdErrCapture = capturingHandler();
OverthereExecutionOutputHandler stdOutHandler = stdOutCapture;
OverthereExecutionOutputHandler stdErrHandler = stdErrCapture;
int exitCode = dest.execute(stdOutHandler, stdErrHandler, jdkInstallCommand);
assertThat(exitCode, is(equalTo(0)));
CmdLine setJavaInPath = new CmdLine().addRaw(“set PATH=\”c:\\Program Files\\Java\\jdk1.7.0_75\\bin\”;%PATH%”);
exitCode = dest.execute(stdOutHandler, stdErrHandler, setJavaInPath);
assertThat(exitCode, is(equalTo(0)));
}
}
That’s it for today. You can view the full sourcecode of the application on Github https://github.com/shekhargulati/blog-installers-automation-test.
Shekhar Gulati 