Web Service Testing with soapUI

For a few years now I have used the open source application soapUI to simplify Web Service testing.  Both to test services that I have written and to test external services that I have had to consume.  In this post I'll cover the basic types of testing you can do with soapUI and provide some practical tips on how to use soapUI when working with Web Services.  If you don't want to go through the detailed examples I suggest jumping straight to the Practical Tips section at the end.

Getting Started

After downloading and installing soapUI the easiest way to get started is to create a new project from an initial WSDL or WADL.  For demonstration purposes I'll use the free Weather Web Service at http://www.webservicex.net/globalweather.asmx.  We will create a test for each operation in this SOAP Web Service.  We will also create a Test Suite, that allows us to run all our tests at the click of a button.  Furthermore, we will create a Load Test for our Test Suite and a Mock Service to simulate the functionality of the Web Service.

To do this go to File and select New soapUI Project.  Populate the New soapUI Project screen in the following manner and click OK:

Basically, paste the URL to the web service WSDL file into the Initial WSDL/WADL field and additionally check the field to generate a MockService.  SoapUI will now load the Web Service definition and ask you questions about how to create the MockService:

For this demonstration change the path to "/WeatherMock" and check the Starts the MockService immediately checkbox, but otherwise accept the defaults.  After clicking OK, specify the name of the Mock Service as "Weather MockService" (or a name of your choice) and click OK again. 

This will complete generating artifacts for the SOAP 1.2 version of the GlobalWeather Web Service.
Next you will be asked the exact same questions, for the SOAP 1.1 version.  Since we don't plan to use SOAP 1.1 you can just hit Cancel.  (I haven't found a way to configure soapUI to only generate artifacts for SOAP 1.2.  If you know how to, please let me know! :)

Running Tests

To execute a given test, drill down to the requests that soapUI auto-generated and insert values that make sense for the given Web Service operation.  In our case, in the Navigator on the left, click on GlobalWeatherSoap12, then GetCitiesByCountry and then double-click on Request 1.
We'll set the Country name as Iceland and then hit the green arrow button to execute the test.

Voila, you should get results back from the Web Service, a list of Icelandic cities.  To check if the Web Service response is valid (conforms to the WSDL), right-click in the results window and click Validate.

Lets give this test the name "City Test".  (Right-click Request 1 and select rename) 
Then do a similar test for the GetWeather operation.  Click on GetWeather and then double-click on Request 1. Put in "Reykjavik" for CityName and "Iceland" for CountryName and run the test.  Also rename the test to "Weather Test".  After running the test, the results should look something like this:

Build a TestSuite

Now that we have created two tests, lets add them to a TestSuite so we can easily re-run them at any time.  In the Navigator on the left, right-click GlobalWeatherSoap12 and select Generate TestSuite.  Select Use existing Requests in interface, and check Generates a default LoadTest for each created TestCase and Single TestCase with one Request for each Operation:  

After clicking OK give the TestSuite the name "Weather TestSuite".   Double-click on GlobalWeatherSoap12 TestSuite under Weather TestSuite to show the TestCase editor.
Then click the green arrow button to run the TestSuite.  If all goes well you should be presented with a results-screen like this, indicating a successful test run:

If the TestSuite run fails you'll see FAILED at the top instead of FINISHED and you should get a message explaining why a given test failed.

Add Assertions

To make the tests more meaningful lets add some assertions to validate the responses from the Web Service we are calling.  You do this by opening up the test Request Editor (double-click on a given test step under TestSuite in the Navigator) and then click the plus-sign next to the green arrow button (second from the left).

For both tests lets add the assertion called SOAP Response, to ensure the Web Service is returning a valid SOAP response.  Then add the assertion called Response SLA and specify the response time as 2000 ms.  That basically means that we are going to consider the test a failure if we don't get a response within two seconds.  Lastly, lets add some content validation by selecting a Contains validation.  For the GetCitiesByCountry operation add the string "Iceland" as the content to expect in the response, and for the GetWeather operation add the string "Success". 

Then run the TestSuite again to make sure you get success-results.

Run a Load Test

When we created the TestSuite in last step, we told soapUI to generate a Load Test as well.  You'll find it under the TestSuite in the Navigator on the left, under the heading Load Test.  The default name is LoadTest 1.  Double click on it in the Navigator to open it up.  Before running the LoadTest you can tweak such parameters as number of concurrent threads to run, length of the test-run, and delay between tests.  Once you have made the desired configurations hit the green arrow button to run the Load Test:

In this sample, 5 concurrent threads are running the TestSuite for 60 seconds with a random wait of up to 1 second between the start of each test.
You can track the progress of the load test with the progress bar in the upper right corner.  If it reaches a 100% without reporting any test errors you are good to go.

Use the Mock Service

Back when we imported the Weather Web Service we told soapUI to generate and start a Mock Service.  That service can now be accessed at http://localhost:8088/WeatherMock.  This is  convenient for example if you are developing against a Web Service that has been designed (WSDL/WADL available) but not yet implemented.  Then you can have the Mock return an actual Web Service response to test your code even though the actual implementation hasn't been completed.  A default response has already been generated (under WeatherMock Service,  GetCitiesByCountry, and Response 1), which you can edit as you like.

You can also have the Mock service return different responses depending on which request it receives.  To demonstrate this, lets create two new MockResponses for the GetWeather operation:

• Click on GetWeather under the Weather MockSerive in the Navigator and select New MockResponse.  
• Give it the name "ReykjavikWeatherResponse".  Accept the automatically generated response, but put the value "Reykjavik" in the GetWeatherResult tag.  (Or even better copy the actual response from calling GetWeather for Reykjavik, which should give you a fully valid response).  
• Create another response called AkureyriWeatherResponse and put the text "Akureyri" in the GetWeatherResult tag.  

Now put logic in the Mock service for when to return each response:

• Double-click on GetWeather in the Navigator to show the MockOperation Editor.  
• Select ReykjavikWeatherResponse under MockResponses.
• Select QUERY_MATCH under Dispatch.
• Click the plus sign to add a new match. 
• Give it the name "Reykjavik". 
• Select Reykjavik and then populate the XPath value with:
  
  declare namespace web='http://www.webserviceX.NET';
  declare namespace soap='http://www.w3.org/2003/05/soap-envelope';
  /soap:Envelope/soap:Body/web:GetWeather/web:CityName
  
  This XPath query will grab the value from the CityName tag.
• Under Expected Value enter "Reykjavik".
• Under Dispatch to, select ReykjavikWeatherResponse.
• Repeat the same steps to create a Match that returns AkureyriWeatherResponse when the city name in the request is Akureyri.

Now you can test your mock Match-logic by opening up your GetWeather test and adding http://localhost:8088/WeatherMock as the endpoint to use (select the current endpoint in the dropdown and pick add new endpoint...).  Then run the test and play around with changing the city name in the request to get different responses from the Mock service.

Additionally, if you want to test your client-side error handling you can have soapUI generate a soap:Fault response and have your Mock return it.  You do that by creating a MockResponse and then clicking the apostrophe icon in the MockResponse Editor.  Then edit the auto generated response as appropriate.

Practical Tips: What to use soapUI for

Now that we have covered the basics of soapUI, here are some practical tips for putting it to use during your software development.

Test Web Services You Have to Consume

When consuming external services, before delving into code, use soapUI to "kick the tires" of the Web Service.  This especially applies when consuming newly written services.  Rather than potentially spending hours pulling your hair over why your client code isn't working, spend a few minutes with soapUI validating and getting familiar with the Web Service you are about to consume. In particular:

• Create and run simple tests for key Web Service operations
• Make sure there are no security/access problems
• Is the web service using some proprietary authentication protocol (NTLMv2 comes to mind) that might give you trouble during implementation?
  
• Have soapUI validate that responses conform to the Web Service contract (WSDL/WADL)
• Visually inspect responses 
• Do they make sense or are they some illegible auto generated garble that should really be cleaned up and restructured by the Web Service developer? 
• Do the responses meet your needs?
• Test a few boundary cases
• Does the Web Service implement proper error handling?  Or does it blow up with an HTTP 500 error or some non SOAP compliant text message?  Does the level of error reporting meet your needs?
• Add all your tests to a TestSuite.  That way you can quickly "ping" the Web Service to make sure everything is working on the other end.  When a problem arises, taking your code out of the loop is a good way to make sure the issue is on the remote end and not with your own code.

When developing against a newly written external Web Service, it is very rare that the Web Service works 100% out of the box as expected.  There is usually a fair amount of communication needed between the Web Service developer and the client-side developer to tweak things until the Web Service works as needed.  By using soapUI, you can inspect the Web Service right at the time it is delivered to you, and quickly spot things that may need to be fixed.  I usually take 5 or 10 minutes to do so and almost always have a list of things that need to be modified.  The response time for getting those changes implemented is usually very short, as the Web Service developer is still engaged in the project and things fresh in his mind.  If I don't get back to him/her until some weeks later when I finally get around to implementing my client-side code, the other developer has probably moved onto other things and/or forgotten why he implemented the service in a certain way.   

Lastly, if you have received a WSDL/WADL file, but the service hasn't actually been implemented, and you NEED to start implementation against the service (not ideal), then consider using soapUI to create a Mock service from the WSDL/WADL file.  That way you can have your code hit the Mock service and at least get some preliminary feedback on whether your client code is working.  Since creating a Mock service is really a breeze with soapUI it can sometimes be more practical than implementing Mock objects in your code.

Test Your Own Web Services

When writing a Web Service for others to consume it can be handy to have a soapUI TestSuite to sanity test your service.  Of course you should still write unit and integration tests for your code, but having a good soapUI TestSuite can be quick and easy way to find out if all your services are running as expected.  When you get that 4 AM phone call saying that something is broke, fire up soapUI and at the click of a button sanity test all of your Web Service operations.  If you are smart, you'll hand the TestSuite over to a support team so that you only get woken when the issue truly is on your end ;-)  Just make sure to add proper assertions for Web Service responses and include SLA assertions to test that things are not running dead slow.

If you are concerned about the performance of your Web Service or whether it can handle a given load, then a soapUI Load Test can be a convenient way to test that.  Set the number of threads to imitate the expected number of concurrent users for your service and add SLA assertions to make sure all requests are handled in a timely fashion.  Of course generating all the load from a single machine does not quite imitate real traffic, so for truer numbers consider having coworkers assist you in running simultaneous Load Tests from multiple machines.