Testing Multiple iOS Test Schemes with Fastlane and Generating Test Results
5 min read

Testing Multiple iOS Test Schemes with Fastlane and Generating Test Results


Sometimes, testing on your own mac becomes too extensive when you have hundreds of testing to run on your project.

Something that comes in handy when you use a Continuous Integration(CI) service is Fastlane. Fastlane allows you to save time from writing testing scripts and creates a simpler and more scalable testing process for you.

In this post, we will look at implementing Fastlane and look at some problems that I encountered when integrating Fastlane into a project; mainly **using multiple testing targets and generating a collated test coverage report**.

Fastlane

Fastlane is Continuous Delivery(CD) tool maintained by Google that provides a wide set of tools and customizable features. It can do anything from testing your project, building and signing it, and releasing it on the app store.

It can be integrated into your choice of CI service, such as Jenkins, CircleCI, Travis, Bamboo, GitLab, and so on.

For implementation guide, look at Fastlane's documentations

For the general purpose of this post, we will not discuss the basic integration and usage of Fastlane.

Testing Multiple Schemes

In an iOS project, you may have multiple testing targets associated with one main scheme. You can easily test such project with Fastlane's test action, Scan.

Some problems that arise while using this is that your tests may fail sometimes. Especially if your tests include communicating with the server or waiting for some asynchronous task to finish, tests could frequently exceed the timeout. Because Scan only runs your tests once(just like Xcode), you might see failing tests just because of minor error. But this might not be what you want.

Do you really want to run the test again just because your test decided to lag a second and the whole testing scheme fails? If you want to test safely test fragile tests to remove the pain around your tests, test_center plugin is your solution. This plugin is a set of plugin for Fastlane, and includes multi_scan capability that we needed.

multi_scan

Multi scan supports everything that scan supports, as well as running your tests in batches and re-running tests that may be failing due to fragile test environment. You can eliminate some false-negative test cases by running the failing tests several times. If also provides test completion block to be run after each test batch run.

Installation

To use test_center plugin, create a Pluginfile in your project's fastlane folder and include the following:

gem 'fastlane-plugin-test_center'

That's simple, right? Just remember to include this file in your source control.

Now, you can use the plugin's actions in your Fastfile.

Usage

Here's sample usage of multi_scan

test_run_block = lambda do |testrun_info|
failed_test_count = testrun_info[:failed].size
passed_test_count = testrun_info[:passing].size
try_attempt = testrun_info[:try_count]
batch = testrun_info[:batch]

# UI.abort_with_message!('You could conditionally abort')
UI.message("\ὠA everything is fine, let's continue try #{try_attempt + 1} for batch #{batch}")
end

multi_scan(
project: File.absolute_path('../YOUR_PROJECT.xcodeproj'),
scheme: YOUR_SCHEME,
try_count: 3,
batch_count: 4,
parallel_testrun_count: 4,
testrun_completed_block: test_run_block
)

This code will run your tests up to 3 times, and use 4 batches to run your tests. After completing each testrun, it will run the test_run_block.

Parameters

Here are additional parameters for multi_scan:

Parameter Description Default Value
try_count The number of times to retry running tests via scan 1
batch_count The number of test batches to run through scan. Can be combined with :try_count 1
retry_test_runner_failures Set to true If you want to treat build failures during testing, like 'Test runner exited before starting test execution', as 'all tests failed' false
invocation_based_tests Set to true If your test suit have invocation based tests like Kiwi false
quit_simulators If the simulators need to be killed before running the tests true
collate_reports Whether or not to collate the reports generated by multiple retries, batches, and parallel test runs true
parallel_testrun_count Run simulators each batch of tests and/or each test target in parallel on its own Simulator 1
pre_delete_cloned_simulators Delete left over cloned simulators before running a parallel testrun true
testrun_completed_block A block invoked each time a test run completes. When combined with :parallel_testrun_count, will be called separately in each child process

Test Results

This is my own fix for the problem and may not be the best or correct one. If you find a better solution, please share them! :)
{: .notice--info}

One problem that I encountered while testing with multi_scan was that it was not correctly collating the separate test reports for multiple retries, batches, and parallel test runs.

Even though the collate_reports was set to true, I was getting the test report for the last test run. After digging through some issues in the repo, I found this issue.

Essentially, the each coverage report was being overwritten by the next batch of test. So, to solve this we would have to prevent the coverage reports from being written by renaming or moving them.

This is where multi_scan's testrun_completed_block comes in handy. After each test, we will move the test results to a temporary folder.

In your test lane, declare the following block:

test_run_block = lambda do |testrun_info|
temporary_directory = "#{SAVE_PATH}/Logs"
require "fileutils"
FileUtils.cp_r("#{Scan.config[:derived_data_path]}/Logs/Test", temporary_directory)
end

and add it to multi_scan parameters by adding:

multi_scan(
...
testrun_completed_block: test_run_block,
...
)

After each test run, Xcode stores the test results in Scan.config[:derived_data_path]/Logs/Test.
So, we're going to take all files from the /Logs/Test folder and copy it to your temporary directory. FileUtils.cp_r will copy all contents of the given folder recursively, so you will not miss any important files. However, before running the test, you may need to create the folder by running FileUtils.mkdir_p(temporary_directory).

Now that we've successfully gathered all the test reports, it's time to generate a test coverage report!

Xcov

Xcov is another Fastlane tool that can generate code coverage reports and even post summaries on Slack. Because we saved the test reports in a temporary directory, we have to use additional parameters to generate a coverage report through xcov.

Installation

To use xcov in your Fastlane workflow, include gem 'xcov' in your Gemfile of fastlane folder.

Usage

To use our temporary directory as the source for xcov, we can include the derived_data_path parameter.

Here is a sample code for using xcov:

project: PROJECT_NAME,
scheme: SCHEME,
output_directory: OUTPUT_DIRECTORY,
derived_data_path: TEMPORARY_DIRECTORY
)

Generating a coverage report for multiple test result files

Okay, now that we've provided the correct files for xcov to parse and generate result from, everything should work corectly, right?

Not quite.

After looking through xcov's source code and issues, I found that xcov only uses the first test result file from the source folder without merging the sepearte reports. But we need to merge all the separate reports from the different test schemes to get a better picture of our test coverage.

This is doable with xcrun xccov merge command.

xcrun xccov merge --outReport OUTPUT_PATH/out.xccovreport --outArchive OUTPUT_PATH/out.xccovarchive report_0.xccovreport report_0.xccovarchive report_1.xccovreport report_1.xccovarchive ....

I integrated this change in my own PR, so if you'd like to generate a single test coverage report for multiple test schemes, you can add

gem 'xcov', :git => "https://github.com/mininny/xcov.git"

to your Gemfile.

This change is now merged into the main code, so you don't need this extra step! :)

With that, you can use xcov like you normally would, and you will get a collate test coverage report in your output folder.

Sending a Slack Notification 🚀

It's time to get notified of your glorious test success...or failure.

Both multi_scan and xcov supports sending a message to Slack via webhook.

First we need the webhook address of your desired channel. Follow this guide from slack.

With that address, add the following to your multi_scan and xcov parameters.

slack_url: YOUR_WEBHOOK,
slack_message: CUSTOM_MESSAGE

For multi_scan, you can also add

slack_only_on_failure: true,

to only send notification when your test has failed.


There were many new changes to code coverage report in Xcode 11, but we'll look at the changes and the magic that goes behind xcodebuild later.