This is a very quick overview of what we are trying to do. I guess I will write more about this topic in the future ;)
Why a New Testing Framework?
Some time ago, Rene and I hosted a “Test Driven Development for iOS with Swift” workshop, and after that we discussed some differences between JUnit 4 and XCTest, like a few JUnit features we were missing in XCTest. A few days later we saw a great talk about JUnit 5 by Jens Schauder.
We decided that we want to try and bring some JUnit features to Swift. Like dynamic tests, assumptions, test extensions, … To be able to do this, we need a flexible test framework architecture.
We want to support running tests written with different testing technologies (our framework, XCTest, Quick, …). And we want to be able to run them with different runners (XCTest compatible, console, …).
A TestEngine is responsible for knowing all the tests in the project written with a given technology and for knowing how to execute a test. For example, there might be a XCTestEngine, a QuickTestEngine and a SwiftTestEngine. When an engine executes a single test, it reports the result in a technology-agnostic way (Pass/Fail/Inconclusive + Message).
A Runner is responsible for actually running the tests and displaying the results. For example, a XCTestRunner would ask all engines for their tests and create a list of tests that can be understood by XCTest from that. Then, when XCTest tries to execute a test, the runner will actually execute it with the test’s engine, and then translate the result to “XCTFail” if the test failed.
We would probably write our own ConsoleTestRunner and an XCTestRunner which would run the tests in a way that is compatible with the new XCTest API (i.e. passes them to XCTMain). We assume that, with the second runner, we will be compatible with existing IDEs. But the better integration would be when IDEs would implement their own runner. Then they would be immediately compatible with all testing frameworks that provide a TestEngine.
On top of that, we can implement our own test engine, where we can experiment with stuff. Like having more nuanced test results, not only “pass/fail”, which we would need for ignored tests and assumptions.
We want to implement a prototype, of course. But we also want to gather some feedback early, so we will talk to some people who are interested in Testing and/or Swift.
Do you have some feedback for us? What features / capabilities would you like to see in a new testing framework? Please write a short email: Feedback SwiftTest.
P.S.: Do you want to learn and/or practice Test Driven Development in Swift (or any other language)? Have a look at my TDD Basics Training!