Skip to content

Testing Milestone

Due Date

Initial due date: Sunday, January 28
Final due date: Same as 1.0 Release
Grading: Team

Description

For this milestone you will implement and run both unit and behavioral tests. You will likely use third-party libraries and tools for creating and running your unit tests. Typically, the official documentation for your platform will have a section on "Testing" which tells you how they recommend you do automated tests. You will also devise a method for automating some of this testing and making it part of your workflow.

The first part of the milestone requires you to research these testing tools, chose some, learn to use them, and create and add to your repo one simple unit and one behavior test to demonstrate your progress.

The second part, due at the end of the semester, requires you to have added multiple tests to your repo. You should add the unit tests as you write the code.

Background

Automated tests are code that you write to test your program. Depending on the situation you might have different sets of tests: some you run before every commit, some before every merge, some before every release, etc.

The goal of the automated tests is to catch bugs early, before they get to the user. When you have a large set of test cases you feel more confident that the change you made to your code will not break something else in the program. The use of tests becomes more and more important as the size of the codebase increases beyond the ability of any one person to understand it all. It is rumored that Oracle database's 25 Million lines of C code only work because they have millions of automated tests that must pass before any new code is merged.

Unit Testing

In unit testing you write tests which verify that your methods/functions do what you think they should do. If a function is meant to sort an array of integers, then you write a unit test that calls that functions with a few example arrays and then check the result to make sure they are sorted. You are a programmer, so you know that it is most likely that errors will appear in boundary conditions, such as when the array is empty, when it has 1 element, when all the elements are the same, when they are in inverse order, etc. So, your examples are informed, not just random.

Unit testing has been around for decades and there are now many Unit testing frameworks for every programming language imaginable. The most popular of these tie in with your IDE to give you pretty output. In most IDEs you can run junit tests and get a special window with red marks for the tests that failed and green marks for those that passed. You will use a unit testing framework in your code.

The unit tests are also part of your repo. Make sure to push them to github.

You will also set it up so that running these tests is a simple matter of just a click, or one command. Everyone should run all unit tests to make sure they pass before they commit their code and push their changes to github.

Behavior Testing

In behavior testing (aka Instrumented testing, UI testing) you write test code that pretends to be the user using your app. Your tests should make sure your app does what the user expects it to do, that is, what your Specifications say. Behavior tests test the whole program.

In order to write code that pretends to be the user you will need to use libraries built for this purpose. Luckily, there are many of them available for all platforms.

For example, if you are building a webapp then you might want to use Selenium which lets you automate web browsing, that is, write a program that emulates a user using a web browser to access your site, do some stuff, get results, and check that those results are what he should be getting. They have the Selenium IDE which lets you write tests by just clicking (if you use it, make sure you explain how to use the command-line runner to run your tests). There are also many other website testing libraries out there, such as casper.js and Nightwatch. Or try Cypress.

All major web frameworks have a section in their manual which gives their recommendations for unit and behavior testing strategies, for example, see Django testing, Angular testing, react testing.

All major mobile platforms have full documentation on how to do testing on their platform, see android testing, iOS XCTest, ionic guide to testing, etc. They all provide libraries which let you write code that pretends to be a user of your app.

There is also Detox for react-native apps.

Unity has its own IntegrationTestsRunner.

Deliverables

For the first deadline:

  1. Have at least one unit and one behavioral test in your repo. Typically in a tests folder.
  2. In your Readme.md file add a Testing section which lists
    1. the commands/instructions one must type to run your tests
    2. the directory (or filename regexep) where all your tests are located.
  3. Add an Issue titled "Testing Video" to your repo and and put in there a video, or more, showing how the tests run.
  4. Once your tests are committed, create a git tag named v0.2 on the commit that marks the release: git tag v0.2
  5. Push tags to GitHub: git push --tags.

For the final deadline:

  1. Multiple unit tests (even if they are silly).
  2. Multiple behavioral tests. Test the core functionality of the app.
  3. In your Readme.md file add a Testing section which lists
    1. the commands/instructions one must type to run your tests
    2. the directory (or filename regexep) where all your tests are located
  4. Add an Issue titled "Testing Video" to your repo and and put in there a video, or more, showing how the tests run.
  5. Optionally, setup a continuous integration service, like travis to automatically run your tests.

Grading Rubric

Grade is 100 minus the total points lost, with minimum of 0. See grading scale in Syllabus.

The first deadline is worth 40% of the grade. Points lost for the first deadline are:

  1. Not enough tests. 0 – 60 points
  2. Tests are just want the IDE/generator produces automatically. 50 points
  3. Bad instructions in Readme.md. 0 – 60 points
  4. No video. 100 points

The final deadline is worth 60% of the grade. Points lost for the final deadline are:

  1. Not enough tests. 0 – 80 points
  2. Bad instructions in Readme.md. 0 – 50 points
  3. No video. 100 points