Unit test on python: Coverage report with gitlab ci

About

First, even test has 100% coverage, it does not mean no bugs. However, 100% coverage sounds compelling and reliable than those does not.

This is an article about how to generate coverage reports after testing and hosted them in gitlab

Prerequisites

Which project to work on

As this article follows my another blog, we reuse this project.

Gitlab CI

In case you have no idea about what is Gitlab CI. Please read documentation here for an introduction.

Write yml

Template

There are many templates from Gitlab. I pick this one.

YML

Image

This statement has two outcomes

• Define we need to run in docker
• Define which docker image we need

In our case, we need python:2.7 as pip and python are installed

image: python:2.7

Stages

Although 1 stage is sufficient in our case, let us define 2 stages for demonstration purpose.

Execute order: test -> deploy

stages:
- test
- deploy

Jobs

Define a global setup script for all jobs.

before_script:
    - pip install -r requirements.txt

Install dependencies before running any tests.

Define a job runTest which stop running unit tests as soon as possible if there are failures.

runTest:
    stage: test
    script:
        - nosetests -sx

Define a job coverage which generates a coverage report.

coverage:
    stage: test
    script:
        - nosetests -c .noserc -q --cover-html-dir=build --cover-html
        - coverage report -m
    coverage: '/TOTAL.+ ([0-9]{1,3}%)/'
    artifacts:
        paths:
            - build
        expire_in: 1 day

Notes:

• Jobs in the same stage run in parallel. In order word, failure job cannot stop rest of the jobs at the same stage.
• artifacts allows us to save reports for other jobs later on
• As reports here are temporary, expire_in make sure they will be deleted.
• coverage defines which number means coverage and such number will be presented in the README.md
• For example, below are shell output when running commands for coverage.

$ nosetests -c .noserc -q --cover-html-dir=build --cover-html
.....
$ coverage report -m
Name                   Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------
tests\test_person.py      16      0      0      0   100%
use_mock\person.py         9      0      0      0   100%
------------------------------------------------------------------
TOTAL                     25      0      0      0   100%

Define a job apidoc which generate API doc for this module

apidoc:
    stage: test
    script:
        - cd docs
        - fab make
    artifacts:
        paths:
            - docs/build/html
        expire_in: 1 day

Define a job pages which generate gitlab page

pages:
    stage: deploy
    before_script:
        - "true"
    script:
        - mkdir -p public/coverage
        - cp -fr docs/build/html/* public/
        - cp -fr build/* public/coverage/
    dependencies:
        - coverage
        - apidoc
    artifacts:
        paths:
            - public

Notes

• Belongs to deploy stage instead of test
• pages is a keyword for Gitlab Page
• Files under public are all Gitlab Pages. So, we move reports and docs there.
• dependencies defines we need artifacts from those jobs.
• before_script overrides the global one. It means do nothing here.
• artifacts here saves pages for gitlab server

Debug

• Syntax error

There is an official tool for linting your yml file.

• Run time error

You must read logs from the runner’s report and find out what’s going on.

In our case, we can run a docker container with image docker:2.7 and try commands defined one by one and see whether there are any errors in local environment.

Product

• For apidoc, click here.
• For coverage, click here.
• For badges in README.md, click here

References

• https://docs.gitlab.com/ce/ci/yaml/

​