Unit test on python: Coverage report with gitlab ci

About

First, even test has 100% coverage, it does not mean no bugs. However, 100% coverage sounds compelling and reliable than those does not.

This is an article about how to generate coverage reports after testing and hosted them in gitlab

Prerequisites

Which project to work on

As this article follows my another blog, we reuse this project.

Gitlab CI

In case you have no idea about what is Gitlab CI. Please read documentation here for an introduction.

Write yml

Template

There are many templates from Gitlab. I pick this one.

YML

Image

This statement has two outcomes

• Define we need to run in docker
• Define which docker image we need

In our case, we need python:2.7 as pip and python are installed

image: python:2.7

Stages

Although 1 stage is sufficient in our case, let us define 2 stages for demonstration purpose.

Execute order: test -> deploy

stages:
- test
- deploy

Jobs

Define a global setup script for all jobs.

before_script:
    - pip install -r requirements.txt

Install dependencies before running any tests.

Define a job runTest which stop running unit tests as soon as possible if there are failures.

runTest:
    stage: test
    script:
        - nosetests -sx

Define a job coverage which generates a coverage report.

coverage:
    stage: test
    script:
        - nosetests -c .noserc -q --cover-html-dir=build --cover-html
        - coverage report -m
    coverage: '/TOTAL.+ ([0-9]{1,3}%)/'
    artifacts:
        paths:
            - build
        expire_in: 1 day

Notes:

• Jobs in the same stage run in parallel. In order word, failure job cannot stop rest of the jobs at the same stage.
• artifacts allows us to save reports for other jobs later on
• As reports here are temporary, expire_in make sure they will be deleted.
• coverage defines which number means coverage and such number will be presented in the README.md
• For example, below are shell output when running commands for coverage.

$ nosetests -c .noserc -q --cover-html-dir=build --cover-html
.....
$ coverage report -m
Name                   Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------
tests\test_person.py      16      0      0      0   100%
use_mock\person.py         9      0      0      0   100%
------------------------------------------------------------------
TOTAL                     25      0      0      0   100%

Define a job apidoc which generate API doc for this module

apidoc:
    stage: test
    script:
        - cd docs
        - fab make
    artifacts:
        paths:
            - docs/build/html
        expire_in: 1 day

Define a job pages which generate gitlab page

pages:
    stage: deploy
    before_script:
        - "true"
    script:
        - mkdir -p public/coverage
        - cp -fr docs/build/html/* public/
        - cp -fr build/* public/coverage/
    dependencies:
        - coverage
        - apidoc
    artifacts:
        paths:
            - public

Notes

• Belongs to deploy stage instead of test
• pages is a keyword for Gitlab Page
• Files under public are all Gitlab Pages. So, we move reports and docs there.
• dependencies defines we need artifacts from those jobs.
• before_script overrides the global one. It means do nothing here.
• artifacts here saves pages for gitlab server

Debug

• Syntax error

There is an official tool for linting your yml file.

• Run time error

You must read logs from the runner’s report and find out what’s going on.

In our case, we can run a docker container with image docker:2.7 and try commands defined one by one and see whether there are any errors in local environment.

Product

• For apidoc, click here.
• For coverage, click here.
• For badges in README.md, click here

References

• https://docs.gitlab.com/ce/ci/yaml/

​

Unit test on python: How to mock out stdout and stderr

About

This article talks about how to mock out stdout and stderr in unit test by the help of python mock.

Prerequisites

pip is the easiest way to install mock.

pip install mock

Source codes can be found here

Basic usage

Trivial class

We need a trivial class for demonstration how to write unit tests with mock.

import sys

class Person(object):
    def speak(self, message):
        """Speak out given message in stdout

                Parameters:

                - `message`: A string going to speak

                Return:

                - `self`: This instance
               """
        sys.stdout.write('%s\n' % message)
        return self

    def yell(self, message):
        """Yell out given message in both stdout and stderr

                Parameters:

                - `message`: A string going to yell

                Return:

                - `self`: This instance
               """
        sys.stdout.write('%s\n' % message)
        sys.stderr.write('%s!\n' % message)
        return self

Then, we would like to test whether speak() and yell() have been print to stdout and stderr.

Test speak()

    @mock.patch('use_mock.person.sys.stdout')
    def test_speak(self, mockStdout):
        self.myPerson.speak('haha')
        mockStdout.write.assert_has_calls([
            mock.call('haha\n')
        ])

Let’s go through this snippet line by line.

@mock.patch('use_mock.person.sys.stdout')

This is a function decorator which mocks out sys.stdout inside module use_mock.person ONLY.

Mock out sys.stdout is critical when writing unit tests as it provides an easy way for us to manipulate outcomes of running them. For example, throwing a system exception like Out Of Memory etc. We can verify our codes can survive any specified strange scenarios as a result.

Back into our case, mocking sys.stdout allows us to capture any arguments passing to them an avoid them to be printed on console.

def test_speak(self, mockStdout):

Prefix test_ is required by the unittest library. Otherwise, it won’t count as a test method. Reference

mockStdout is the mock outcome from previous decorator.

        self.myPerson.speak('haha')

Runs speak() from the instance self.myPerson with argument haha.

Expect to see haha\n in sys.stdout.write after running this line.

        mockStdout.write.assert_has_calls([
            mock.call('haha\n')
        ])

Verify argument haha\n has been passed to sys.stdout.write.

Test yell()

    @mock.patch('use_mock.person.sys')
    def test_yell(self, mockSys):
        self.myPerson.yell('haha')
        mockSys.stdout.write.assert_has_calls([
            mock.call('haha\n')
        ])
        mockSys.stderr.write.assert_has_calls([
            mock.call('haha!\n')
        ])

Contents are almost the same with previous section. However, there are slightly differences.

• We mock use_mock.person.sys instead of use_mock.person.sys.stdout
• assert runs onmockSys.stdout and mockSys.stderr

The take away is that you can define how deep to mock by giving a concise path to patch().

In our case, we mock out sys directly as we need to test sys.stderr and sys.stdout.

Something extra

• About getting coverage report

References

• https://docs.python.org/3/library/unittest.mock.html#module-unittest.mock
• https://docs.python.org/2/library/unittest.html
• https://stackoverflow.com/questions/8522689/how-to-temporary-hide-stdout-or-stderr-while-running-a-unittest-in-python
• https://stackoverflow.com/questions/12998908/is-it-possible-to-mock-pythons-built-in-print-function

​

A workaround on fixing internal DNS not working well in Ubuntu

About

During my works, I need to use an internal DNS for resolving my private domain. However, it does not work quite well. Sometime it resolves the private instantly sometime it just resolving nothing.

I tried to get used to it. However, it really piss me off given that it becomes more and more frequently. So, here is a workaround (solution) I found after digging a really while :)

Workaround: Disable dnsmasq in Ubuntu

• edit /etc/NetworkManager/NetworkManager.conf
• comment the dns=dnsmasq line by putting a # in front of it
• Run command sudo restart network-manager on your terminal

Why it work?

According to a blog here, Ubuntu relies on a dnsmasq local process for handling DNS’s jobs. It works well on VPN scenario. DNS traffics resolve on your local area network instead of routing through your VPN.

However, there is a noticeable statement on how DNSMASQ works on multiple DNS server.

As for dealing with DNS failures, dnsmasq often sends the DNS queries to more
than one DNS servers (if you received multiple when establishing your
connection) and will detect bogus/dead ones and simply ignore them until they
start returning sensible information again.

As a matter of fact that, there are 2 DNS in my Ubuntu. The Primary one is my internal DNS while the 2nd one is a Google DNS 8.8.8.8. Of cause Google DNS returns negative response for my private domain and DNSMASQ shuts everything down as a result.

Therefore, I disable the dnsmasq process which forces the DNS server to be queried in my order.

Everything works harmonically. Perfect :P

​