2022/07/03

Make use of free tier gitlab pages for hosting with openapi document & coverage report

1. Goal

In this blog, I am going to make use of gitlab pages for hosting my openapi documents & coverage reports.

TL;DR,

FYI, this is a follow up blog for my flask learning. Before this blog, I have written a blog for showing how to setup the vscode’s debugger from scratch. Make sure to check them out if they are helpful for your cases.

1.1 Contents overview

  • In section 2, I will explain how to build the openapi document in cicd
  • In section 3, I will explain how to build the coverage report in html in cicd
  • In section 4, I will explain how to wrap up everything, and upload to the gitlab pages
  • In bonus section, I will write down my advices on writing openapi.yaml from scratch

2. Build OpenAPI document in cicd

In this section, I assume you have a valid openapi.yaml at the first place. If you don’t, please read my advices in bonus section before writing this file from scratch. It may help you a bit.

General idea

  • Convert the yaml to html files
  • Upload the html files to gitlab pages
  • Do this only if I am updating master / tagging

In order to build the OpenAPI document in gitlab cicd, you need following yaml

...
stages:
  - build
  - test
  - buildPages
  - deploy
...
buildOpenapiPages:
  image: node:18-alpine
  stage: buildPages
  dependencies: []
  script:
    - wget https://github.com/swagger-api/swagger-ui/archive/refs/tags/v4.12.0.tar.gz -O swagger.tar.gz
    - tar xvf swagger.tar.gz
    - mkdir -p build/openapi/
    - cp -fr swagger*/dist/* build/openapi/
    - npm install @apidevtools/swagger-cli
    - ./node_modules/.bin/swagger-cli bundle -o build/openapi/openapi.json ./openapi.yaml
    - sed -i
      "s#https://petstore\.swagger\.io/v2/swagger\.json#${OPENAPI_JSON_FILE_URL}#g"
      build/openapi/swagger-initializer.js
  artifacts:
    untracked: false
    expire_in: 1 hour
    paths:
      - build/openapi/
  only:
    - main
    - tags

Before explaining script, I will explain rest of the fields first

  • I have defined a new stage buildPages for building the related pages
    • Ensure we build pages only if it passed the pipeline
  • I only upload build/openapi/ since I only need files there
    • untracked is suggested from the IDE. It does not really matter
    • Set expire_in to a relative short period since I don’t need them after the pipeline
  • Use only to ensure this task will only be run on main branch, or tags (tagging)
  • Set dependencies to empty array in order to speed up the pipeline process. Otherwise, it will download the artifacts in the test stage

In script, there are 2 things happening

  • Installing a standalone swagger version as suggested by the official guide
  • Convert openapi.yaml to openapi.json as required by the standalone swagger ui
    • By using @apidevtools/swagger-cli
    • That’s why I need image node:18-alpine. Version is not important as long as it can run the above tool

After the installation & conversion, I put everything under build/openapi for uploading as gitlab’s artifacts.

A side note on ${OPENAPI_JSON_FILE_URL}. Since the swagger ui need this file in order to render my APIs properly, I need to make sure the swagger ui can find this file in the runtime. So, I copy this json file next to the swagger ui folder. Then, it will shares the same gitlab pages url with swagger ui.

3. Build coverage report in cicd

Back in 2017, my old blog described how to do that in general. So, in 2022, I will just modified a bit from my old works.
The yaml snippet required in this section.

runTests:
  ....
      - coverage html -d build/coverage
  ...
  artifacts:
   ...
    untracked: false
    expire_in: 1 hour
    paths:
      - build/coverage/
  • For artifacts, it is similar to section 2. But, I need build/coverage/ this time
  • I generate the coverage report by using coverage html -d build/coverage
  • A side note one the stage.
    • I generate the document at the end of task in test stage instead of creating a buildCoverageReport task.
    • This can save me some pipeline time for rerunning the coverage again.

That’s it.

4. Upload to gitlab pages

Finally, I need to copy all artifacts from previous stages to public/ in a specific pages task.

pages:
  stage: deploy
  image: alpine:3.16
  variables:
    GIT_STRATEGY: none
  script:
    - mkdir -p public
    - cp -r build/* public/
  artifacts:
    untracked: false
    expire_in: 1 hour
    paths:
      - public
  dependencies:
    - runTests
    - buildOpenapiPages
  only:
    - main
    - tags
  • Set dependencies explicitly. Personal habit. It inherits all artifacts from previous stages by default
  • Set GIT_STRATEGY to none for speeding up the pipeline process since I just need artifacts and nothing else
  • Rest of the keys are explained in previous stages

That’s it. After the pipeline’s run, I will have 2 different documentations under public/ that will hosted by gitlab pages

  • public/coverage/
  • public/openapi/

Check out the links in section 1 for the demo.

A side note on writing gitlab pages. Old gitlab pages will be erased whenever you publish new gitlab pages. So, please do not assume it will keep your stuff forever.

5. Bonus Section advices on writing openapi.yaml

5.1 Get an IDE

Personally, I am using vscode with extension 42Crunch.vscode-openapi. Features like linting, code hinting, and previewing with style are really important for writing documents (Or, codes). You don’t need to follow my IDE setup but I strongly recommend you to get an IDE that support above features. It will save you many time for debugging your syntax etc.

At least, you can use the online editor from swagger

5.2 Essential websites for openapi reference

IMO, there are no quick ways for learning openapi’s writing. It really takes time for studying openapi specifications. Otherwise, you cannot just really write something you want.

Personally, I will suggest you to go through their basic openapi guide first. It provide certain common snippets for you to start with.

Then, after you gathering certain set of snippets, I suggest you to start reading the schema section in openapi specification. As the meaning of the specification, it describes every bits in your snippets clearly.

Finally, you will behave just like me. You will keep jumping between schema & guide.

  • In schema, you will have a general idea about what fields you need
  • Go to guide, you will probably get some examples for your idea
  • Back to schema, try to tweak from the examples by understanding the schema throughly
  • Loop until you completed the yaml

Of cause, IDE helps you a lot in this process. For example, you can read what your changes IMMEDIATELY.

That’s it. I wish you can complete this yaml file soon. In my case, I spent couple hours lol…

References

​
Posted by at 沒有留言:
Labels: , , , , , , ,

2022/06/26

Start developing python flask with uwsgi and vscode's debugger inside docker!

1. Goal

In this blog post, I am trying to setup the vscode such that I can develop my flask project with debugger supported. Literally, I am trying to do following setup:

vscode + docker + uwsgi + flask

Although I am using a flask project I left since 2021 DEC, this methodology fits any project, and any frameworks. It is 2022. We should curse any projects that cannot develop with debugger.

  • TLDR, show me your codes
  • Go section 3 if you just interested on the setup’s explanation
    By the way, during my research time, there is no public solution for my requirements. Hopefully, this is the 1st public solution for vscode / uwsgi + flask + docker :)

1.1 Target Audiences

  • Any new python project
  • Any existing python project

1.2 Contents overview

In section 2, I will, briefly, explain the general idea on configuring the debugger in vscode.
In section 3, I will explain the changes file by file

Side track

A gif from vscode’s debugger introduction for showing how debugger can help our development.

2. General idea on configuring vscode’s debugger

Although I strongly recommend you to spend sometime on reading the vscode’s Debugging documentation, there are some highlights on some critical (I think) ideas.

2.1 launch.json: launch request vs attach request LINK

The best way to explain the difference between launch and attach is to think of a launch configuration
as a recipe for how to start your app in debug mode before VS Code attaches to it, while an attach
configuration is a recipe for how to connect VS Code’s debugger to an app or process
that’s already running.

In my own wording, for those applications that cannot launch in your own local terminal directly, you will need attach. Otherwise, you will need launch. For example, you will need attach for an application hosted by uwsgi (IE, our project in this blog). You will need launch if you can run the flask application locally. (IE, python -m flask run)

2.1.1 launch based environment

For launch based environment, no magic setups are required. You can get the basic configuration by clicking the Add configuration button in launch.json. Then, you can define the necessary env / args for your own application manually. Below is an example of a configuration in launch.json

{
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "justMyCode": false,
            "purpose": ["debug-test"],
            "env": {
                "PYTEST_ADDOPTS": "--no-cov"
            }
        }

2.1.2 attach based environment

Since vscode is not responsible for launching the application, you need to instruct vscode to attach to the application in order to activate the debugger for your application. Read section 3 for a concrete example.

Besides instructing vscode to attach somewhere, you need to configure the pathMappings. Literally, you need to tell vscode how to map the filesystem between your local vscode, and the remote filesystem (the application you attach). For example, ~/git/PROJECT is the project root in your local device while /var/www/PROJECT is where your application lives in production. You will need to specify the filesystem mapping between ~/git/PROJECT & /var/www/PROJECT. Otherwise, debugger cannot stop at a particular line for you.

2.2 tasks.json: The Makefile (Or, script section in package.json) in vscode LINK

  • tasks.json is not REQUIRED in our problem.
  • Without this, you can still make the debugger works.
  • With this, you will just save sometimes on typing just like Makefile or Script in package.json

3. Code explanation

Link to the git diff for related configurations

There are massive code changes. Before explaining the changes, I will state the goals of them:

  • I am running the vscode’s debugger in attach mode
    Since there are limitations (features) from my base docker image, I must run the flask application via uwsgi & superviord. Otherwise, I will runs my debugger in launcher mode. It is much easier.

  • I need to run flask with debugpy CONDTIONALLY
    Since I would like to use the same codebase between development & production, I don’t want to run the debugger in the production. So, I need to make sure the debugpy can be started only if it is a development environment

  • If I need debugger, I need to ensure

    • uwsgi creates one and only one worker for our flask app
    • uwsgi does not kill our worker
    • the debugpy’s server is on the worker thread

According to my tests, the attach mode of debugpy is really bad on multithreading. Since uwsgi must runs our flask in multithreading manner, I need to instruct uwsgi creates only 1 permanent worker. Otherwise, debugpy’s server cannot listen on my desired network port.

In uwsgi, the flask request handler’s codes are run by the workers. If you don’t creates the debugpy’s server on the worker process, the breakpoint functions in request handlers will not work as expected.

Of cause, if you need to debug codes tin both (master & workers) processes, you will need 2 different debugpy’s server. Then, you need 2 vscode’s debugger, for attaching to the servers separately.

3.1 Related to Vscode configuration

In this section, I will explain changes related to vscode configurations. As described on section 2, I am just instructing the vscode to attach to the application I want.

launch.json

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Attach to docker container",
            "type": "python",
            "request": "attach",
            "connect": { "host": "127.0.0.1", "port": 5678 },
            "pathMappings": [{ "localRoot": "${workspaceFolder}", "remoteRoot": "/workspace" }],
            "justMyCode": false
        }
    ]
}
  • As explained in section 2, we need attach based configuration.
    • We need to instruct the vscode to connect to 127.0.0.1:5678. This will be the debugger port that the application opened because our later configurations.
  • Since /workspace is the file root of our application inside the docker container, we instruct vscode to map our ${workspaceFolder} to /workspace. Checkout variable references from vscode if you are not familiar with them.
  • Rest of the options are not that important. Please read this manual for detail if you are interested

tasks.json

        {
            "type": "shell",
            "label": "local - docker-compose up -d with debugpy",
            "command": "docker-compose",
            "args": [
                "--env-file", "docker/setup_debugger.env",
                "-f", "docker/docker-compose-local-dev.yml",
                "up", "-d"
            ],
            "presentation": {
                "reveal": "always",
                "panel": "shared"
            }
        }
  • The only 1 important command in this file. It runs following command
    • docker-compose --env-file docker/setup_debugger.env -f docker/docker-compose-local-dev.yml up -d
    • This environment file docker/setup_debugger.env instructs how application to launch a debugger for vscode to attach. Refer to the explanations in below section

requirement-text.txt

  • Install debugpy, which is a library for vscode’s debugger

3.2 Related to the uwsgi configuration

app/uwsgi_dev.ini

[uwsgi]
module = app.app.main:createFlaskApp()
enable-threads = true
worker-reload-mercy = 999999999
cheaper-algo = backlog
  • worker-reload-mercy & cheaper-algo. are important options for instructing uwsgi not killing our worker.
  • enable-threads fixes warning from uwsgi
  • For all option’s detail, please refer to uwsgi option document

docker/docker-compose-local-dev.yaml

  • I split the environment variables in different files (services.CoreServer.env_file)
    • IMO, it is easier for maintaining them in different environments
  • services.CoreServer.environment allows me to run debugpy CONDITIONALLY
  • Without services.CoreServer.ports, the vscode debugger cannot attach to our debugpy’s server
  • Without stop_grace_period, I need to wait 10 seconds for stopping a container. For development, I want to restart it faster.

docker/xxx.env

As the file name suggested, I group environment variables accordingly.
In order to run a debugger successfully, please make sure you set option cheaper to 1. Otherwise, uwsgi will runs more than 1 worker. It will fails our setup

Note: As I told earlier, I am using some features (limitations) from the base docker image. That’s how they expect developers to do the configurations. Therefore, in your own setup, you don’t really need UWSGI_XXX. You can write all required UWSGI options in your uwsgi.ini.

3.3 Related to the flask application

app/app/main.py

  • createFlaskApp is how uwsgi build our application. So, we need to keep this
  • MyFlaskAppFactory keeps rest of the logic for building a flask application
  • DebugpyWsgiMiddleware runs the debugpy & wait for the debugger to attach before processing any wsgi requests
    • In uwsgi, you need to tell debugpy where is the python interpreter by using debugpy.configure({"python": pythonExecutablePath})
  • MyFlaskAppFactory._initializeFlaskWithDebugpy
    • shows how I stop debugpy being made in production environment
    • Then, I inject my DebugpyWsgiMiddleware on flaskApp.wsgi_app in order to make sure the debugger starts on the worker thread instead of the master thread

Note:
Other blog setup does not fit my case since I don’t really know the pid of the master process due to my special base docker image.

4. How to run the application with vscode debugger

After the configuration above, let’s run the flask application with debugger.

  • Run the docker-compose command as illustrated in section 3.1 for starting all the docker containers
  • curl any api in the server. For example curl http://127.0.0.1:56733/
  • Expect to see the output of the curl being stalled
    • Go to Run & Debug & run Attach to docker container
  • Expect to see the server output from the curl above
  • Now, you can setup any breakpoints, and debug whatever you want with that debugger

5. [Bonus] Enhance CICD with code coverage & junit report

In the old blog here, I show you how to add coverage report in the cicd in detail. To recap in short, all you need is following yaml

coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'

In 2022, gitlab provides much more on the coverage & testing. For detail, you can refer to following links:

  script:
      - flake8
      - coverage run -m pytest --junit-xml=junit.xml
      - coverage report
      - coverage xml
  coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
      junit: junit.xml

Then, in the cicd pipeline, it will generates junit.xml & coverage.xml for the gitlab builtin features as listed above

6. References

​
Posted by at 沒有留言:
Labels: , , , , , , , , ,
訂閱： 文章 (Atom)