2018/11/25

A stupid way to semi automate steps for creating Merge Request on gitlab

Hello all. I have been away for writing a blog a while. Today, I would like to share how I semi automate steps for creating a MR on gitlab via command line.

Show me source codes

This is simply a script for filling information, for regular MR contents, and open a browser, which is a GUI for me to verify the contents, to fire MR. It, however, saves my time for filling boring and redundant MR contents. Here we go.

import subprocess

# Where is your project: NAMESPACE/REPO_NAME
MR_URL_ARGVS = {
    'NAMESPACE': 'mondwan',
    'REPO_NAME': 'scripts_for_creating_gitlab_mr',
}

# What will be the arguments for the MR URL
MR_SETTINGS = {
    # 'source_branch': 'feature_branch_1',
    'target_branch': 'master',
    # 'force_remove_source_branch': 'true',
    'title': 'New%20MR',
    # User ID in the gitlab
    'assignee_id': '1158561',
}

# The base url
MR_URL_BASE = \
    'https://gitlab.com/{NAMESPACE}/{REPO_NAME}/merge_requests/new?'.format(
        **MR_URL_ARGVS
    )

# Fill up source branch by current branch name via system git command
currentBranchName = subprocess.check_output(
    'git rev-parse --abbrev-ref HEAD'.split(' ')
).strip('\n')
MR_SETTINGS['source_branch'] = currentBranchName

# Create params list for the MR which will be pasted at the END of URL
getParams = '&'.join([
    'merge_request%%5B%s%%5D=%s' % (k, v) for k, v in MR_SETTINGS.items()
])

# The final url
url = MR_URL_BASE + getParams

# Launch a browser for given url with command line in ubuntu
print 'gnome-open "%s"' % url
Below explain how it works part by part.

How it works

If you studied carefully, gitlab accept a hand made URL for creating Merge request.
We can prefill whatever information in that MR if we can craft a specific URL.
Below is how to craft the first part of the URL, where variables are the namespace and repository name.

# Where is your project: NAMESPACE/REPO_NAME
MR_URL_ARGVS = {
    'NAMESPACE': 'mondwan',
    'REPO_NAME': 'scripts_for_creating_gitlab_mr',
}

# The base url
MR_URL_BASE = \
    'https://gitlab.com/{NAMESPACE}/{REPO_NAME}/merge_requests/new?'.format(
        **MR_URL_ARGVS
    )
The next question is how can I fill text into a field like title, contents, or source branch. Get Params is the answer.
For example, if you append string merge_request%5Btitle%5D=New%20MR at the end of the URL, title of that page will be New MR.
Below shows how to create a MR where title is New MR, assigned to an user with id 1158561, and set master to be the target branch

# What will be the arguments for the MR URL
MR_SETTINGS = {
    # 'source_branch': 'feature_branch_1',
    'target_branch': 'master',
    # 'force_remove_source_branch': 'true',
    'title': 'New%20MR',
    # User ID in the gitlab
    'assignee_id': '1158561',
}



# Create params list for the MR which will be pasted at the END of URL
getParams = '&'.join([
    'merge_request%%5B%s%%5D=%s' % (k, v) for k, v in MR_SETTINGS.items()
])
You may wonder how can I know what should be the key in order to fill up a field. The answer is looking them up via your browser’s console. For example, I would like to look up the key of source branch. I will see something like that in the console after watching the corresponding DOM element from the HTML.
As you can see in the diagram, we can locate merge_request[source_branch] in the HTML. That’s the key (source_branch) we need.
For me, it will be convenient if my script is able to fetch the name of current branch and put that in to the source branch field.

# Fill up source branch by current branch name via system git command
currentBranchName = subprocess.check_output(
    'git rev-parse --abbrev-ref HEAD'.split(' ')
).strip('\n')
MR_SETTINGS['source_branch'] = currentBranchName
Finally, I would like to launch a browser through command line by using gnome-open

 # The final url
url = MR_URL_BASE + getParams

# Launch a browser for given url with command line in ubuntu
print 'gnome-open "%s"' % url
Then, I can copy and paste that URL into the command line. A browser, for me it is Firefox, will be jumped out eventually in Ubuntu.

The repository for this script

Alternative

For advance developers, you may try to automate the entire process by using official APIs. For python developers, you can try to code them up with this cli library.
Although I have not tried to code this alternative, I guess this is much more time consuming than my approach, in term of time spent on development, listed above. One thing, however, I can sure is that this alternative is much more efficient than my approach.

Change logs

20181125: Add alternative section, refine some of the wordings

2018/05/16

Python FTPS client with self-sign certificate and ftplib

About

Example from ftplib should fit perfectly for developers working with a real certificate. In this blog post, I am focusing on coding a FTPS client with self-sign certificate only.

Warning

  • Although I code this example in Python 2.7.14, Python 3.16 is recommended since the poor support on session reuse issue. We will go through this at Hiccup section.

Show me codes

from ftplib import FTP_TLS
import ssl

# Create a SSL context for FTPS object later on
context = ssl.SSLContext(ssl.PROTOCOL_TLS)

# In my example, I would like to force the FTP client to use TLS_V1
# instead of SSLv2 or SSLv3
context.options |= ssl.OP_NO_SSLv2
context.options |= ssl.OP_NO_SSLv3 

# We need to initialize this SSL context with self-sign certificate
# Otherwise, there is SSL handshake errors later on 
context.load_cert_chain(
    "location_of_cert_chain_in_pem", "location_of_keyfile_in_pem")

# Define the CA bundle
context.load_verify_locations(cafile="location_of_ca_bundle_in_pem")


# Instantiate a ftps client object with our context above
ftps = FTP_TLS(host='TARGET_IP', context=context)
# Verbose debug information for testing the implementation 
ftps.set_debuglevel(2) 

# login anonymously before securing control channel
ftps.login()
# Secure the channel
ftps.prot_p()

# Upload the file
with open('TARGET_FILE', 'rb') as fp:
    ftps.storbinary('STOR REMOTE_FILE_PATH', fp)
The first part of implementation focus on how to instantiate a SSLContext, while the 2nd part focus on how to config the FTP client to use our SSLContext, and the 3rd part shows how we can upload a file with this client.

Hiccups

FTPS session reuse

Session reuse is a feature provided by some FTPS server such as vsftpd. However, as stated in this stackoverflow question, ftplib in python 2.7.14 does not support session reuse until python 3.16 (16 May 2018).
In my opinion, there are 2 solutions for developers in python 2.7.14.
The first one is disable session reuse even though there is a security flaw as described in this blog
The second one is replacing ftplib with a system command lftp.

Updated

20180516: curl is not able to reuse the TLS session.

References

2018/05/13

A trick for showing more transactions for 360 rewards in Standard Chartered Bank

About

While reading transactions for 360 rewards in Standard Chartered Bank, I found out a way for listing more than 10 records in a page. In my opinion, this is much better for reading all transactions in 1 page. I will first go through how to use that trick and then explain how this trick work and show some alternative for this usage finally.

How to

First go to the transaction history page

Analyse the URL

Make sure your URL in your browser is exactly like below. (In developer terminologies, make sure there are no params in your URL)
https://360rewards.standardchartered.com/HK/Transaction

Paste following at the end of the URL

Let’s say, you would like to show 100 transactions for the first credit card in last month, paste following parameters at the end your URL
?CycleId=1&paging.pagesize=100
As a result, your URL should look like following
https://360rewards.standardchartered.com/HK/Transaction?CycleId=1&paging.pagesize=100
After clicking enter with this new URL, you can read at most 100 records at the same screen :D

Explanation

Till now (13 May 2018), transaction page in 360 rewards relies on the parameters in the URL to fine tune number of records to show in the page. Usually, there are 3 or 4 parameters in the URL we can play with.

CycleId

Value of this parameter ranges from 0 to 3 which exactly map to the button in Time Period respectively.
Value Time Period
0 Current
1 Last month
2 2 month ago
3 3 month ago
Therefore, by changing value in the CycleID, you can filter out transaction records by date.

paging.pagesize

As the parameter name, it controls how many transactions records to show in one page

paging.currentpage

When you click around the buttons in transaction web page, you may see several more parameters. Value of this parameter control which page of records you are watching. For example, you have 100 transaction records in total while pagesize is 10. Then, you are reading 30th to 40th transaction records if the currentpage is 3.

Filter

Yet another parameter in the URL. Value of this parameter defines which credit card you would like to show. Transactions records other than this credit card will be discarded.

Alternative usage

After the explanation section, readers are able to fine tune the page’s behaviour while clicking around in the transaction web page. For example, you can select credit card and 2 month ago first. Then, change / add parameters I mentioned above to fine tune the behaviour according to your wish.
Moreover, this trick can be applied to any other websites as long as they are using the similar mechanism. Next time, remember to take a look the URL and see whether there are anything interesting for you to play with :)
Enjoy.

Updates:

  • 20181121: This trick does not work anymore since sc is fixed I guessed

2018/05/11

Alternative for add-apt-repository for installing new ubuntu packages

Background

add-apt-repository is a way for Ubuntu system to add 3rd party repository for installing certain unofficial software packages. However, add-apt-repository is not builtin in Ubuntu. You need to install software-properties-common as a prerequisites. Personally, I think they take too long while I am writing a Dockerfile. So, I have found an alternative for doing the same thing.

How to

Generally, there are 2 things we need to do:
  • Add the repository
  • Add the key for that repository

Add the repository

You can simply add following two lines into /etc/apt/sources.list. (Here I want to add python 2.7.1x packages into the Ubuntu system)
deb http://ppa.launchpad.net/jonathonf/python-2.7/ubuntu trusty main 
# deb-src http://ppa.launchpad.net/jonathonf/python-2.7/ubuntu trusty main

Add the key for that repository

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F06FC659
The last argument F06FC659 is the KEY ID. Take a look from this blog to find out the ID of a key.

Finally

Update the list and install the package you want
apt update
apt install XXXYYY

References