Python FTPS client with self-sign certificate and ftplib

About

Example from ftplib should fit perfectly for developers working with a real certificate. In this blog post, I am focusing on coding a FTPS client with self-sign certificate only.

Warning

• Although I code this example in Python 2.7.14, Python 3.16 is recommended since the poor support on session reuse issue. We will go through this at Hiccup section.

Show me codes

from ftplib import FTP_TLS
import ssl

# Create a SSL context for FTPS object later on
context = ssl.SSLContext(ssl.PROTOCOL_TLS)

# In my example, I would like to force the FTP client to use TLS_V1
# instead of SSLv2 or SSLv3
context.options |= ssl.OP_NO_SSLv2
context.options |= ssl.OP_NO_SSLv3 

# We need to initialize this SSL context with self-sign certificate
# Otherwise, there is SSL handshake errors later on 
context.load_cert_chain(
    "location_of_cert_chain_in_pem", "location_of_keyfile_in_pem")

# Define the CA bundle
context.load_verify_locations(cafile="location_of_ca_bundle_in_pem")


# Instantiate a ftps client object with our context above
ftps = FTP_TLS(host='TARGET_IP', context=context)
# Verbose debug information for testing the implementation 
ftps.set_debuglevel(2) 

# login anonymously before securing control channel
ftps.login()
# Secure the channel
ftps.prot_p()

# Upload the file
with open('TARGET_FILE', 'rb') as fp:
    ftps.storbinary('STOR REMOTE_FILE_PATH', fp)

The first part of implementation focus on how to instantiate a SSLContext, while the 2nd part focus on how to config the FTP client to use our SSLContext, and the 3rd part shows how we can upload a file with this client.

Hiccups

FTPS session reuse

Session reuse is a feature provided by some FTPS server such as vsftpd. However, as stated in this stackoverflow question, ftplib in python 2.7.14 does not support session reuse until python 3.16 (16 May 2018).

In my opinion, there are 2 solutions for developers in python 2.7.14.

The first one is disable session reuse even though there is a security flaw as described in this blog

The second one is replacing ftplib with a system command lftp.

Updated

20180516: curl is not able to reuse the TLS session.

References

• https://docs.python.org/2/library/ftplib.html
• https://stackoverflow.com/questions/5534830/ftpes-ftp-over-explicit-tls-ssl-in-python
• https://stackoverflow.com/questions/14659154/ftpes-session-reuse-required
• https://docs.python.org/2/library/ssl.html#ssl-security

​

A trick for showing more transactions for 360 rewards in Standard Chartered Bank

About

While reading transactions for 360 rewards in Standard Chartered Bank, I found out a way for listing more than 10 records in a page. In my opinion, this is much better for reading all transactions in 1 page. I will first go through how to use that trick and then explain how this trick work and show some alternative for this usage finally.

How to

First go to the transaction history page

https://360rewards.standardchartered.com/HK/Transaction

Analyse the URL

Make sure your URL in your browser is exactly like below. (In developer terminologies, make sure there are no params in your URL)

https://360rewards.standardchartered.com/HK/Transaction

Paste following at the end of the URL

Let’s say, you would like to show 100 transactions for the first credit card in last month, paste following parameters at the end your URL

?CycleId=1&paging.pagesize=100

As a result, your URL should look like following

https://360rewards.standardchartered.com/HK/Transaction?CycleId=1&paging.pagesize=100

After clicking enter with this new URL, you can read at most 100 records at the same screen :D

Explanation

Till now (13 May 2018), transaction page in 360 rewards relies on the parameters in the URL to fine tune number of records to show in the page. Usually, there are 3 or 4 parameters in the URL we can play with.

CycleId

Value of this parameter ranges from 0 to 3 which exactly map to the button in Time Period respectively.

Value	Time Period
0	Current
1	Last month
2	2 month ago
3	3 month ago

Therefore, by changing value in the CycleID, you can filter out transaction records by date.

paging.pagesize

As the parameter name, it controls how many transactions records to show in one page

paging.currentpage

When you click around the buttons in transaction web page, you may see several more parameters. Value of this parameter control which page of records you are watching. For example, you have 100 transaction records in total while pagesize is 10. Then, you are reading 30th to 40th transaction records if the currentpage is 3.

Filter

Yet another parameter in the URL. Value of this parameter defines which credit card you would like to show. Transactions records other than this credit card will be discarded.

Alternative usage

After the explanation section, readers are able to fine tune the page’s behaviour while clicking around in the transaction web page. For example, you can select credit card and 2 month ago first. Then, change / add parameters I mentioned above to fine tune the behaviour according to your wish.

Moreover, this trick can be applied to any other websites as long as they are using the similar mechanism. Next time, remember to take a look the URL and see whether there are anything interesting for you to play with :)

Enjoy.

Updates:

• 20181121: This trick does not work anymore since sc is fixed I guessed

​

Alternative for add-apt-repository for installing new ubuntu packages

Background

add-apt-repository is a way for Ubuntu system to add 3rd party repository for installing certain unofficial software packages. However, add-apt-repository is not builtin in Ubuntu. You need to install software-properties-common as a prerequisites. Personally, I think they take too long while I am writing a Dockerfile. So, I have found an alternative for doing the same thing.

How to

Generally, there are 2 things we need to do:

• Add the repository
• Add the key for that repository

Add the repository

You can simply add following two lines into /etc/apt/sources.list. (Here I want to add python 2.7.1x packages into the Ubuntu system)

deb http://ppa.launchpad.net/jonathonf/python-2.7/ubuntu trusty main 
# deb-src http://ppa.launchpad.net/jonathonf/python-2.7/ubuntu trusty main

Add the key for that repository

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F06FC659

The last argument F06FC659 is the KEY ID. Take a look from this blog to find out the ID of a key.

Finally

Update the list and install the package you want

apt update
apt install XXXYYY

References

• https://askubuntu.com/questions/173496/alternative-to-sudo-add-apt-repository-ppaupubuntu-com-web
• https://askubuntu.com/questions/493460/how-to-install-add-apt-repository-using-the-terminal
• https://www.ghacks.net/2011/03/31/adding-missing-ppa-keys-to-an-apt-keyring/

​

Copy with or without symbolic links in local or remote manner

About

Have you ever puzzled about how to deal with symbolic files when copying files? Here are some useful commands for you. You can also find them in this gist

Background

Given a directory structure like this:

.
├── a
│   └── a.txt
├── b
│   └── b.txt
├── c
├── s1 -> a
└── s2 -> b

3 directories a, b and c. 2 symbolic links s1 and s2

Local

Copy files / directories locally

Case 1: Copy with symbolic links preserved

Suppose you need to copy from src to dst, you can run following:

$ cp -r src/* dst/
$ tree dst
dst
├── a
│   └── a.txt
├── b
│   └── b.txt
├── c
├── s1 -> a
└── s2 -> b

Case 2: Copy without symbolic links but a real directory

Suppose you need to copy without a symbolic link but a real directory, you can run following:

$ cp -rL src/* dst/
$ tree dst
dst
├── a
│   └── a.txt
├── b
│   └── b.txt
├── c
├── s1
│   └── a.txt
└── s2
    └── b.txt

Remote

Copy files / directories from host to host. We need command scp.

Case 1: Copy with symbolic links preserved

Suppose you need to copy from src to dst, you can try tar scp combo:

# Tar the entire folder
$ tar cf src.tar.gz src/

# Scp to the remote side
# Remember to replace `localhost` and `pwd` with your own variables
$ scp src.tar.gz localhost:`pwd`/subdir/

# Untar the zip in `subdir/src.tar.gz` in folder `subdir`
$ tar -x -C subdir/ -f subdir/src.tar.gz

# Rename the `subdir/src` to `subdir/dst`
$ mv subdir/src subdir/dst

$ tree subdir/dst
subdir/dst/
├── a
│   └── a.txt
├── b
│   └── b.txt
├── c
├── s1 -> a
└── s2 -> b

Or, you can do them on the fly:

# Open a new terminal for the destination host and run following command
# Use NC to listen on any port (EG: 12345) and untar it on the fly
$ nc -l 12345 | tar xf -

# Open another new terminal for the source host and run following command
# Tar the entire folder and send them through netcat
# Remember to replace `localhost` and `pwd` with your own variables
$ tar cf - src/ | nc localhost 12345

# Verify in destionation host
$ mv src dst
$ tree dst
dst
├── a
│   └── a.txt
├── b
│   └── b.txt
├── c
├── s1 -> a
└── s2 -> b

Case 2: Copy without symbolic links but a real directory

Suppose you need to copy without a symbolic link but a real directory, you can run following:

# Scp to the remote side
# Remember to replace `localhost` and `pwd` with your own variables
$  scp -r src/* localhost:`pwd`/dst
$ tree dst
dst/
├── a
│   └── a.txt
├── b
│   └── b.txt
├── c
├── s1
│   └── a.txt
└── s2
    └── b.txt

​

A study on SSL certificates

Background

Recently, I have a chance to deploy a real signed SSL certificate on company’s Gitlab server. By following this link from Gitlab, browser shows a lovely lock icon easily.

However, there is a trouble when configuring container registry with such certificate. Gitlab runners throw an error about unable to get issuer certificate. Just like this and that.

Eventually, I find out this is a problem from intermediate certificates and fix it with some struggles. Below are the glossaries I have crawled.

Glossaries

SSL certificates

aka X.509 certificate

Encoding

• DER
  • a binary encoded certificate
• PEM
  • BASE 64 ASCII encoded certificate
  • Contains line ----BEGIN----

Usually, you would say I have a DER encoded certificate instead of DER certificate.

Extension

• CRT
  • Can be encoded with DER, PEM
  • Common on Linux
• CER
  • Similar to CRT
  • Common on Window
• KEY
  • Public key or private key

Actions

• View
  • get human readable string
    • PEM
      • openssl x509 -in cert.xxx -text -noout
    • DER
      • openssl x509 -in certificate.der -inform der -text -noout
• Transform
  • DER 2 PEM
    • openssl x509 -in cert.crt -inform der -outform pem -out cert.pem
  • PEM 2 DER
    • openssl x509 -in cert.crt -outform der -out cert.der
• Combinations
  • Concatenate multiple certificates in one file
  • EG: Combine intermediate certificates with your certificate

CSR

• Certificate Signing Request
• A step before becoming a CER / CRT (Or signed by CA)
• Contains information like Common Name, Organization Name etc
• Decode
  • openssl req -in server.csr -noout -text

Intermediate certificate

• Certificate(s) between your site and root certificate
  • Construct chain of trust
• A proxy protecting the root certificate
  • A good explain from stackexchange
• Certificate ordering follow RFC4346
  • 1st Server certificate
  • Any intermediate follow each other one by one

TLS

A successor protocol or an enhancer version of SSL which support DV, OV, EV, BV etc.

References

• https://support.ssl.com/Knowledgebase/Article/View/19/0/der-vs-crt-vs-cer-vs-pem-certificates-and-how-to-convert-them
• https://support.comodo.com/index.php?/comodo/Knowledgebase/Article/View/250/17/
• https://support.comodo.com/index.php?/Knowledgebase/Article/View/1091/37/certificate-installation--nginx
• https://www.symantec.com/page.jsp?id=ssl-information-center
• https://www.sslshopper.com/what-is-a-csr-certificate-signing-request.html
• https://security.stackexchange.com/questions/128779/why-is-it-more-secure-to-use-intermediate-ca-certificates
• https://hk.godaddy.com/en/help/what-is-an-intermediate-certificate-868
• https://serverfault.com/questions/476576/how-to-combine-various-certificates-into-single-pem

​

How to do list intersection efficiently in python

Question

In previous blog, I have investigated how to do array initialization efficiently. This time, I would like to investigate how to do intersections in Python efficiently.

For those need to read code immediately, here is the gist for this blog

Methods

Before digging in the implementations, let’s define what intersection is. By intersection, I mean finding same elements between 2 given lists.

A = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
B = [2, 4, 6, 8, 10]
# c = a ^ b
# which is c = [2, 4, 6, 8]

How do we get c = [2, 4, 6, 8] efficiently?

M1: double loop

A double loop for finding same elements

def doubleLoop():
    C = [a for a in A for b in B if a == b]

def doubleLoop2():
    C = []
    for a in A:
        for b in B:
            if a == b:
                C.append(a)

Noted

• list comprehension vs append

M2: Set intersection

Cast A and B to set and do intersection.

setA = set(A)
setB = set(B)

def setAnd():
    C = sorted(setA.intersection(setB))

def setAnd2():
    # No sort
    C = setA.intersection(setB)

def setAnd3():
    # Inline construct
    C = sorted(set(A).intersection(set(B)))

def setAnd4():
    # Inline construct and no sort
    C = set(A).intersection(set(B))

Noted:

• setAnd ensures the ordering sames with output from doubleLoop or doubleLoop2 as set has no concept of ordering
• setAnd2 demonstrates the meaning of no ordering
• setAnd3 demonstrates how slow for casting them in a run time manner
• setAnd4 is similar to `setAnd2

Result

Results of running each implementations for 1000000 times

# Double loop
# 1.4873290062
# Double loop2
# 1.70801305771
# Set AND
# 0.525309085846
# Set AND2
# 0.199652194977
# Set AND3
# 0.982024908066
# Set AND4
# 0.657478094101

Conclusion

• Always use set to do intersections. No matter the results need to be sorted or not, it is always faster than the double loop way.
• List comprehension is always faster than append. Check out more from my previous blog

Explanation

setAnd vs setAnd2

• setAnd do an extra sort comparing to setAnd2
  • As set has no ordering, it need to be cast to list before which is an extra cost

setAnd vs setAnd3

A performance checks between using a pre-cast sets setA and setB and runtime-cast sets set(A) and set(B). Even though the runtime version is slower, it is still faster than double loop obviously.

setAnd3 vs setAnd4

Same as setAnd vs setAnd2

Gotcha

• Set’s intersection works charm on unique elements inside involved lists. Benchmark here may not be applied to the multi-set one
• Set has no concept of ordering. If ordering is a must for you, you need to do it yourself.

​