2018/05/16

Python FTPS client with self-sign certificate and ftplib

About

Example from ftplib should fit perfectly for developers working with a real certificate. In this blog post, I am focusing on coding a FTPS client with self-sign certificate only.

Warning

  • Although I code this example in Python 2.7.14, Python 3.16 is recommended since the poor support on session reuse issue. We will go through this at Hiccup section.

Show me codes

from ftplib import FTP_TLS
import ssl

# Create a SSL context for FTPS object later on
context = ssl.SSLContext(ssl.PROTOCOL_TLS)

# In my example, I would like to force the FTP client to use TLS_V1
# instead of SSLv2 or SSLv3
context.options |= ssl.OP_NO_SSLv2
context.options |= ssl.OP_NO_SSLv3 

# We need to initialize this SSL context with self-sign certificate
# Otherwise, there is SSL handshake errors later on 
context.load_cert_chain(
    "location_of_cert_chain_in_pem", "location_of_keyfile_in_pem")

# Define the CA bundle
context.load_verify_locations(cafile="location_of_ca_bundle_in_pem")


# Instantiate a ftps client object with our context above
ftps = FTP_TLS(host='TARGET_IP', context=context)
# Verbose debug information for testing the implementation 
ftps.set_debuglevel(2) 

# login anonymously before securing control channel
ftps.login()
# Secure the channel
ftps.prot_p()

# Upload the file
with open('TARGET_FILE', 'rb') as fp:
    ftps.storbinary('STOR REMOTE_FILE_PATH', fp)
The first part of implementation focus on how to instantiate a SSLContext, while the 2nd part focus on how to config the FTP client to use our SSLContext, and the 3rd part shows how we can upload a file with this client.

Hiccups

FTPS session reuse

Session reuse is a feature provided by some FTPS server such as vsftpd. However, as stated in this stackoverflow question, ftplib in python 2.7.14 does not support session reuse until python 3.16 (16 May 2018).
In my opinion, there are 2 solutions for developers in python 2.7.14.
The first one is disable session reuse even though there is a security flaw as described in this blog
The second one is replacing ftplib with a system command lftp.

Updated

20180516: curl is not able to reuse the TLS session.

References

2018/05/13

A trick for showing more transactions for 360 rewards in Standard Chartered Bank

About

While reading transactions for 360 rewards in Standard Chartered Bank, I found out a way for listing more than 10 records in a page. In my opinion, this is much better for reading all transactions in 1 page. I will first go through how to use that trick and then explain how this trick work and show some alternative for this usage finally.

How to

First go to the transaction history page

Analyse the URL

Make sure your URL in your browser is exactly like below. (In developer terminologies, make sure there are no params in your URL)
https://360rewards.standardchartered.com/HK/Transaction

Paste following at the end of the URL

Let’s say, you would like to show 100 transactions for the first credit card in last month, paste following parameters at the end your URL
?CycleId=1&paging.pagesize=100
As a result, your URL should look like following
https://360rewards.standardchartered.com/HK/Transaction?CycleId=1&paging.pagesize=100
After clicking enter with this new URL, you can read at most 100 records at the same screen :D

Explanation

Till now (13 May 2018), transaction page in 360 rewards relies on the parameters in the URL to fine tune number of records to show in the page. Usually, there are 3 or 4 parameters in the URL we can play with.

CycleId

Value of this parameter ranges from 0 to 3 which exactly map to the button in Time Period respectively.
Value Time Period
0 Current
1 Last month
2 2 month ago
3 3 month ago
Therefore, by changing value in the CycleID, you can filter out transaction records by date.

paging.pagesize

As the parameter name, it controls how many transactions records to show in one page

paging.currentpage

When you click around the buttons in transaction web page, you may see several more parameters. Value of this parameter control which page of records you are watching. For example, you have 100 transaction records in total while pagesize is 10. Then, you are reading 30th to 40th transaction records if the currentpage is 3.

Filter

Yet another parameter in the URL. Value of this parameter defines which credit card you would like to show. Transactions records other than this credit card will be discarded.

Alternative usage

After the explanation section, readers are able to fine tune the page’s behaviour while clicking around in the transaction web page. For example, you can select credit card and 2 month ago first. Then, change / add parameters I mentioned above to fine tune the behaviour according to your wish.
Moreover, this trick can be applied to any other websites as long as they are using the similar mechanism. Next time, remember to take a look the URL and see whether there are anything interesting for you to play with :)
Enjoy.

Updates:

  • 20181121: This trick does not work anymore since sc is fixed I guessed

2018/05/11

Alternative for add-apt-repository for installing new ubuntu packages

Background

add-apt-repository is a way for Ubuntu system to add 3rd party repository for installing certain unofficial software packages. However, add-apt-repository is not builtin in Ubuntu. You need to install software-properties-common as a prerequisites. Personally, I think they take too long while I am writing a Dockerfile. So, I have found an alternative for doing the same thing.

How to

Generally, there are 2 things we need to do:
  • Add the repository
  • Add the key for that repository

Add the repository

You can simply add following two lines into /etc/apt/sources.list. (Here I want to add python 2.7.1x packages into the Ubuntu system)
deb http://ppa.launchpad.net/jonathonf/python-2.7/ubuntu trusty main 
# deb-src http://ppa.launchpad.net/jonathonf/python-2.7/ubuntu trusty main

Add the key for that repository

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F06FC659
The last argument F06FC659 is the KEY ID. Take a look from this blog to find out the ID of a key.

Finally

Update the list and install the package you want
apt update
apt install XXXYYY

References