2017/07/23

Copy with or without symbolic links in local or remote manner

About

Have you ever puzzled about how to deal with symbolic files when copying files? Here are some useful commands for you. You can also find them in this gist

Background

Given a directory structure like this:
.
├── a
│   └── a.txt
├── b
│   └── b.txt
├── c
├── s1 -> a
└── s2 -> b
3 directories a, b and c. 2 symbolic links s1 and s2

Local

Copy files / directories locally
Suppose you need to copy from src to dst, you can run following:
$ cp -r src/* dst/
$ tree dst
dst
├── a
│   └── a.txt
├── b
│   └── b.txt
├── c
├── s1 -> a
└── s2 -> b
Suppose you need to copy without a symbolic link but a real directory, you can run following:
$ cp -rL src/* dst/
$ tree dst
dst
├── a
│   └── a.txt
├── b
│   └── b.txt
├── c
├── s1
│   └── a.txt
└── s2
    └── b.txt

Remote

Copy files / directories from host to host. We need command scp.
Suppose you need to copy from src to dst, you can try tar scp combo:
# Tar the entire folder
$ tar cf src.tar.gz src/

# Scp to the remote side
# Remember to replace `localhost` and `pwd` with your own variables
$ scp src.tar.gz localhost:`pwd`/subdir/

# Untar the zip in `subdir/src.tar.gz` in folder `subdir`
$ tar -x -C subdir/ -f subdir/src.tar.gz

# Rename the `subdir/src` to `subdir/dst`
$ mv subdir/src subdir/dst

$ tree subdir/dst
subdir/dst/
├── a
│   └── a.txt
├── b
│   └── b.txt
├── c
├── s1 -> a
└── s2 -> b
Or, you can do them on the fly:
# Open a new terminal for the destination host and run following command
# Use NC to listen on any port (EG: 12345) and untar it on the fly
$ nc -l 12345 | tar xf -

# Open another new terminal for the source host and run following command
# Tar the entire folder and send them through netcat
# Remember to replace `localhost` and `pwd` with your own variables
$ tar cf - src/ | nc localhost 12345

# Verify in destionation host
$ mv src dst
$ tree dst
dst
├── a
│   └── a.txt
├── b
│   └── b.txt
├── c
├── s1 -> a
└── s2 -> b
Suppose you need to copy without a symbolic link but a real directory, you can run following:
# Scp to the remote side
# Remember to replace `localhost` and `pwd` with your own variables
$  scp -r src/* localhost:`pwd`/dst
$ tree dst
dst/
├── a
│   └── a.txt
├── b
│   └── b.txt
├── c
├── s1
│   └── a.txt
└── s2
    └── b.txt

2017/07/11

A study on SSL certificates

Background

Recently, I have a chance to deploy a real signed SSL certificate on company’s Gitlab server. By following this link from Gitlab, browser shows a lovely lock icon easily.
However, there is a trouble when configuring container registry with such certificate. Gitlab runners throw an error about unable to get issuer certificate. Just like this and that.
Eventually, I find out this is a problem from intermediate certificates and fix it with some struggles. Below are the glossaries I have crawled.

Glossaries

SSL certificates

aka X.509 certificate

Encoding

  • DER
    • a binary encoded certificate
  • PEM
    • BASE 64 ASCII encoded certificate
    • Contains line ----BEGIN----
Usually, you would say I have a DER encoded certificate instead of DER certificate.

Extension

  • CRT
    • Can be encoded with DER, PEM
    • Common on Linux
  • CER
    • Similar to CRT
    • Common on Window
  • KEY
    • Public key or private key

Actions

  • View
    • get human readable string
      • PEM
        • openssl x509 -in cert.xxx -text -noout
      • DER
        • openssl x509 -in certificate.der -inform der -text -noout
  • Transform
    • DER 2 PEM
      • openssl x509 -in cert.crt -inform der -outform pem -out cert.pem
    • PEM 2 DER
      • openssl x509 -in cert.crt -outform der -out cert.der
  • Combinations
    • Concatenate multiple certificates in one file
    • EG: Combine intermediate certificates with your certificate

CSR

  • Certificate Signing Request
  • A step before becoming a CER / CRT (Or signed by CA)
  • Contains information like Common Name, Organization Name etc
  • Decode
    • openssl req -in server.csr -noout -text

Intermediate certificate

  • Certificate(s) between your site and root certificate
    • Construct chain of trust
  • A proxy protecting the root certificate
  • Certificate ordering follow RFC4346
    • 1st Server certificate
    • Any intermediate follow each other one by one

TLS

A successor protocol or an enhancer version of SSL which support DV, OV, EV, BV etc.

References

2017/07/02

How to do list intersection efficiently in python

Question

In previous blog, I have investigated how to do array initialization efficiently. This time, I would like to investigate how to do intersections in Python efficiently.
For those need to read code immediately, here is the gist for this blog

Methods

Before digging in the implementations, let’s define what intersection is. By intersection, I mean finding same elements between 2 given lists.
A = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
B = [2, 4, 6, 8, 10]
# c = a ^ b
# which is c = [2, 4, 6, 8]
How do we get c = [2, 4, 6, 8] efficiently?

M1: double loop

A double loop for finding same elements
def doubleLoop():
    C = [a for a in A for b in B if a == b]

def doubleLoop2():
    C = []
    for a in A:
        for b in B:
            if a == b:
                C.append(a)
Noted
  • list comprehension vs append

M2: Set intersection

Cast A and B to set and do intersection.
setA = set(A)
setB = set(B)

def setAnd():
    C = sorted(setA.intersection(setB))

def setAnd2():
    # No sort
    C = setA.intersection(setB)

def setAnd3():
    # Inline construct
    C = sorted(set(A).intersection(set(B)))

def setAnd4():
    # Inline construct and no sort
    C = set(A).intersection(set(B))
Noted:
  • setAnd ensures the ordering sames with output from doubleLoop or doubleLoop2 as set has no concept of ordering
  • setAnd2 demonstrates the meaning of no ordering
  • setAnd3 demonstrates how slow for casting them in a run time manner
  • setAnd4 is similar to `setAnd2

Result

Results of running each implementations for 1000000 times
# Double loop
# 1.4873290062
# Double loop2
# 1.70801305771
# Set AND
# 0.525309085846
# Set AND2
# 0.199652194977
# Set AND3
# 0.982024908066
# Set AND4
# 0.657478094101

Conclusion

  • Always use set to do intersections. No matter the results need to be sorted or not, it is always faster than the double loop way.
  • List comprehension is always faster than append. Check out more from my previous blog

Explanation

setAnd vs setAnd2

  • setAnd do an extra sort comparing to setAnd2
    • As set has no ordering, it need to be cast to list before which is an extra cost

setAnd vs setAnd3

A performance checks between using a pre-cast sets setA and setB and runtime-cast sets set(A) and set(B). Even though the runtime version is slower, it is still faster than double loop obviously.

setAnd3 vs setAnd4

Same as setAnd vs setAnd2

Gotcha

  • Set’s intersection works charm on unique elements inside involved lists. Benchmark here may not be applied to the multi-set one
  • Set has no concept of ordering. If ordering is a must for you, you need to do it yourself.

2017/06/23

Tricks share: Edit multiple places at the same time on atom

About

As a developer, editors are one of tools I need to deal with all the time. Here are tricks I used on improving the typing experiences in atom

Trick 1: Edit multiple words at the same time

Given text below
Scenario Outline: eating
  Given there are <start> cucumbers
  When I eat <eat> cucumbers
  Then I should have <left> cucumbers

  Examples:
    | start | eat | left |
    |  12   |  5  |  7    |
    |  12   |  10 |  2   |
    |  20   |  5  |  15  |
Suppose you need to fix the numbers in start column to be 30
Double click those number with Left Ctrl pressed.
Type 30 to change all of highlighted words

Trick 2: Shortcut on changing identical words

Given text below
Scenario Outline: eating
  Given there are <start> cucumbers
  When I eat <eat> cucumbers
  Then I should have <left> cucumbers

  Examples:
    | start | eat | left |
    |  30   |  5   |  7    |
    |  30   |  10 |  2   |
    |  30   |  5   |  15  |
Suppose you would like to change cucumbers as oranges. Beside using replacement from Ctrl f, you can use Ctrl D selection.
Click cucumbers and press Ctrl D 3 times as there are 3 cucumbers.
Then, type oranges which changes all the highlighted words.

Trick 4: Edit rows at the same time

Scenario Outline: eating
  Given there are <start> cucumbers
  When I eat <eat> cucumbers
  Then I should have <left> cucumbers

  Examples:
    | start | eat | left |
    |  30   |  5  |  7   |
    |  30   |  10 |  2   |
    |  30   |  5  |  15  |
Suppose you would like add a column country with all same value Hong Kong.
Add a column country manually
Scenario Outline: eating
  Given there are <start> cucumbers
  When I eat <eat> cucumbers
  Then I should have <left> cucumbers

  Examples:
    | country | start | eat | left |
    |  30   |  5  |  7   |
    |  30   |  10 |  2   |
    |  30   |  5  |  15  |
Select rest of lines with left click while pressing Left Ctrl.
Type | Hong Kong for adding such value to highlighted columns
Fix the spacing in the headers by pressing spaces
Scenario Outline: eating
Given there are <start> cucumbers
When I eat <eat> cucumbers
Then I should have <left> cucumbers

  Examples:
    | country   | start | eat | left |
    | Hong Kong |  30   |  5  |  7   |
    | Hong Kong |  30   |  10 |  2   |
    | Hong Kong |  30   |  5  |  15  |

Round up…

Tricks for editing things at the same time are very common on coding. Let me know if you have any more tricks about that :D

2017/06/05

Run luci on openwrt docker image

About

As the old blog is so popular and readers may have problems on installation, I have found a new way for setup luci development with docker and openwrt image.

Prerequisites

Although I am using Window for this blog, it will be much easier if you try in Linux environment.
For Docker installation in Window, I recommend you to read my another blog to distinguish differences between docker-toolbox and native docker

Version checking

At the time on writing this blog, I am using Docker version 17.05.0-ce, build 89658be in WIN 10 platform. Actually, version and platform do not matter.
$ docker -v
Docker version 17.05.0-ce, build 89658be

Get the openwrt docker image

According to openwrt, you need following calls
docker import http://downloads.openwrt.org/attitude_adjustment/12.09/x86/generic/openwrt-x86-generic-rootfs.tar.gz openwrt-x86-generic-rootfs
Verification
docker images
REPOSITORY                           TAG                   IMAGE ID            CREATED             VIRTUAL SIZE
openwrt-x86-generic-rootfs           latest                2cebd16f086c        6 minutes ago       5.283 MB
NOTES:
  • For Window users, you may get errors from docker-engine. Make sure you are using Terminal from Docker which named as Docker QuickStart Terminal

Run the container with such image

$ docker run --rm -it openwrt-x86-generic-rootfs sh
NOTES:
  • --rm stop and remove such container from docker history after exiting the container
  • -it runs the container in an interactive manner

Linux users

Restart the uhttpd first.

(CONTAINER) / # /etc/init.d/uhttpd restart
bind(): Address already in use
Error: No sockets bound, unable to continue
(CONTAINER) / #  netstat -atn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
You may or may not need to restart such uhttpd service. However, you need to ensure such container is listening on port 80.

Get container’s IP

(CONTAINER) / # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02
          inet addr:172.17.0.2  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe11:2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:31 errors:0 dropped:0 overruns:0 frame:0
          TX packets:31 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2090 (2.0 KiB)  TX bytes:1866 (1.8 KiB)

Browse http://172.17.0.2 with your browser

Window users

Steps same with Linux users except the browsing step. Hopefully, you are not able to browse 172.17.0.2 without adding a route.

Find out the IP of the docker-engine VM

Reading banner from Docker QuickStart Terminal
                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/

docker is configured to use the default machine with IP 192.168.99.100
For help getting started, check out the docs at https://docs.docker.com

Start interactive shell
Here, 192.168.99.100 is the IP we want.

Launch a window CMD with administrator permission

Add route

C:\WINDOWS\system32>route ADD 172.17.0.0 MASK 255.255.0.0 192.168.99.100

Test with ping

C:\WINDOWS\system32>ping 172.17.0.2

Ping 172.17.0.2 (使用 32 位元組的資料):
回覆自 172.17.0.2: 位元組=32 時間<1ms TTL=63

172.17.0.2 的 Ping 統計資料:
    封包: 已傳送 = 1,已收到 = 1, 已遺失 = 0 (0% 遺失),
大約的來回時間 (毫秒):
    最小值 = 0ms,最大值 = 0ms,平均 = 0ms
Control-C
^C

Browse http://172.17.0.2 with your browser

Development

Mount files to the container

In order to do the development, you need to put your source codes inside the container by mounting them into the container.
Normally, below are 2 common mount locations.
/ # ls /usr/lib/lua/luci/
cacheloader.lua  config.lua       fs.lua           i18n/            model/           sys.lua          tools/
cbi.lua          controller/      http.lua         init.lua         sauth.lua        sys/             util.lua
cbi/             debug.lua        http/            ip.lua           sgi/             template.lua     version.lua
ccache.lua       dispatcher.lua   i18n.lua         ltn12.lua        store.lua        template/        view/
/ # ls /www/
cgi-bin/      index.html    luci-static/
As an example, I have mount 2 resources one.pdf and folder pixi
$ docker run --rm -it -p 8080:80 -v /c/Users/Mond/Documents/GitHub/pixi.js/:/www/luci-static/resources/pixi -v /c/Users
/Mond/Documents/one.pdf:/www/luci-static/resources/one.pdf  openwrt-x86-generic-rootfs sh

BusyBox v1.19.4 (2013-03-06 20:07:44 UTC) built-in shell (ash)
Enter 'help' for a list of built-in commands.

/ # ls /www/luci-static/resources/
bandwidth.svg    cbi.js           icons            one.pdf          wifirate.svg     xhr.js
cbi              connections.svg  load.svg         pixi             wireless.svg

Relevant blogs

References