2017/01/19

How to do array initialization on python efficiently

Question

Suppose we would like to create an array with 100 elements.
2 types of elements will be tested
  • None
  • integers from 0 to 99

Methods

3 ways to initialize an array
  • By append

def byAppend():
    a = []
    for i in xrange(100):
        a.append(None)

def byAppendInit():
    a = []
    for i in xrange(100):
        a.append(i)
  • By list comprehension

def byListComprehension():
    [None for i in xrange(100)]

def byComprehensionInit():
    [i for i in xrange(100)]
  • By multiplication

def byMultiplication():
    [None] * 100

def byMultiplicationInit():
    a = [None] * 100
    for i in xrange(100):
        a[i] = i

Results


import timeit
print 'By append'
print(timeit.timeit('byAppend()', setup="from __main__ import byAppend"))
print 'By append init'
print(timeit.timeit('byAppendInit()', setup="from __main__ import byAppendInit"))
print 'By list comprehension'
print(timeit.timeit('byListComprehension()', setup="from __main__ import byListComprehension"))
print 'By list comprehensionInit'
print(timeit.timeit('byComprehensionInit()', setup="from __main__ import byComprehensionInit"))
print 'By byMultiplication'
print(timeit.timeit('byMultiplication()', setup="from __main__ import byMultiplication"))
print 'By byMultiplication init'
print(timeit.timeit('byMultiplicationInit()', setup="from __main__ import byMultiplicationInit"))
Output from consoles
C:\Python27\python.exe C:/Users/Mond/PycharmProjects/hackercup/round1/test.py
By append
10.2452175728
By append init
9.83314044088

By list comprehension
4.45988583823
By list comprehensionInit
4.37547215318

By byMultiplication
0.840903578289
By byMultiplication init
5.83873665989

Conclusion

  • If you need to create a “fixed sized” array only, multiplication is the fastest.
  • If you need to initialize array with some constants, list comprehension performs better.
  • Never ever using the way for appending.

Gotcha

For 2d array initialization, you must be careful not to do it via multiplication.
Below are codes for initializing a 2D array with 3 rows and try to add a string hello to the first array

a = [[]] * 3
a[0].append('hello')
print a
What a will be output is
[['hello'], ['hello'], ['hello']]
instead of
[['hello'], [], []]
Therefore, we can see initializing 2D array via multiplication is not a good way as they are all the same array even though arrays seems to be multiplied (initialized).
The proper way is using list comprehension
# The underscore `_` means I do not care what value it is
a = [[] for _ in xrange(3)]

a[0].append('hello')
print a
Then, a will be output as
[['hello'], [], []]

2017/01/09

Clean up cache inactive ports

As there is a shortage on my mac book, I need to free storage from mac ports
sudo port -f clean --all all
sudo port -f uninstall inactive

How to update npm modules

So, when was the last time you have been updated your npm modules?
After I have been updated for few of machines, I believe it is quite useful and necessary for me to record steps for doing so to save my life. Here we go.

1. Get a list of outdated packages

npm provides a useful command for us to do so. Below is an example for listing global modules which are outdated. Of cause, you can run that in your local project once you got the idea.
$ npm -g outdated
Package                 Current  Wanted  Latest  Location
eslint                   3.12.2  3.13.0  3.13.0  /opt/local/lib > eslint
eslint-config-airbnb     13.0.0  14.0.0  14.0.0  /opt/local/lib > eslint-config-airbnb
eslint-plugin-jsx-a11y    2.2.3   3.0.2   3.0.2  /opt/local/lib > eslint-plugin-jsx-a11y
eslint-plugin-react       6.8.0   6.9.0   6.9.0  /opt/local/lib > eslint-plugin-react
npm                     2.15.11   4.0.5   4.0.5  /opt/local/lib > npm
Here is the documentation about npm outdated. Briefly speaking, you should take care versions in the middle column.
NOTE:
  • Current column shows versions you own now.
  • Wanted column shows latest versions you can get after considering your package dependencies
  • Latest column shows the latest available version from corresponding projects

2. Update the npm modules

As my npm is managed by macport, I will let macport to upgrade my npm.
$ sudo port upgrade outdated
Install rest of the modules by specifying name and version together.
$ sudo npm -g install eslint@3.13.0 eslint-config-airbnb@14.0.0 eslint-plugin-jsx-a11y@3.0.2 eslint-plugin-react@6.9.0
NOTE:
  • My global modules requires a root privilege
Sometime, there are warnings about mismatched dependencies.
npm WARN eslint-config-airbnb@14.0.0 requires a peer of eslint@^3.13.0 but none was installed.
npm WARN eslint-config-airbnb@14.0.0 requires a peer of eslint-plugin-import@^2.2.0 but none was installed.
npm WARN eslint-config-airbnb-base@11.0.1 requires a peer of eslint@^3.13.0 but none was installed.
npm WARN eslint-config-airbnb-base@11.0.1 requires a peer of eslint-plugin-import@^2.2.0 but none was installed.
npm WARN eslint-plugin-jsx-a11y@3.0.2 requires a peer of eslint@^2.10.2 || 3.x but none was installed.
npm WARN eslint-plugin-react@6.9.0 requires a peer of eslint@^2.0.0 || ^3.0.0 but none was installed.
Don’t be afraid. Make sure they are really a problem first.
$ npm -g list
If you can read those warnings again, they are really problems. Semver explains syntax in the version, for example ^, x etc. I really suggest you to go through that article before solving the warnings.
Basically, they are talking about version mismatched. Take a serious look and install the required packages and problems will be solved.

3. End

You can double check the modules by calling npm outdated again and see whether there are missing modules.
In my opinion, the troublesome parts are resolving versions between packages. This is a smooth and easy process after resolving them.

4. More about npm’s dependencies

Actually, I have posted a question about missing dependencies in StackOverflow here. There are links for you to further study npm’s dependencies. (Hopefully, :D)

5. Search with semver

The best way to know whether the semver includes certain version is testing.
For example, I would like to know how many versions can be search for eslint with ^3.17.x
mondwan@mondwan-All-Series:~/Documents/git/abc$ npm -g view eslint@^3.17.x | grep ^eslint
eslint@3.17.0  
eslint@3.17.1

Change logs

  • 2017-03-14: Add a command for searching package’s version
  • 2017-03-11: Add a reference from StackOverflow

2016/10/28

Createa a bootable USB stick for ubuntu on ubuntu

Things below are just an alternative. There is an official way to do this.

Get a USB stick

Make sure you have an usb stick with enough storage. Mine is 8 GB.

Format the USB Stick

There are plenty options to do this. My solution is using the built in one disks
  • Click the USB device
  • Click the square button
  • Select EXT4 as the type of parition
  • Done

Download Ubuntu ISO

Bootable USB

  • Install and Use UnetBootin
sudo apt-get install unetbootin
  • Launch UnetBootin
  • Select the Ubuntu ISO
  • Select your USB drive
  • OK
  • Done
Now, you have a Bootable USB for ubuntu :D

2016/02/26

Lua string.match vs string.gmatch

Overview

Just now, I spent a while on coding with string.gmatch and string.match. In my opinion, they are not quite intuitive for their usage. So, I tried to placed my example here for future references.

Code example

  • gmatch aims for global search while match ends at 1st match.
> line = "Jan  1 08:17:02 ABC hotplug: [OK] ifup: 'lan' => 'br-lan'"
> pat = '([^ ]+)'

# End at Jan
> return line:match('([^ ]+)')
Jan

# Equivalent to split by ' '
> for w in line:gmatch(pat) do print(w) end
Jan
1
08:17:02
ABC
hotplug:
[OK]
ifup:
'lan'
=>
'br-lan'
  • gmatch does not good at dealing with ^
> line = "Jan  1 08:17:02 ABC hotplug: [OK] ifup: 'lan' => 'br-lan'"
> pat = '^([A-Z][a-z][a-z])'
# Able to get Jan
> =line:match(pat)
Jan
# Unable to get anything
> for w in line:gmatch(pat) do print(w) end
>

Reference

  1. http://stackoverflow.com/questions/28593385/lua-string-match-vs-string-gmatch
  2. http://lua-users.org/wiki/StringLibraryTutorial

2016/02/09

Notes on studying LTN12

Background
Suppose you don’t know what LTN is, LTN stands for Lua technical note.
Below are notes dropped from LTN12 which is the 12th documents talking about filters, sink , source, chain and pump.
Source codes from the author is available here
Interpretations
XXX: My interpretations may or not may correct. Let me know if there are any misunderstands.
Functions accept successive chunks of input, and produce successive chunks of output.

Mathematically speaking,

Suppose f1, f2, ..., fn are filters s.t. F = f1 + f2 + ... + fn
Suppose x1, x2, ... xn are input s.t. X = x1 + x2 + ... + xn

F(x) == f1(x1) + f2(x2) + ... + fn(xn)
  • What is a chain?
A function combines two *similar* functions. By *similar*, I mean same function signatures and return data type.

Mathematically speaking,

Suppose f1, f2, ..., fn are functions s.t. C = f1 + f2 + ... + fn
Suppose x1, x2, ... xn are input s.t. X = x1 + x2 + ... + xn

C(x) == f1(x1) + f2(x2) + ... + fn(xn)
  • What are sources and sinks?
Filters form internal nodes inside an arbitrary *data flow*. *Data* in *data flow* means things shared between producer and consumer while *flow* means the direction of that things which is from *Producer* to *Consumer*

Given picture above, we can interpret *Sources* and *Sinks* as initial and final nodes of the *data flow*.
  • What is pump?
The driving force between internal nodes (filters)
  • Example application?
-- Goal: convert any input text ends with \r\n

-- source.file() returns a kind of filter
-- normalize() returns a kind of filter
-- source.chain() shows how to combine *similar* filters
-- input is a kind of *source*
input = source.chain(source.file(io.stdin), normalize("\r\n"))

-- sink.file() returns a kind of filter
-- output is a kind of *sink*
output = sink.file(io.stdout)

-- Driving force between input and output
-- Or, start the jobs
pump(input, output)

2016/01/15

Rescue vim process from other dead terminal session

Due to the fact that my office’s PC always down with no reason and leaving bunch of vim zombie sessions in my remote machine, I am tired for rescuing them times by times.
Finally, I found a way to resolve this issue.

Setup

  • OS: Ubuntu 14.04
  • Command required: reptyr and ps

Find PIDs of those zombie vim session

$> ps aux | grep vi

Rescue them

$> reptyr PID

Permission Problem

In case you got error like below
$ reptyr 1851
Unable to attach to pid 1851: Operation not permitted
The kernel denied permission while attaching. If your uid matches
the target's, check the value of /proc/sys/kernel/yama/ptrace_scope.
For more information, see /etc/sysctl.d/10-ptrace.conf
Fix this by editing configuration from 1 to 0
$vi /etc/sysctl.d/10-ptrace.conf

 kernel.yama.ptrace_scope = 0
Reload rule
$ sudo sysctl -p /etc/sysctl.d/10-ptrace.conf
Done.

References

2015/10/02

octave on MAC OS X

About

octave is a GNU project which is a kind of alternative for MatLab. Below covers how I install this stuff in my MBA.

Tutorials

Since I am using port, I follow guides from the official wiki here. Below are the actual commands I have typed.
sudo port selfupdate
sudo port upgrade outdated

sudo port install octave
sudo port install octave +gui

sudo port install arpack -accelerate+atlas
If you can run below commands successfully, congratulation. This post is end. However, if you have gotchas like me, checkout below notes.

Gotcha

  • There are errors like missing fortran compiler
      port install gcc49
    
Gcc other than 49 may work too. But I test with this one.
  • There are errors like missing tex
     port install texlive-basic
     port install texlive-latex
    
After running above calls, run again
   port install octave
It should fix all the problems.

Open octave

You can simply type octave in your spotlight and octave will be there.

2015/07/17

How to use git to do development #2

Recall

Last time I have shown you my configurations. This time I am going to share my visions about git merge and git rebase.

What is merge and what is rebase?

If you are confused about these 2 operations, I suggest you to go through slides here.
For merge, it adds a node (commit) in order to join two parallel branches (with a parent node some where).
For rebase, it moves one of the parallel branches onto another one. “NO” new nodes will be added during this operation.

Workflow?

By workflows, I mean how you tactic and regulate branches in your project. For example, master is a production branch while hotfix are branches that branched off from master. Those hotfix branches must be merged or rebased eventually etc.
The concept of workflow is a kind of regulations or agreements between developers.
IMHO, you do not need this if you are one man band :) Do whatever you like.
If you are not sure which one to start, I suggest you to go through this one first. Even you do not agree with the writer, he gives you hints for modeling your workflow with git.

So, which workflow I have adopted?

Actually, I am a fans of terminal. I would like to keep all my history as linear as possible which is easier for me to read. Therefore, I will embrace git rebase instead of git merge.
However, this does not means I do not git merge at all. Since result of git merge and git rebase are the same eventually, I will use git merge if I am running out of time.
Alternatively, if you are good at using GUI tools like SourceTree, go for git merge instead of git rebase since those GUI is able to tell you commits coming from which branch.