Sharing on customizing ubuntu image and create USB stick

Background

Currently, installing the Ubuntu and setting up those machines are my jobs. In order to minimize the duplicated workloads (eg: install apache, python etc), I decided to build a custom image which bundled with those necessary packages at first instead of using the official one directly.

Prerequisite

• ubuntu customization kit (aka UCK)

sudo apt-get install uck

• unetbootin

sudo apt-get install unetbootin

• ubuntu image

A base image for customization.

Steps

• follow instructions here

the link above guides you how to use UCK and create a USB drive from this image.

• Alternative for creating USB

In case, the above link does not work for you. UnetBootin is an alternative for you.

• Further customization

You do not need to add all the things at once. You can always replaced the base image with the one you have built. However, I am still looking for a VCS for this process. Let me know if you have any idea.

References

1. http://www.linux.com/learn/tutorials/739139-roll-your-own-customized-ubuntu-with-uck
2. http://www.howtogeek.com/109736/how-to-create-a-custom-ubuntu-live-cd-or-usb/
3. http://www.geekyprojects.com/ubuntu/build-your-own-custom-ubuntu-livecd/

​

Customize MAC OS Terminal

To be honest, I am not satisfy with the default terminal on MAC OS X. Below are my suggestions for heavy working with terminal on OSX.

ITERM2

Make sure you have installed iTerm2 for replacing the default terminal application.

Most of my works rely on VIM. However, the mouse option will not work on default terminal. So, I change to iTerm2.

Terminal Sexy

Here provides you bunch of the colour profiles for your terminal. Personally, I love hybird, pretty and pastel and x dot share.

BetterTouchTool

Besides the configuration on iTerm, I need short cuts for calling out, maximising, minimising, etc the iTerm application. I am using this for achieving this purpose.

Word jumping

Besides commands from here, I have setup another commands for word jumping on terminal.
Answer from superuser helps me a lot.

​

Notes on using macport

What is macport

Macport is a package manager tools on MAC OSX. It works like what apt-get does on Debian system.

Tricks

To save my time in the futures, here are experiences which learnt from painful try and error process.

BASH AUTO competition

There is no auto completion (IE Tab will not work) by default. Please read this for setting up an auto completion.

List of common commands

• port search PKG or port search —regex ‘PKG‘

A command for you to search packages.

• port install PKG

Install a package from macport

• port uninstall PKG

Uninstall a package from macport

• port select PKG VERSION

Select a version for a package. Sometime, packages come with more than one version (py27, py3). This command allows you to select which version will be used when you call python instead of calling python2.7 explicitly.

• man port

A command for reading the manual page of port.

​

Tricky inside the nltk implementation

In the previous assignment, we have been tried on using uni-gram and bi-gram powered by nltk.

Of cause, I can finish the assignment. However, Implementations for methods score_ngram and freq under class *CollocationFinder are quite tricky. Cost for calling them unwisely is O(n^2). Trust me. This is not a fun experience :D

• Problems

Let me show you my algorithm before pointing out the reasons behind O(n^2)

# Cal uni-gram
# uc is an instance of class FreqDist
for gram in uc.items():
   unigram_scores.append({
        'words': gram[0],
        'score': uc.freq(gram[0])

    })

Well. Codes above are easy to understand and seems to flawless for computing the probability of an uni-gram. However, their cost is O(n^2).

• Why?

After digging in to the source code of nltk, below function is the root cause for O(n^2).

def N(self):
        """
        Return the total number of sample outcomes that have been
        recorded by this FreqDist.  For the number of unique
        sample values (or bins) with counts greater than zero, use
        ``FreqDist.B()``.

        :rtype: int
        """
        return sum(self.values())

My codes call freq() for every possible uni-gram. Cost for this operation is O(n). Then,
N() is called by every freq(). Cost for N() is O(n). As a result, cost is O(n * n) = O(n^2).

• How to solve?

Read codes below.

+N = uc_freq.N()

 # Calculate the probability for unigram
# uc_freq is an instance of FreqDist
-for gram in uc.items():
+for gram in uc_freq.items():
+    val = float(uc_freq[gram[0]]) / N
     unigram_scores.append({
         'words': gram[0],
-        'score': uc.freq(gram[0])
+        'score': round(val, 2)
     })

To summarise, we need to eliminate the side effect for N() during the for loop operations.

Therefore, I calculate the N outside the for loop operation and compute the probability myself instead of calling freq().

Although cost for N() is O(n) and the cost of for loop operation is still O(n), they are not multiplied. As a result, they are still running with cost n O(n) = O(n).

​

Analyze the trend of singing happy birthday in Mong Kong

Claim

Umbrella Revolution is a memorable and a breakthrough of the Hong Kong’s protests. Although I am focusing on (or trying to) analyzing the phenomenons from this protest which does not involved the democracy related issues, I love and support the Yellow Ribbon’s side.

Song of Happy birthday

This is a funny way and becomes a trend for protesters to repel the harassment from the opponents. Protesters sings the song of happy birthday to reply any insane acts from opponents. Here is an example for showing how this song works in this protest. Although it is ridiculous, it keeps both sides in peace.

How this trend diffused ?

Recap what we learnt from the lectures:

Processes of social diffusion of new practices:
1) New ideas and social practices are introduced by notable example (e.g. by very heavy advertising)
2) Initially, the rate of adoption is slow because new ways are unfamiliar, customs resist change, and results are uncertain
3) As early adopters convey more information about how to apply the new practices and their potential benefits, the innovation is adopted at an accelerating rate
4) After a period in which the new practices spread rapidly, the rate of diffusion slows down
5) The use of the innovation then either stabilizes or declines, depending upon its relative functional value

Information from the facebook, the trend originated from here (around 1:10) which posted on Oct 4. Views clicks (till now 2014/10/17) is 107,271 and around 1,045 shares. Facts meet point 1.

Below graphs shows the trend for keywords like “生日” (happy birthday) “示威” (protests) in google trend.

You can see trends for keywords “生日” on Oct 4 and Oct 5 are more or less the same. (Point 2 and Point 3). Suddenly, there are huge climbs on Oct 6 (Point 4). The trend decreased after Oct 6 (Point 5)

Summary

After comparing the theory from lecture notes with the facts, I learn and understand better the steps involved in social diffusion.

​

Shortcuts on MAC OS Terminal

To save my life, I would like to document down the shortcuts for default MAC OS terminal.

• Switch between terminal tabs

Command + Arrow left or right

• Page Up / Down

fn + Arrow Up or Down

• Jump to the Top or Bottom of the terminal

fn + Arrow Left or right

• Move one line up or down

Command + Arrow Up or Arrow Down

• Jump to the head of the input line

Ctrl A

• Jump to the end of the input line

Ctrl E

• Remove all the characters before the current pointer

Ctrl U

• Remove all the characters after the current pointer

Ctrl K

• Remove a word before the current pointer

Ctrl W

• Navigate a command line word by word

Ctrl + Arrow Left or Right

​

Tell you how to open the same pdf twice

For opening the same pdf twice,

Open the PDF for the first time -> Check out Menu bar -> Press Window -> Press New window

For showing those two pdf window in the same screen together,

Menu bar -> Window -> Tile -> Horizontally or Vertically

Reference

​

Notes about Sentiment polarity

Recently, we are going through the materials about Sentiment Polarity. This is really an interesting topic. Here are few notes about this topic.

• Dimensions

We need aspects to quantify a word’s characteristic. Here are the 3 aspects positive, objective and negative for telling the word’s characteristic (in digits).

If we claim this word is positive, we believe this word will be used on telling something good. On the other hand, if we claim this word is negative, we believe this word will be used on telling something bad. However, if we claim this word is objective, we believe that this word tells us nothing about any subjective attitude.

• Graphical representation of Sentiment Polarity

There is a website which show you how to represent a word in 3 dimensions spaces.

Point to note that is they are using triangular graph instead of Cartesian coordinate system.

If you are not familiarize this type of presentation, here are my interpretations.

* As much as objective, as less as subjective value

Fact is a fact. There are no good or bad for a fact since this is an objective stuff. Actually, the triangular shape is telling you this behavior.

* Subjective value is either positive or negative or neither of them (IE both of them are zero)

Despite the objectiveness controls the level of subjective value, subject value must be either positive or negative or both of them are zero.

* What's next

After you can categorize the wording, you can get a bunch of digits. Then, you can do further analysis by using k-clustering, term weighting extra.

Update Note 1 (2014/10/11): Beautify the blog’s contents

​

Django series: Tricks on doing Test Driven Development

Preface

This post will not talk about how to write TDD in django. Instead, it covers how to do this HAPPIER (in my opinion).

In case, you do not know how to do TDD under the django framework, please take a look here.

Tricks

• How to run test cases?

python manage.py test

• How to colorize the above output?

Take a look here

Personally I am using the pyrg

• How to change the TEST RUNNER?

Actually, you can change the test runner of django (EG: nose2) instead of using the default one. Take a look here.

Note that you should write this statement TEST_RUNNER='djnose2.TestRunner' instead of TEST_RUNNER=djnose2.TestRunner in order to make this work.

• How to capture the output from the running tests (eg: pprint, logging etc) with COLOR?

pyrg -- manage.py test -- -B

The difficulty is how to pass options to your test runner.

The first -- tellspyrg that following (manage.py test -- -B) MUST BE arguments instead of OPTIONS. The second -- tells manage.py the -B MUST BE the ARGUMENT instead of the OPTIONS. Then, you can pass the -B options to your custom test runner (I am using nose2).

​

To the world of social networking

Hello world! First blog post about this course and this blogger.

After attending first few lecture talks, I can feel the power and the potential of studying social networking. It is amazing that people invent intrinsic ways to analyze and model User Generated Contents by using Natural Language Processing and probability theory.

Actually, I am really interest on how modern social medias like facebook, google plus analyze user's behavior and how to make use of such kind of data to do advertisements pushing and promotions efficiently.

What I learnt from lecture 3 allows me to analyze users' content. Conceptually, by using term weighting, we are able to convert data from text based context to digits. Then, by using either or both of the vector space model and k-clustering, we are able to classify the type of the user context. Once we are able to categorize type of content, the remaining stuff is simply coding.

k-cluster

Besides the implementations, I am really excited about tricks on improving the accuracy of the classifications and how can those big companies profit from them. Hopefully, the following lectures will cover those issues one by one.

Django series: About django-admin.py

Following documents what I have walked through from here

• Relationship between django-admin.py and manage.py

manage.py is a wrapper of django-admin.py which take cares the following before delegated to django-admin.py:

* It puts your project’s package on sys.path.
* It sets the DJANGO_SETTINGS_MODULE environment variable so that it points to your project’s settings.py file.
* It calls django.setup() to initialize various internals of Django.

• List of helpful commands
  • dumpdata, output all data in database related to the given apps
  • inspectdb, create a model for each table inside the database
  • loaddata, load data from fixture to the database
  • flush, erase all data from the database

​

Java TDD = PROJECT + ANT + JUNIT

About

This blog is about what issues I have faced and tackle on writing a JAVA project with TDD (Test driven development). As you may not know, I am not an experienced Java developer. All the java coding skills came from a course 2 years ago.

Project

My goal is building up an application for fetching concert’s information from EVENTFUL. Although there is an official java binding provided, some of it’s dependencies (eg: XML) are not supported on ANDROID device by default. So, I am going to make a new one.

ANT

For brevity, I will say this tool works like grunt or makefile which ease the burdens of testing and compile your source code.

Typically, it required a makefile named as build.xml

Structure of the build.xml

XML := 1 * Project
Project := 1~N * Target
Target := 1~N * Task

FAQ

• Different between value and location on properties tag
• How to colour the ant’s output

JUNIT

A popular testing framework. To be honest, all I need is the fabulous assertions.

FAQ

• Installation on MAC BOOK AIR (MAC OS)

Command sudo port install junit works well. Make sure you have updated your port database if you are fail to issue the above command.

• Compilation on command line

PATH is the most difficult issue in this part. Make sure you have included both junit.jar and hamcrest-core.jar on compilation.

• How to run JUNIT on command line

2 points to note. The first one is make sure you have included all jar files on your java class path. The second one is how you call your test class. Below is an example

/opt/local/share/java/junit.jar:/opt/local/share/java/hamcrest-core.jar:.:./com/ideafactory/dataquery/test/PerformerTest.class org.junit.runner.JUnitCore com.ideafactory.dataquery.test.PerformerTest

​

Install luci on ubuntu 12.04

Abstract

After struggling around 1 hour, I have successfully installed luci on my ubuntu VM. For future reference, here is the steps I have tried.

Dependencies

• lua
• cmake
• uci
• luci

# lua

Installation

    $> sudo apt-get install lua5.1 liblua5.1-0-dev

Notes

Hopefully, you can run lua interpreter on your terminal by typing lua directly on terminal for now on.

# cmake

Installation

$> sudo apt-get install cmake

# uci

Installation

Edited from here

# Some extra package are required:
$> sudo apt-get install cmake lua5.1

# The libubox library is required. It should to compiled from source.
# To do this, first you have to get the source from git:  
$> git clone git://nbd.name/luci2/libubox.git libubox
$> cd libubox

# Please follow the next steps to build libubox:
$> mkdir build
$> cd build
$> cmake ..
$> make ubox

# Install libubox:
$> sudo mkdir -p /usr/local/include/libubox
$> sudo cp ../*.h /usr/local/include/libubox
$> sudo cp libubox.so /usr/local/lib
$> sudo ldconfig

# Get UCI source from git:
$> git clone git://nbd.name/uci.git uci
$> cd uci

# Please follow the next steps to build uci:
$> mkdir build
$> cd build
$> cmake ..
$> make all

# Install uci:
$> sudo mkdir -p /usr/local/include/uci
$> sudo cp ../uci.h ../uci_config.h /usr/local/include/uci
$> sudo cp ../uci_blob.h ../ucimap.h /usr/local/include/uci
$> sudo cp libuci.so /usr/local/lib
$> sudo cp uci /usr/local/bin
$> sudo cp lua/uci.so /usr/local/lib/lua/5.1/
$> sudo ldconfig

# Testing:
$> mkdir test
$> cat > test/test << EOF
> config 'test' 'abc'
> option 'test_var' 'value'
> EOF

$> uci -c `pwd`/test show test
Output:
test.abc=test
test.abc.test_var=value

$> uci -c `pwd`/test set test.abc.test_var=foobar
$> uci -c `pwd`/test commit test

$> uci -c `pwd`/test show test
Output:
test.abc=test
test.abc.test_var=foobar

# luci

Installation

# According to the official news,
# luci's repository has been merged with openwrt.
# Below are updated commands for downloading their sources.
$> git clone https://github.com/openwrt/luci.git
$> cd luci
$> git checkout luci-0.12
# Remove builtin folders if fail
$> rm -fr contrib/uhttpd

# compilataion
$> make runshell

Notes

1. This is not a permanent installation

Above installation is not a permanent one, you need to run above commands for every terminal session in order to use luci.

Actually, missing environment variables are the root cause for this phenomenon. For advanced Linux users, I strongly recommend you put those environment variables (named with LUA_*) into your ~/.bashrc after calling make runshell so that you do not need to run them again and again.

Write down following contents at the end of ~/.bashrc

# make sure you have update the home directory :)
export LUA_PATH="/home/mondwan/Documents/git/luci/host//usr/lib/lua/?.lua;/home/mondwan/Documents/git/luci/host//usr/lib/lua/?/init.lua;$LUA_PATH;"
export LUA_CPATH="/home/mondwan/Documents/git/luci/host//usr/lib/lua/?.so;$LUA_CPATH;"
export LUCI_SYSROOT='/home/mondwan/Documents/git/luci/host'
export LD_LIBRARY_PATH='/home/mondwan/Documents/git/luci/host/usr/lib:'

2. Missing iwinfo.h

If it tells you about missing iwinfo.h, please read this link. Try to revert those files will be good enough. Below are extracted commands from the link.

$> git checkout 89678917~1 contrib/package/luci/Makefile
$> git checkout 89678917~1 modules/admin-full/src/luci-bwc.c

Test installation

Before doing anything, here is the script for testing whether you have installed luci successfully or not.

$> lua
> UCI = require 'luci.model.uci'
> UCI.cursor() 

if you can run them without errors you have accomplished the installation.

Further reading

• Development on luci

References

* http://www.wakoond.hu/2013/06/using-uci-on-ubuntu.html

Update history

Update on 26/1/2015: Fix the incorrect directory structure CSS
Update on 16/2/2015: A new guide to compile luci
Update on 12/3/2015: Update text format and add notes about make runshell
Update on 21/5/2015: Add a reference forurther reading
Update on 2/6/2015:  Update bashrc contents
Update on 3/8/2015:  Update notes about installing uci
Update on 4/2/2016:  Update documents for UCI installation

Update on 26/8/2016: Update make commands for uci and luci installation