Walkthrough the basic of redis

Goal of writing this blog

• Figure out the database type of Redis
  • Say, mysql is kind of a RDBMS (Relational database management system)
• Figure out the data persistence on Redis
  • Say, in RDBMS, data will be persistent unless we delete them explicitly
• Figure out terminologies in Redis on saving data
  • Say, in RDBMS (mysql etc), we have table / record / schema

Database type of Redis

According to the official introduction,

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker.

Here are comparisons between RDBMS and Redis:

• https://www.javatpoint.com/redis-vs-rdbms

The role of Redis in our big picture is acting as a cache or message broker. So, in the following, I will focus on these 2 areas.

Data persistence

In my scenario, redis is acting as a cache or message broker. Therefore, I need to concern following configurations:

• Keep them in memory, and do not saving data into files (Secondary memory).
• Size of cache
• Timeout of cache

Keep them in memory not files

• Disable RDB & AOF
• Keep data as long as redis live

save ""
appendonly no
persistence-available no

Size of cache

• Number of database?
  • Stick to the default number?
  • https://www.mikeperham.com/2015/09/24/storing-data-with-redis/
• Size of the cache?
  • Related configurations: maxmemory

Timeout of Cache

• Related Redis Command: TTL / EXPIRE
• Related configuration: maxmemory-policy

Final configuration should be

database 4
save ""
appendonly no
persistence-available no
maxmemory 3GB
maxpolicy volatile-ttl

Terminologies

Redis stands for?

It means REmote DIctionary Server

Source: https://redis.io/topics/faq

RDB vs AOF

• RDB stands for Redis database backup

• AOF stands for Append only file

• https://stackoverflow.com/a/52492740/2647757

• https://redis.io/topics/persistence
• http://oldblog.antirez.com/post/redis-persistence-demystified.html

Keys and Values

If we were to apply this data structure concept to the relational world, we could say that databases expose a single data structure - tables. Tables are both complex and flexible. There isn’t much you can’t model, store or manipulate with tables.
...
Keys are how you identify pieces of data. We’ll be dealing with keys a lot, but for now, it’s good enough to know that a key might look like users:leto. One could reasonably expect such a key to contain information about a user named leto. The colon doesn’t have any special meaning, as far as Redis is concerned, but using a separator is a common approach people use to organize their keys.

...
Values represent the actual data associated with the key. They can be anything.

Source: https://www.openmymind.net/redis.pdf

LRU

less recently used (LRU)

Source: https://redis.io/topics/lru-cache

Useful references

• https://www.openmymind.net/redis.pdf
• https://redis.io/documentation

​

Use python's pathlib to implement file path across OS

Why file path is an issue across platform

Nowadays, there are 3 common OS, which are Windows, Linux, and MacOS. Since Linux, and MacOS shares same file path’s implementation, developers do not need to worry about file path issues on between them. The problem is between Windows, and others.

In Windows, file path can be expressed in either 2 formats:

• \folder1\folder2\file1.txt (Window format)
• /folder1/folder2/file1.txt (POSIX format)

The difference is the direction of the slash. A side note is that file path in Linux / MacOS is expressed in POSIX format ONLY.

Actually, if applications in Windows can eat them both properly, I don’t need to write this blog since I can simply written in 2nd format at all. According to my experiences, even applications in Windows accepts either one of them only. It is a try and error process for picking a correct format for each applications.

Back to the title of this blog, file path issue will be bigger if your implementation runs across platform. For example, a library (Python zipfile library) may accepts POSIX format but the format of input file path may be Window format at all. So, the question is whether python provide library for us to do this conversion or not.

A side note on python zipfile. This library accepts both kind of formats since py3.8

pathlib

Pathlib is a builtin library for us to fix this problem. Below demonstrates 2 ways on conversion via such library. For other advance usage, please consult the library here.

Get POSIX file path from different input format

import pathlib
winPath = r'\workspace\xxx\test_fixture\user-restore-success.zip'
posixPath = '/workspace/xxx/test_fixture/user-restore-success.zip'
pWIN = pathlib.PureWindowsPath(winPath)
pPOSIX = pathlib.PureWindowsPath(posixPath)
pWIN.as_posix()
#'/workspace/xxx/test_fixture/user-restore-success.zip'
pPOSIX.as_posix()
#'/workspace/xxx/test_fixture/user-restore-success.zip'

Get Window file path

str(pWIN)
str(pPOSIX)

Notes

• Always favor PureWindowsPath when doing conversion

PureWindowsPath is able to convert between Window path, and POSIX path. PurePosixPath, on the other hand, is not able to do so since Backslash () is a valid filename in POSIX path.

For detail, please refer to the discussions here

• Bug in Path.resolve() on Windows platform

According to here, Path.resolve() in windows cannot return the absolute file path if such file is not existed at first. If you need a reliable way to get absolute path of a file right now, use os.path.abspath instead

​

A gotcha on python's round method (Banker's rounding)

Before studying the gotcha, let’s have a quiz, and see whether you will fall into the trap on round() or not.

Try to round below 22 float numbers to nearest integer, and see whether you can get them all correct. Below is the quiz in python.

floatNumbers = [1.1, 1.2, 1.3, 1.4, 1.49, 1.5, 1.51, 1.6, 1.7, 1.8, 1.9]
roundNumbers = list(map(round, floatNumbers))
formatFloatNumbers = ['%.02f' % num for num in floatNumbers]
formatRoundNumbers = ['%.02f' % num for num in roundNumbers]
print('Q', formatFloatNumbers)
print('A', formatRoundNumbers)

floatNumbers = [2.1, 2.2, 2.3, 2.4, 2.49, 2.5, 2.51, 2.6, 2.7, 2.8, 2.9]
roundNumbers = list(map(round, floatNumbers))
formatFloatNumbers = ['%.02f' % num for num in floatNumbers]
formatRoundNumbers = ['%.02f' % num for num in roundNumbers]
print('Q', formatFloatNumbers)
print('A', formatRoundNumbers)

Below is the answer.

Q ['1.10', '1.20', '1.30', '1.40', '1.49', '1.50', '1.51', '1.60', '1.70', '1.80', '1.90']
A ['1.00', '1.00', '1.00', '1.00', '1.00', '2.00', '2.00', '2.00', '2.00', '2.00', '2.00']
Q ['2.10', '2.20', '2.30', '2.40', '2.49', '2.50', '2.51', '2.60', '2.70', '2.80', '2.90']
A ['2.00', '2.00', '2.00', '2.00', '2.00', '2.00', '3.00', '3.00', '3.00', '3.00', '3.00']

The gotcha is on 2.50 case. Result of rounding 2.50 will be 2.00 instead of 3.00. And, this is a default, and correct behavior on rounding.

Why?

This rounding behavior is named as Round half to even

In short, this is a limitation from hardware. Since I am not going to go through the hardware’s limitation here, I suggest you to read this to have a full picture on them.

In fact, I would like to focus on another naming of this rounding: Banker's Rounding

Banker’s Rounding

Beside surprised by the behavior, I am also surprised on another name of this rounding technique, which is Banker's Rounding.

As the name stated, it is a rounding used by banks. So, why would banks adopt this strange rounding at all? Interestingly, this is because of Fairness.

Earning money is a job of banks (and for all companies). But, banks need to do it in a legal, and fair way. Rounding is essential on bank’s deals. For example, time deposits, and credit card’s loan etc. To round up / round down a number to the nearest integer, there are 9 cases (from 0.1 to 0.9).

Numbers will be evenly distributed among these 9 cases in statistical point of view. Banks round down numbers in 0.1 ~ 0.4 while round up numbers in 0.6 ~ 0.9. Chances for banks paying more or lesser is even (fairness) for these 8 cases. The problematic case is 0.5. If banks round up on 0.5, it will pay much more (5 cases), investors must be angry about that. Vice versa, if banks round down on 0.5, clients must be mad about that.

As a result, banks adopt round half to even to fix this fairness issues. After adopting this rounding, a single 0.5 case will be divided into 2 cases (even or odd cases). Banks will have equal chances on rounding up / down numbers. That’s why it is also named as Banker's rounding

IMO, this is a really clever trick to make every parties happy.

References

• http://chayingluen.blogspot.com/2013/06/blog-post.html
• https://docs.python.org/3.6/library/functions.html?highlight=round#round
• https://docs.python.org/3.6/tutorial/floatingpoint.html#tut-fp-issues
• https://en.wikipedia.org/wiki/Rounding#Round_half_to_even
• https://wiki.c2.com/?BankersRounding

​

A stupid way to semi automate steps for creating Merge Request on gitlab

Hello all. I have been away for writing a blog a while. Today, I would like to share how I semi automate steps for creating a MR on gitlab via command line.

Show me source codes

This is simply a script for filling information, for regular MR contents, and open a browser, which is a GUI for me to verify the contents, to fire MR. It, however, saves my time for filling boring and redundant MR contents. Here we go.

import subprocess

# Where is your project: NAMESPACE/REPO_NAME
MR_URL_ARGVS = {
    'NAMESPACE': 'mondwan',
    'REPO_NAME': 'scripts_for_creating_gitlab_mr',
}

# What will be the arguments for the MR URL
MR_SETTINGS = {
    # 'source_branch': 'feature_branch_1',
    'target_branch': 'master',
    # 'force_remove_source_branch': 'true',
    'title': 'New%20MR',
    # User ID in the gitlab
    'assignee_id': '1158561',
}

# The base url
MR_URL_BASE = \
    'https://gitlab.com/{NAMESPACE}/{REPO_NAME}/merge_requests/new?'.format(
        **MR_URL_ARGVS
    )

# Fill up source branch by current branch name via system git command
currentBranchName = subprocess.check_output(
    'git rev-parse --abbrev-ref HEAD'.split(' ')
).strip('\n')
MR_SETTINGS['source_branch'] = currentBranchName

# Create params list for the MR which will be pasted at the END of URL
getParams = '&'.join([
    'merge_request%%5B%s%%5D=%s' % (k, v) for k, v in MR_SETTINGS.items()
])

# The final url
url = MR_URL_BASE + getParams

# Launch a browser for given url with command line in ubuntu
print 'gnome-open "%s"' % url

Below explain how it works part by part.

How it works

If you studied carefully, gitlab accept a hand made URL for creating Merge request.

We can prefill whatever information in that MR if we can craft a specific URL.

Below is how to craft the first part of the URL, where variables are the namespace and repository name.

# Where is your project: NAMESPACE/REPO_NAME
MR_URL_ARGVS = {
    'NAMESPACE': 'mondwan',
    'REPO_NAME': 'scripts_for_creating_gitlab_mr',
}

# The base url
MR_URL_BASE = \
    'https://gitlab.com/{NAMESPACE}/{REPO_NAME}/merge_requests/new?'.format(
        **MR_URL_ARGVS
    )

The next question is how can I fill text into a field like title, contents, or source branch. Get Params is the answer.

For example, if you append string merge_request%5Btitle%5D=New%20MR at the end of the URL, title of that page will be New MR.

Below shows how to create a MR where title is New MR, assigned to an user with id 1158561, and set master to be the target branch

# What will be the arguments for the MR URL
MR_SETTINGS = {
    # 'source_branch': 'feature_branch_1',
    'target_branch': 'master',
    # 'force_remove_source_branch': 'true',
    'title': 'New%20MR',
    # User ID in the gitlab
    'assignee_id': '1158561',
}



# Create params list for the MR which will be pasted at the END of URL
getParams = '&'.join([
    'merge_request%%5B%s%%5D=%s' % (k, v) for k, v in MR_SETTINGS.items()
])

You may wonder how can I know what should be the key in order to fill up a field. The answer is looking them up via your browser’s console. For example, I would like to look up the key of source branch. I will see something like that in the console after watching the corresponding DOM element from the HTML.

As you can see in the diagram, we can locate merge_request[source_branch] in the HTML. That’s the key (source_branch) we need.

For me, it will be convenient if my script is able to fetch the name of current branch and put that in to the source branch field.

# Fill up source branch by current branch name via system git command
currentBranchName = subprocess.check_output(
    'git rev-parse --abbrev-ref HEAD'.split(' ')
).strip('\n')
MR_SETTINGS['source_branch'] = currentBranchName

Finally, I would like to launch a browser through command line by using gnome-open

 # The final url
url = MR_URL_BASE + getParams

# Launch a browser for given url with command line in ubuntu
print 'gnome-open "%s"' % url

Then, I can copy and paste that URL into the command line. A browser, for me it is Firefox, will be jumped out eventually in Ubuntu.

The repository for this script

https://gitlab.com/mondwan/scripts_for_creating_gitlab_mr

Alternative

For advance developers, you may try to automate the entire process by using official APIs. For python developers, you can try to code them up with this cli library.

Although I have not tried to code this alternative, I guess this is much more time consuming than my approach, in term of time spent on development, listed above. One thing, however, I can sure is that this alternative is much more efficient than my approach.

Change logs

20181125: Add alternative section, refine some of the wordings

​

"%.*s" in printf()

Background

Currently, I am dealing with regex example in C.  There is a abnormal printf command.

......

printf ("'%.*s' (bytes %d:%d)\n", (finish - start),  to_match + start, start, finish);

......

Ouput:

$& is '1 is nice 2' (bytes 5:16)

I am wondering why there are 3 "%s" format options but there are 4 parameters to map those options.

Dig deeper...

To make the story short, "%.*s" is where the amazing happen. This option take 2 arguments: length of the string and the starting point of a string. (XXX: so ?) OK. Take a look the example code below.

#include <stdio.h>



int main(int argc, const char *argv[])

{

        char a[] = "0123456789";



        int len, offset;

        len = 4;



        for (offset = 0; offset < 3; offset++) {

                printf("%.*s\n", len, a + offset);

        } 



        return 0;

}

Output:

0123
1234
2345

In other word, "%.*s" allows you to print a subset of a string without doing tedious jobs like following.

char *b = (char *) malloc(4+1);
memcpy(b, a+offset, len);
b[len+1] = '\0';
printf("%s\n",b);

Introduction to enyoJS

Background

Diverged from xtuple paragraph, enyo is one of xtuple dependencies. Or, in other words, xtuple use this framework to build their client side UI.

Introduction

enyo js
http://enyojs.com/about/
http://enyojs.com/get-enyo/#Bootplate
Difference between enyo and jQuery
jQuery focus on manupliating HTML elements while enyo focus on encapsulating HTMLs as a module and reuse them

Dig deeper

1. Cloning their "blootplate" and play around

The 1st question I would like to ask is what blootplate is.
After reading a while from their reference, I understand several things.
a) You can produce a web page with a minify source easily if you can make use of their shell script and directory structure.
b) If you have not interested on building an APP they claimed like me(well... it means a HTML5 webpage), you can simply read source/App.js. This file control what "debug.html" look like.

2. Diving back to their developer guide

Well... their developer guide make more sense than then point 1 I just pass-through.
I understand 2 more keywords in enyo namespace:
kind: A kind is a JavaScript constructor for an object that's been defined using the enyo.kind method. (That's why you can write `new App()`)
enyo.Control: A kind provided by enyo's core. A factory for users like us to create enyo.instance with user-defined properties.
enyo.Control() VS enyo.kind:
var t = enyo.Control(.....);
t.renderInto(document.body);
enyo.kind({name:'test',kind:enyo.Control...});
var k = new test();
k.renderInto(document.body);


In other word, enyo.kind allows you to publish your enyo.instance to a global namespace as a constructor while enyo.Control return a pointer of your enyo.instance.

3. Walk through their example

Here is their tutorial and here is my github repository, which folk from enyo.bootplate, to play with the provided example.
Notes:
1. "published":
properties insides "published" will be given a setter and getter method.
2. "event handler":
There are 2 parameters, object that's the source of the event and the event object, will be passed into the event handler.
3. "jsonp example":
Since Twitter have updated their APIs, the original example does not work. I have coded another apps in order to try their Ajax wrapping. Not Bad :D

4. Summery

It is fun for me to code with enjo JS. However, there are no too much online sources for you. If you get problems with enjo JS, you better be able to solve it yourself........

PS: Currently, # of jquery tag on stackoverflow is 389,821 while enyo is 120....

5. Useful links

http://enyojs.com/docs/2.2.0/

https://github.com/enyojs/enyo/wiki/Encapsulation-in-Enyo

https://github.com/enyojs/enyo/wiki/Style-Guide

https://github.com/enyojs/enyo/wiki/Event-Handling

Library for drawing Graph on web: jsPlumb.js

Background
jsPlumb is a open-source javascript library for drawing graph on web platform. FYI, please go in their official website http://jsplumbtoolkit.com/home/jquery.html. There are many demos and relevant documentations for you to start your development.

My opinions
To be honest, I have not read all of the documentations. However, in beginner point of view, their documentation is not programmer friendly and examples are quite hard to follow.

After working for few hours, I have written a test website to play around with this library. Here is the link: https://github.com/mondwan/jsplumb_test

Two things I would like to drop down after building up the above test page
1. No idea for why css position:relative is required for jsPlumb.draggable() function properly. Actually, I have reported it as a bug to jsplumb on github. However, they claim that this is not a bug.

2. jsPlumb.draggable() is not a pure wrapper which means jsplumb element cannot be updated correctly if you are using 3rd party dragging library. (I have struggle a while...)

Learning u-boot Part 2

My goal
I need to some how implement a feature for loading firmwares via ethernet cable only (IE fading out no SERIAL CONSOLE RS232). A suggestion from my supervisor is starting from bootloader. Basically, that's why this thread coming out :D

1. Direction
According to the /doc/README.Netconsole, u-boot provides an access via netcat. So, this is my starting point.

2. Enable netconsole
Netconsole is disabled by default of my board setting due to minimize the files size of that executable file (I guess). So, the following recording how I enable netconsole.

According to
/common/device.c:240~242
240 #ifdef CONFIG_NETCONSOLE
241         drv_nc_init ();
242 #endif

It is clear that we need to define that constant in order to enable netconsole.

Add lines
#define CONFIG_NETCONSOLE /* enable nc */
into /includes/configs/<board>.h

Recompile the u-boot binary and loading into the device.
PS. How to load bootloader into the device will not be discussed here.

For now, bootloader should be able to use nc.

3.  Follow up
Keep following the README.Netconsole, execute following commands via serial console.
setenv nc setenv stdin nc\; setenv stdout nc
setenv ncip <YOUR_SERVER>
setenv netmask <YOUR_SERVER_NETMASK>
saveenv
run nc

In your server run the following script file,
#! /bin/bash

[ $# = 1 ] || { echo "Usage: $0 target_ip" >&2 ; exit 1 ; }
TARGET_IP=$1

stty -icanon -echo intr ^T
nc -u -l 6666 < /dev/null &
nc -u ${TARGET_IP} 6666
stty icanon echo intr ^C

# interupt by CTRL+T
# $1 is your device ip

For now, magic happen :D
You can interact with the device as if you are using serial console cable.

4. Summery
I can setup a network link between embedded device and my pc. The rest of the tasks are putting things together. For example, when and how to enable netconsole on the embedded device. I will cover these stuffs later on (I hope) since I have no progress till now.

Learning u-boot Part1

Firstly, I have no idea how many parts I will write on this topic "Learning uboot".

Introduction
Namely part 1, let's take a look what u-boot is. Official website for u-boot is http://www.denx.de/wiki/U-Boot/WebHome.
u-boot is a kind of bootloader on embedded device (eg. routers). Mother Board initialization is what it suppose to do.

What happen after powering on your device
Power on -> bootloader (ROM) -> firmware (STORAGE somewhere) -> user-space interface ( COMMAND LINE or something like that)

My goal
My job is implementing a function for loading firmwares via cat5 cable (IE no SERIAL CONSOLE RS232). A suggestion from my supervisor is starting from bootloader. Basically, that's why this thread coming out :D

1. Browsing the u-boot project
Since I have no expereince on related topics, I can only read documentations, tutorials, googling etc....
Here is my summery for files useful for my goal.

Assume prefix <u-boot project> is presented
/include/configs/cavium_cns3000.h   /* u-boot console default environment variable value */
/board/cavium/cns3000/vega.c           /* board manufacturer info */
/common/env_common.c                   /* where environment variable will be assign */
/common/devices.c                            /*  How devices will be enabled */
/common/main.c                                /* How Interactive  Shell implemented */
/net/net.c                                            /* How tftp, arp, ping implemented */

2a. Execution flow
cpu/..../start.S
board/.../lowlevel_init.S
board/.../lowlevel_init.c
cpu/.../start.S
lib_.../board.c

2b. Command flow
/common/main.c
main_loop() -> readline() -> run_command() ->
/common/command.c
find_cmd() -> corresponding function()

3. Short summery
Compile issue:
Make sure your board model is supported by u-boot. (Ask your board manufacturer what to do for bootloader issue)
U-boot provides you a way to build relevant configuration files. (`make <board>_config`)
Make sure your tool chain directory is presented in $PATH.

Script issue:
Following commands are useful for scripting in u-boot.
setenv; setting environment variable
saveenv: saving environment variable permanently on rom
printenv: printing environment variable
run <var>: run the variable contents
coninfo: list of available devices


Script example:
setenv loader tftpboot 0x800000 loader.bin\;erase 0x10000000 +\$(filesize)\;cp.b 0x800000 0x10000000 \$(filesize)\;reset
run loader


Reference
Illustrate the u-boot project hierarchy in chinese
http://gordenhao.pixnet.net/blog/post/29189867-%5B%E8%B3%87%E6%96%99%5D-u-boot-%E7%B0%A1%E4%BB%8B
How to compile uboot
http://home.educities.edu.tw/fushiyun2000/gameconsole_snk_neogeox_hack_compile_official_uboot_program.htm
Execution flow
https://sites.google.com/site/myembededlife/Home/u-boot/u-boot-startup-sequence

"va_*()" in c programming

Quick and dirty summery
va_*() are macros helping you to implement functions with dynamic arguments numbers.

Dig deeper
what is va_*()  ?
They are va_start(), va_arg(), va_end() respectively. Those macros will be used if you want to implement a function with vary arguments. For example, printf().

va_start(); /* call before va_arg(), otherwise va_arg() cannot function properly*/
va_arg(); /* return 1st, 2nd, 3rd.... arguments for 1st, 2nd, 3rd ...calls */
va_end(); /* call before returning value */

There are some examples codes on the reference link.

Reference
http://tw.myblog.yahoo.com/jw!kg_rIFWTHgO4kRtDoy15QxVeWQ--/article?mid=21&prev=22&l=f&fid=5

"extern" in c programming

Quick and dirty summery
1. extends the visibility for a variable or function
2. While using with variable, it declares a variable but not defining it.
3. Special case for (2) is "extern int bar=0;", it treat as definition


Dig deeper
Declaration := A variable or function signature exists somewhere in the program but the memory is not allocated for them.
Definition := Despite the stuff declaration does, it also allocates memory for that variable/function.

PS.
/* function */
int foo (int arg1, int arg2); /* Declaration */
int foo (int arg1, int arg2){
    /* Definition */
     return 0;
}
/* variable */
extern int bar; /* Declaration */
int bar=0; /* Definition */



Reference:
http://www.geeksforgeeks.org/understanding-extern-keyword-in-c/

"volatile" in c programming

Quick and dirty summery
"volatile" is a keyword to force compile to fetch a variable's value from memory every-time it reads this variable during the variable's life time.

Dig deeper
Since there maybe optimizations on the C compiler (maybe true or not), compiler will cache the value of a variable if there are no obvious side effects on that variable. "volatile" is a keyword to tell compiler not to do optimization on that variable.



Reference:
http://cboard.cprogramming.com/c-programming/73163-volatile-keyword.html