I bumped into InsecureMag (https://www.helpnetsecurity.com/insecuremag-archive/) - the free magazine on security. As I'm interested in it I decided to check it out. It's totally free. But being lazy I wanted to have all the issues on my harddisk. As usual bash comes with help:
#!/bin/bash
for i in {1..49}; do
link="https://www.helpnetsecurity.com/dl/insecure/INSECURE-Mag-$i.pdf"
wget -c "$link"
done
Have a nice reading :)
Sunday, February 21, 2016
Raspberry Pi - Setting Static IP address
For Christmas I bought for myself :) Raspberry Pi and installed Raspbian and OpenELEC on this. For now I usually use it as low-energy uploader/downloader or photo batch processor.
I want to have remote access to it also from outside my home network. By default Raspbian uses DHCP to obtain IP address from my home DHCP server. As I need to forward ports from my ISP router to my Raspberry PI, I need to give it always static IP address. I've been messing around with it a bit until I figured it out. Here is the recipe:
1. Turn off dhcp client which will always override things
$ sudo update-rc.d dhcpcd disable
You can also do it old school way by removing execute right:
$ sudo chmod a-x /etc/init.d/dhcpcd
2. Add the following lines to /etc/network/interfaces (the bold ones are new)
auto lo
auto eth0
iface lo inet loopback
#iface eth0 inet manuala
iface eth0 inet static
address 192.168.1.100
gateway 192.168.1.254
netmask 255.255.255.0
3. We need /etc/resolv.conf with static dns entry. This file is always rewritten by resolvconf utility, so we need to change it. In /etc/resolvconf.conf we add one line:
name_servers=192.168.1.254
My nameserver is my gateway at the same time.
4. Restart networking services by:
$ sudo /etc/init.d/networking restart
Works for me. We can check if everything is ok running:
$ /sbin/ifconfig
I want to have remote access to it also from outside my home network. By default Raspbian uses DHCP to obtain IP address from my home DHCP server. As I need to forward ports from my ISP router to my Raspberry PI, I need to give it always static IP address. I've been messing around with it a bit until I figured it out. Here is the recipe:
1. Turn off dhcp client which will always override things
$ sudo update-rc.d dhcpcd disable
You can also do it old school way by removing execute right:
$ sudo chmod a-x /etc/init.d/dhcpcd
2. Add the following lines to /etc/network/interfaces (the bold ones are new)
auto lo
auto eth0
iface lo inet loopback
#iface eth0 inet manuala
iface eth0 inet static
address 192.168.1.100
gateway 192.168.1.254
netmask 255.255.255.0
3. We need /etc/resolv.conf with static dns entry. This file is always rewritten by resolvconf utility, so we need to change it. In /etc/resolvconf.conf we add one line:
name_servers=192.168.1.254
My nameserver is my gateway at the same time.
4. Restart networking services by:
$ sudo /etc/init.d/networking restart
Works for me. We can check if everything is ok running:
$ /sbin/ifconfig
Saturday, January 9, 2016
Checking the proxy servers
Sometimes we need some privacy on the web. So I needed good proxy servers. I googled a bit and found good list of them here:
http://proxylist.hidemyass.com/
But checking them one by one is boring, so I copied the interesed one into the file called list (copy/paste interesting fragments from web browser :):
198.169.246.30|80|flagCanada
200.180.32.58|80|flagBrazil
211.144.81.68|18000|flagChina
84.45.246.38|3128|flagUK
95.168.217.24|3128|flagCzechia
There are some environment variables called http_proxy and https_proxy. We can set them system-wide to use proxy like this:
export http_proxy="http://host:port"
or
export http_proxy="http://username:pass@host:port"
if proxy requires autentication.
So to check if proxy works we just need to set this env. variable and try to download something with wget. We can do this in the loop and get this list as input
--- cut ---
#!/bin/bash
n=1
for l in $(cat list); do
HOST=$(echo $l | cut -f1 -d '|')
PORT=$(echo $l | cut -f2 -d '|')
export http_proxy="http://${HOST}:${PORT}"
export https_proxy=$http_proxy
wget https://www.eff.org/files/https-everywhere2.jpg -T 10 -O ${n}.jpg
unset http_proxy
unset https_proxy
n=n+1
done;
--- cut ---
-T means timeout
-O means name of the output file
This way I got several files numberd as proxy server position on the list. The 0 size ones are non-working proxies.
-rw-r--r-- 1 pi pi 0 Jan 9 17:56 10.jpg
-rw-r--r-- 1 pi pi 20216 Jun 12 2013 1.jpg
-rw-r--r-- 1 pi pi 0 Jan 9 19:23 2.jpg
-rw-r--r-- 1 pi pi 20216 Jun 12 2013 3.jpg
-rw-r--r-- 1 pi pi 0 Jan 9 19:23 4.jpg
-rw-r--r-- 1 pi pi 0 Jan 9 19:23 5.jpg
http://proxylist.hidemyass.com/
But checking them one by one is boring, so I copied the interesed one into the file called list (copy/paste interesting fragments from web browser :):
198.169.246.30|80|flagCanada
200.180.32.58|80|flagBrazil
211.144.81.68|18000|flagChina
84.45.246.38|3128|flagUK
95.168.217.24|3128|flagCzechia
There are some environment variables called http_proxy and https_proxy. We can set them system-wide to use proxy like this:
export http_proxy="http://host:port"
or
export http_proxy="http://username:pass@host:port"
if proxy requires autentication.
So to check if proxy works we just need to set this env. variable and try to download something with wget. We can do this in the loop and get this list as input
--- cut ---
#!/bin/bash
n=1
for l in $(cat list); do
HOST=$(echo $l | cut -f1 -d '|')
PORT=$(echo $l | cut -f2 -d '|')
export http_proxy="http://${HOST}:${PORT}"
export https_proxy=$http_proxy
wget https://www.eff.org/files/https-everywhere2.jpg -T 10 -O ${n}.jpg
unset http_proxy
unset https_proxy
n=n+1
done;
--- cut ---
-T means timeout
-O means name of the output file
This way I got several files numberd as proxy server position on the list. The 0 size ones are non-working proxies.
-rw-r--r-- 1 pi pi 0 Jan 9 17:56 10.jpg
-rw-r--r-- 1 pi pi 20216 Jun 12 2013 1.jpg
-rw-r--r-- 1 pi pi 0 Jan 9 19:23 2.jpg
-rw-r--r-- 1 pi pi 20216 Jun 12 2013 3.jpg
-rw-r--r-- 1 pi pi 0 Jan 9 19:23 4.jpg
-rw-r--r-- 1 pi pi 0 Jan 9 19:23 5.jpg
Sunday, October 18, 2015
Inside my video player ASUS HDP-R1 - CPU usage without top
For several years I've been using ASUS HDP-R1 video player for my media files. Recently I've had samoe problems playing big mkv file. I decided to dig into the player to see what's inside and check the CPU usage while playing movies.
We can telnet to the player:
Venus login: root
warning: cannot change to home directory
BusyBox v1.1.3 (2011.05.25-05:45+0000) Built-in shell (ash)
Enter 'help' for a list of built-in commands.
/ #
uname -a gives us:
/ # uname -a
Linux Venus 2.6.12.6-VENUS #323634 Wed May 25 13:40:36 CST 2011 mips unknown
And let's check the CPU:
/ # cat /proc/cpuinfo
system type : Realtek Venus
processor : 0
cpu model : MIPS 24K V7.8
BogoMIPS : 269.51
wait instruction : yes
microsecond timers : yes
tlb_entries : 32
extra interrupt vector : yes
hardware watchpoint : yes
ASEs implemented : mips16
VCED exceptions : not available
VCEI exceptions : not available
I just wanted to check CPU usage however neither top nor mpstat commands are available.
My first plan was to compile top binary from sources linking it staticly but it's not x86 architecture so I had no other hardware to do that. Of cource as you suppose there are no compilers available in this box.
I came up with the following solution - read cpu usage directly from the /proc filesystem with /bin/sh.
Let's try:
/ # cat /proc/stat
cpu 4916 3748 25087 1879421 2633 326 459 0
cpu0 4916 3748 25087 1879421 2633 326 459 0
intr 2046721 0 0 86168 40259 0 3704 1916590 0
ctxt 10942464
btime 1388589011
processes 6008
procs_running 3
procs_blocked 0
We are interested in the first line giving us the summary cpu usage of all the cores (in this case only cpu0). The magic numbers are:
Now:
current_user
current_system
current_idle
in 2 seconds (current values become the previous ones):
prev_user = current_user
prev_system = current_system
prev_idle = current_idle
Now we can calculate the cpu usage with this formula:
current_total = current_user + current_system + current_idle
prev_total = prev_user + prev_system + prev_idle
diff_total = current_total - prev_total
diff_idle = current_idle - prev_idle
cpu_usage (%) = (diff_total - diff_idle)/diff_total*100
Now we can write the shell script to calculate every 2 seconds:
#!/bin/sh
p_user=0
p_system=0
p_idle=0
p_total=0
while true; do
#current
l=$(cat /proc/stat | grep "cpu ")
c_user=$(echo $l | cut -f2 -d' ')
c_nice=$(echo $l | cut -f3 -d' ')
c_system=$(echo $l | cut -f4 -d' ')
c_idle=$(echo $l | cut -f5 -d' ')
#calc
c_total=$(($c_idle+$c_user+$c_system))
d_total=$(($c_total-$p_total))
d_idle=$(($p_idle-$p_idle))
totalminusidle=$(($d_total-$d_idle))
#cpu=$((($totalminusidle/$d_total)*100))
echo -n $totalminusidle
echo -n "#"
echo -n $d_total
echo
sleep 2
#prev
p_user=$c_user
p_nice=$c_nice
Psystem=$c_system
p_idle=$c_idle
p_total=$c_total
done
We save it in writable /tmp space as top.sh. It will be there until device reboot.
It's not the perfect script. I can't use bash feaures here as I don't have one. We have a little problem here. The cpu usage will be totally inaccurate because the shell doesn't support floating point arithmetic. And on this machine I don't have any tools to do that (like bc, dc). There is awk but it always returns nan, trying to do some math operations:
/ # awk 'BEGIN{print 2+2}'
nan
So the only not ellegant idea I came up with was to list the values used in the final CPU equation and then do the calculations somewhere else, in spreadsheet for example :). (yes it's not realtime). So when we run the script we got: (we omit the first row as we didn't have the prev values yet):
/tmp # sh ./top.sh
31401#2110151
33#232
34#233
29#228
31#232
34#233
33#232
32#231
31#232
33#232
33#232
32#231
29#230
Feeding spreadsheet with them we have:
a b a/b*100
When the player is just turned on and idle the cpu usage is about 12-14%
While playing mkv hd movie (720p) the cpu usage is about 20%
While playing mkv full hd movie (1080p) the cpu usage is about 25%
That's all for now
We can telnet to the player:
Venus login: root
warning: cannot change to home directory
BusyBox v1.1.3 (2011.05.25-05:45+0000) Built-in shell (ash)
Enter 'help' for a list of built-in commands.
/ #
uname -a gives us:
/ # uname -a
Linux Venus 2.6.12.6-VENUS #323634 Wed May 25 13:40:36 CST 2011 mips unknown
And let's check the CPU:
/ # cat /proc/cpuinfo
system type : Realtek Venus
processor : 0
cpu model : MIPS 24K V7.8
BogoMIPS : 269.51
wait instruction : yes
microsecond timers : yes
tlb_entries : 32
extra interrupt vector : yes
hardware watchpoint : yes
ASEs implemented : mips16
VCED exceptions : not available
VCEI exceptions : not available
I just wanted to check CPU usage however neither top nor mpstat commands are available.
My first plan was to compile top binary from sources linking it staticly but it's not x86 architecture so I had no other hardware to do that. Of cource as you suppose there are no compilers available in this box.
I came up with the following solution - read cpu usage directly from the /proc filesystem with /bin/sh.
Let's try:
/ # cat /proc/stat
cpu 4916 3748 25087 1879421 2633 326 459 0
cpu0 4916 3748 25087 1879421 2633 326 459 0
intr 2046721 0 0 86168 40259 0 3704 1916590 0
ctxt 10942464
btime 1388589011
processes 6008
procs_running 3
procs_blocked 0
We are interested in the first line giving us the summary cpu usage of all the cores (in this case only cpu0). The magic numbers are:
- user: normal processes executing in user mode
- nice: niced processes executing in user mode
- system: processes executing in kernel mode
- idle: twiddling thumbs
- iowait: waiting for I/O to complete
- irq: servicing interrupts
- softirq: servicing softirqs
Now:
current_user
current_system
current_idle
in 2 seconds (current values become the previous ones):
prev_user = current_user
prev_system = current_system
prev_idle = current_idle
Now we can calculate the cpu usage with this formula:
current_total = current_user + current_system + current_idle
prev_total = prev_user + prev_system + prev_idle
diff_total = current_total - prev_total
diff_idle = current_idle - prev_idle
cpu_usage (%) = (diff_total - diff_idle)/diff_total*100
Now we can write the shell script to calculate every 2 seconds:
#!/bin/sh
p_user=0
p_system=0
p_idle=0
p_total=0
while true; do
#current
l=$(cat /proc/stat | grep "cpu ")
c_user=$(echo $l | cut -f2 -d' ')
c_nice=$(echo $l | cut -f3 -d' ')
c_system=$(echo $l | cut -f4 -d' ')
c_idle=$(echo $l | cut -f5 -d' ')
#calc
c_total=$(($c_idle+$c_user+$c_system))
d_total=$(($c_total-$p_total))
d_idle=$(($p_idle-$p_idle))
totalminusidle=$(($d_total-$d_idle))
#cpu=$((($totalminusidle/$d_total)*100))
echo -n $totalminusidle
echo -n "#"
echo -n $d_total
echo
sleep 2
#prev
p_user=$c_user
p_nice=$c_nice
Psystem=$c_system
p_idle=$c_idle
p_total=$c_total
done
We save it in writable /tmp space as top.sh. It will be there until device reboot.
It's not the perfect script. I can't use bash feaures here as I don't have one. We have a little problem here. The cpu usage will be totally inaccurate because the shell doesn't support floating point arithmetic. And on this machine I don't have any tools to do that (like bc, dc). There is awk but it always returns nan, trying to do some math operations:
/ # awk 'BEGIN{print 2+2}'
nan
So the only not ellegant idea I came up with was to list the values used in the final CPU equation and then do the calculations somewhere else, in spreadsheet for example :). (yes it's not realtime). So when we run the script we got: (we omit the first row as we didn't have the prev values yet):
/tmp # sh ./top.sh
31401#2110151
33#232
34#233
29#228
31#232
34#233
33#232
32#231
31#232
33#232
33#232
32#231
29#230
Feeding spreadsheet with them we have:
a b a/b*100
| 33 | 232 | 14,224137931 | ||||||||||||||||||||||||||||||
| 34 | 233 | 14,5922746781 | ||||||||||||||||||||||||||||||
| 29 | 228 | 12,7192982456 | ||||||||||||||||||||||||||||||
| 31 | 232 | 13,3620689655 | ||||||||||||||||||||||||||||||
| 34 | 233 | 14,5922746781 | ||||||||||||||||||||||||||||||
| 33 | 232 | 14,224137931 | ||||||||||||||||||||||||||||||
| 32 | 231 | 13,8528138528 | ||||||||||||||||||||||||||||||
| 31 | 232 | 13,3620689655 | ||||||||||||||||||||||||||||||
| 33 | 232 | 14,224137931 | ||||||||||||||||||||||||||||||
| 33 | 232 | 14,224137931 | ||||||||||||||||||||||||||||||
| 32 | 231 | 13,8528138528 | ||||||||||||||||||||||||||||||
| 29 | 230 | 12,6086956522 |
When the player is just turned on and idle the cpu usage is about 12-14%
While playing mkv hd movie (720p) the cpu usage is about 20%
While playing mkv full hd movie (1080p) the cpu usage is about 25%
That's all for now
Wednesday, February 18, 2015
Bc as CPU benchmark for multicore cpu
Referring to this http://tuxshell.blogspot.com/2009/08/bc-as-cpu-benchmark.html post, I figured out the simple way how to test speed of multicore cpu system or multi cpu system. This command:
$ time echo "scale=5000; a(1)*4" | bc -l
will only be executed on one CPU core. However we can use GNU parallel command to execute it many times on multicore cpu. First we create benchmark script b.sh as follows:
#!/bin/bash
echo "scale=$1; a(1)*4" | bc -l
The first argument is number of digits to calculate. Now we can run in parallel (4 times) this way
$ time parallel ./b.sh ::: 2000 2000 2000 2000
We can determine the number of cores with this command:
$ cat /proc/cpuinfo | grep processor | wc -l
4
If your cpu supports hyper threading - this will give you the number of logical cores, not only the physical ones
$ time echo "scale=5000; a(1)*4" | bc -l
will only be executed on one CPU core. However we can use GNU parallel command to execute it many times on multicore cpu. First we create benchmark script b.sh as follows:
#!/bin/bash
echo "scale=$1; a(1)*4" | bc -l
The first argument is number of digits to calculate. Now we can run in parallel (4 times) this way
$ time parallel ./b.sh ::: 2000 2000 2000 2000
We can determine the number of cores with this command:
$ cat /proc/cpuinfo | grep processor | wc -l
4
If your cpu supports hyper threading - this will give you the number of logical cores, not only the physical ones
Subscribe to:
Posts (Atom)