Sunday, December 8, 2013

Linux Shell script to rename files from CAPS to small letters

This is a small script to rename multiple files from CAPITAL to small letters. Ok, why we require to do this? Recently we bought a new cam which is naming all photos in capital and it is very much difficult for me to change extension of these photos from caps to small to re-size/modify photos using mogrify Linux tool.

#!/bin/bash
#Author: Surendra Anne
for j in *
do
COUNT=$(ls -1 | wc -l)
if [ $j == $(basename $0) ]
then
echo "can not move  the script file which is running this"
else
org_file=$j
new_file=$(echo $j | tr [A-Z] [a-z])
echo "Moving $org_file --> $new_file"
mv $org_file $new_file
fi
done

Comment if there is any issue with this script.

Friday, December 6, 2013

Linux Shell script to close/open ports on a firewall/Iptables

This is a small script which will takecare of blocking and unblocking ports by asking user about his desire. Just copy this code to your system and change permissions and start executing it.
#!/bin/bash
#Author: Surendra Anne(surendra@linuxnix.com)

PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
clear
echo -e "############################\n\n\nPresent ports opened on this machine are
$(iptables -nL INPUT | grep ACCEPT | grep dpt)
\nCompleted listing...\n\n\n#########################"
read -p "To open port enter open, to close etner close) " OPT1
if [[ "$OPT1" == open ]]
then
read -p "Please enter your desired port number to open: " PORT1
if [[ "$PORT1" =~ [0-9]{1,6} ]]
then
iptables -D INPUT $(iptables -nL INPUT --line-numbers | grep "$PORT1" | grep REJECT | awk '{print $1}')
iptables -A INPUT -m tcp -p tcp --dport "$PORT1" -j ACCEPT && { service iptables save;service iptables restart; echo -e "Ports opend through iptables are \n$(iptables -nL INPUT | grep ACCEPT | grep dpt)"; }
else
echo "Please enter a valid port(0-65000)"
fi
elif [[ "$OPT1" == close ]]
then
read -p "Please enter your desired port number to close: " PORT1
        if [[ "$PORT1" =~ [0-9]{1,6} ]]
        then
iptables -D INPUT $(iptables -nL INPUT --line-numbers | grep "$PORT1" | grep ACCEPT | awk '{print $1}')
        iptables -A INPUT -m tcp -p tcp --dport "$PORT1" -j REJECT && { service iptables save;service iptables restart; echo -e "Ports closed through iptables are \n$(iptables -nL INPUT | grep REJECT | grep dpt)"; }
        else
        echo "Please enter a valid port(0-65000)"
        fi
else
echo "Please enter only open or close..! Exiting script now";exit 1
fi


Output: For closing a port

[root@localhost ~]# bash block-unblock-ports.sh 

############################


Present ports opened on this machine are
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80 

Completed listing...


#########################
To open port enter open, to close etner close) close
Please enter your desired port number to close: 80
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]
iptables: Applying firewall rules:                         [  OK  ]
Ports closed through iptables are 
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80 reject-with icmp-port-unreachable 

For opening port:

[root@localhost ~]# bash block-unblock-ports.sh 


############################


Present ports opened on this machine are
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22 

Completed listing...


#########################
To open port enter open, to close etner close) open
Please enter your desired port number to open: 81
Bad argument `7'
Try `iptables -h' or 'iptables --help' for more information.
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]
iptables: Applying firewall rules:                         [  OK  ]
Ports opend through iptables are 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22 
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:81 






Saturday, November 2, 2013

PFOTD: Python ord() function examples

Today our inbuilt function is ord() which is useful for converting a single character to its corresponding ASCII value.

Need of ord() function in Python:

Some times it is require to convert a string to unicode ASCII value and this inbuilt function will give python this capability.

ord() function syntax:

ord('char')

Examples: Let us start with ord() function with some examples and see how we can use them in Python coding.

Example1: Convert a char to its corresponding ascii value.

>>> ord('a')

Output:

97
>>>

Example2: Convert a string to ascii values.

>>> string='Hello World'
>>> for i in string:
... print ord(i)
...

Output:

72
101
108
108
111
32
87
111
114
108
100

Example3: Not satesfied with the output and want to convert a string to list of its corsponding ascii vlaues? use following code

>>> string='Hello World'
>>> list_ascii=[ord(i) for i in string]
>>> print list_ascii

Output:

[72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100]

Related functions: chr(), int()

PFOTD: Python chr() function example

Today our in-built function is chr() which is useful for converting an ASCII value to its corresponding character.

Need of chr() function in Python:


Some times it requires to convert an unicode ASCII value to its character and this in-built function will give python this capability.

Python chr function Syntax:


chr(number)

Examples: Let us start with chr() function with some examples and see how we can use them in Python coding.

Example1: Convert an ASCII value to its corresponding character.

>>> chr(97)

Output:

'a'

>>>

Example2: Convert a string to an ASCII values.

>>>list_ascii=[72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100]

>>> for i in list_ascii:

... print chr(i)

...

Output:

H
e
l
l
o

W
o
r
l
d
>>>


Example3: Not satisfied with the output and want to convert an ASCII to list of its corresponding characters? Use following code

>>>list_ascii=[72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100]

>>> list_char=[chr(i) for i in list_ascii]

>>> print list_char['H', 'e', 'l', 'l', 'o', ' ', 'W', 'o', 'r', 'l', 'd']

>>> str1=''.join(list_char)

>>> print str1

Hello World

Related functions: chr(), int()

 

Thursday, October 31, 2013

Python inbuilt functions series at Coding Mania started

Hi all,

        Today we are starting a series on Python programming called Python Function OF The Day(PFOTD) to show you all how we can use these inbuilt functions to enrich your python capabilities. The concept of this series is simple, learning a simple python inbuilt function for one day. We have around 70+ inbuilt functions which we are going to cover in this series. A brief overview of how this series go on and skeleton of each post is given as below.

Built-in Python functions


Built-in Functions
abs()divmod()input()open()staticmethod()
all()enumerate()int()ord()str()
any()eval()isinstance()pow()sum()
basestring()execfile()issubclass()print()super()
bin()file()iter()property()tuple()
bool()filter()len()range()type()
bytearray()float()list()raw_input()unichr()
callable()format()locals()reduce()unicode()
chr()frozenset()long()reload()vars()
classmethod()getattr()map()repr()xrange()
cmp()globals()max()reversed()zip()
compile()hasattr()memoryview()round()__import__()
complex()hash()min()set()apply()
delattr()help()next()setattr()buffer()
dict()hex()object()slice()coerce()
dir()id()oct()sorted()intern()

The way we are going to write these are as follows

Over view of Python inbuilt function post

  1. Introduction to the function and its need
  2. Syntax of inbuilt function
  3. Examples of these inbuilt functions
  4. Related python inbuilt functions.

That is all about from our end on this. Please feel free to comment your thoughts on this.




Wednesday, May 29, 2013

Linux/Unix Shell scripting: Convert video file to gif image in Linux

This is a small script which will convert your video file into a gif file so that we can share it across Internet so that video player is not required to play it. The script is written in Linux Bash shell


#!/bin/bash
#Author: Surendra Anne(surendra@linuxnix.com)
#Purpose: To convert any video file into a GIF image file. The script will take one video file from command line arguments and do the conversion using mplayer and convert. So please install these two software's before running this script.
#Licence: GPL v.3

[[ "$#" -ne 1 ]] && { echo -e "Script should be run as below\n bash $0 video-file\n Exiting the script..!";exit 1; }
mkdir /tmp/images/
mplayer -ao null $1 -vo jpeg:outdir=/tmp/images/
convert /tmp/images/* /tmp/abc.gif
convert /tmp/abc.gif -fuzz 10% -layers Optimize /home/surendra/Desktop/opt-gif.gif
rm -rf /tmp/{abc.gif images/}

First line in the script will check if a file is passed to script or not, if not it will through some meaningful error message to user.
mplayer command will convert video file into jpeg images and the first convert command will club these image files to single animated gif file. The second convert command will compress the gif file considerably to 10% of actual size. And the file will be stored on desktop. You may have to change the locations to suite your requirements. 

Saturday, May 4, 2013

[SED]: Remove repeated/duplicate words from a file in Linux

In this post we will see how to delete repeated words. There is a human tendency to write fast and and when we try to review our writing we will find repeated words side by side. If you observe I written "and" two times. This is human mind tendency to process before we write actual word. Its hard to read entire file for duplicate words if the file is big enough to skim the text. This even cause to skip some words. A better procedure is to use some tools like SED and Perl/Python to do this with the help of Regular Expressions.

I have a file abc.txt with following data.

cat abc.txt
Output:

This is is how it works buddy
What else else you want

 Remove repeated words with SED as given below.

sed -ri 's/(.*\ )\1/\1/g'  abc.txt

cat abc.txt

Output:

This is how it works buddy
What else you want

Let me explain sed command which we used.

-r option is for enabling Extended Regular Expression which have grouping option with () braces.
-i option for inserting the changes to original file, Be careful with this option as you can not get your original file once modified.
(.*\ ) for mentioning any group of characters and which is followed by same set of characters which is represented by \1. This concept is called back reference, where \1 can store first set of characters enclosed in first (). And these two things (.*\ )\1 is replaced by same word with \1 which is actual back reference to first (.*\ ).




 

Friday, May 3, 2013

Vi editor: Delete matched search pattern from a file

1) How can I search for a word and delete that matched word in vi editor?

This is bit tricky question. With SED its bit easy to do. In vi editor too we can search for a word and delete it with some trick.

Delete matched search term from a file

Step1: Go to command mode and search mode

Step2: Now search for your term and replace it with nothing

:%s/searchterm//g

This will help you to delete all the occurrences of your search term. Let me explain above syntax

:%s is for searching for entire file, if you want to search in a particular line we can just use :s.
/serchterm// will replace 'searchterm' with nothing, which means it will be removed from that line
g for global removal, what it mean is the search and replace operation is applicable for all the occurrences of search-term in a given line.

Delete matched search term line from a file 

Some times its require to delete entire line of your searched term. We can use below code once you go to command mode

:g/searchterm/d

Deleting reverse or inverse of search term lines from a file


We can even delete all the lines which do not contain our search term with below code.

:g!/searchterm/d

Hope this helps, check our other vi editor posts as well.

Thursday, April 25, 2013

AWK Scripting: Learn AWK Built-in variables with examples

AWK inbuilt variables: FS, OFS, RS, ORS, NR, NF, FNR, FILENAME


AWK is supplied with good number of built-in variables which come in handy when working with data files. We will see each AWK built-in variables with one or two examples to familiarize with them. Without these built-in variables it’s very much difficult to write simple AWK code. These variable are used to format output of an AWK command, as input field separator and even we can store current input file name in them for using them with in the script. Some of the AWK concepts already covered are.

AWK scripting: What is an AWK and how to use it?

AWK built-in variables:

  • NR: Current count of the number of input records.
  • NF: Keeps a count of the number of fields
  • FILENAME: The name of the current input-file.
  • FNR: No of records in current filename
  • FS: Contains the "field separator" character
  • RS: Stores the current "record separator" or Row Separator.
  • OFS: Stores the "output field separator".
  • ORS: Stores the "output record separator" or Output RS.
Our sample DB file for this post is db.txt

cat db.txt

John,29,MS,IBM,M,Married
Barbi,45,MD,JHH,F,Single
Mitch,33,BS,BofA,M,Single
Tim,39,Phd,DELL,M,Married
Lisa,22,BS,SmartDrive,F,Married

In order to make it simple we can divide above  inbuilt variables in to groups on basis of their operations.

Group1: FS(input field separator), OFS,
Group2: RS(Row separator) and ORS(Output record separator)
Group3: NR, NF and FNR
Group4: FILENAME variable

Group1: FS(input field separator), OFS


Let us start with FS and OFS built-in variables.

FS AWK variable: This variable is useful in storing the input field separator. By default AWK can understand only spaces, tabs as input and output separators. But if your file contains some other character as separator other than these mention one's, AWK cannot understand them. For example Linux password file which contain ‘:’ as a separator. So in order to mention the input filed separator we use this inbuilt variable.

We will see what issue we face if we don’t mention the field separator for our db.txt.

Example1: Print first column data from db.txt file.

awk '{print $1}' db.txt

Output:

John,29,MS,IBM,M,Married
Barbi,45,MD,JHH,F,Single
Mitch,33,BS,BofA,M,Single
Tim,39,Phd,DELL,M,Married
Lisa,22,BS,SmartDrive,F,Married

If you see entire file is displayed which indicates AWK do not understand db.txt file separator ",". We have to tell AWK what is the field separator.

Example2: List only first column data from db.txt file which have field separator as ‘,’.

awk 'BEGIN{FS=","}{print $1}' db.txt\

Output:

John
Barbi
Mitch
Tim
Lisa

Example3: We can use AWK option –F for mentioning input field separator as shown in below example for printing 4th column.

awk -F',' '{print $4}' db.txt

Output:

IBM
JHH
BofA
DELL
SmartDrive

OFS AWK variable: This variable is useful for mentioning what is your output field separator which separates output data.

Example4: Display only 1st and 4th column and the separator between at output for these columns should be $.

awk 'BEGIN{FS=",";OFS=" $ "}{print $1,$4}' db.txt

Output:

John $ IBM
Barbi $ JHH
Mitch $ BofA
Tim $ DELL
Lisa $ SmartDrive

Note: I given space before and after $ in OFS variable to show better output. You can remove the spaces if required.

I will leave printing only first and fourth columns to readers without using OFS and see the issue.

Group2: RS(Row separator) and ORS(Output record separator)


RS(Row separator) and ORS(Output record separator).

RS AWK Variable: Row Separator is helpful in defining separator between rows in a file. By default AWK takes row separator as new line. We can change this by using RS built-in variable.

Example5: I want to convert a sentence to a word per line. We can use RS variable for doing it.

echo “This is how it works” | awk ‘BEGIN{RS=” ”}{print $0}’

Output:

This
is
how
it
works

ORS(Output Record Separator): This variable is useful for defining the record separator for the AWK command output. By default ORS is set to new line.

Example6: Print all the company names in single line which are in 4th column.

awk -F',' 'BEGIN{ORS=" "}{print $4}' db.txt

Output:

IBM JHH BofA DELL SmartDrive

Group3: NF, NR and FNR

 NF AWK variable: This variable keeps information about total fields in a given row. The final value of a row can be represented with $NF.

Example7: Print number of fields each row in db.txt file.

 awk '{print NF}' db.txt




Output:

5
5
4
5
4

Example8: Print last field in each row of db.txt file.

awk '{print $NF}' db.txt



Output:

77
45
37
95
47

Note: If you observe above two examples We used Just NF for giving us the count of fields in a given row and $NF for displaying last element in each row. $NF will come handy when you are not sure what is your last column number.

NR AWK variable: This variable keeps the value of present line number. This will come handy when you want to print line numbers in a file.

Example9: Print line number for each line in a given file.

awk '{print NR, $0}' db.txt

Output:

1 Jones 2143 78 84 77
2 Gondrol 2321 56 58 45
3 RinRao 2122234 38 37
4 Edwin 253734 87 97 95
5 Dayan 24155 30 47

 This can be treated as cat command -n option for displaying line number for a file.

FNR AWK variable: This variable keeps count of number of lines present in a given file/data. This will come handy when you want to print no of line present in a given file. This command is equivalent to wc -l command.

Example10: Print total number of lines in a given file.

awk 'END{print FNR}' db.txt

Output:

5

From the above output we can conclude that number of lines present in db.txt file is 5.

Group4: FILENAME variable



FILENAME AWK variable: This variable contain file awk command is processing.

Example11: Print filename for each line in a given file.

 awk '{print FILENAME, NR, $0}' abc.txt

Output:

abc.txt 1 Jones 2143 78 84 77
abc.txt 2 Gondrol 2321 56 58 45
abc.txt 3 RinRao 2122234 38 37
abc.txt 4 Edwin 253734 87 97 95
abc.txt 5 Dayan 24155 30 47

In our next post we will see how to use ARRAY's in AWK scripting.

Wednesday, April 24, 2013

AWK Scripting: How to define awk variables

AWK variables: This is our ongoing tutorials on AWK scripting. As we mention earlier AWK is a full pledged language with all statements, arrays, control structures, functions etc. Today we will see how to define a variable in AWK and use it when it’s required. We already covered following AWK concepts

AWK scripting: What is an AWK and how to use it?
AWK scripting: 14 AWK print statment examples
AWK scripting: 8 AWK printf statements examples
AWK scripting: 10 BEGIN and END block examples

What is a Variable?

A variable is defined as storage location to store some value in it so that we can use this in a program. Variables will protect us from varying values in the storage location. This will help us to avoid hardcode a value in to program where ever its used. We can define a variable at the start of program and use the variable across the program, if we want to change the value of it, we can change it where we define it and this value will be updated where ever we use that variable.

Defining a variable in AWK

We can define variable in AWK where ever we require. Variables can be used for
Initializations for values
Arithmetic operations
And many more.

For this concept we will use below db.txt file.

cat db.txt
 
Jones 2143 78 84 77
Gondrol 2321 56 58 45
RinRao 2122234 38 37
Edwin 253734 87 97 95
Dayan 24155 30 47

 

Lets learn AWK variables with examples


Example1:  Add all values in column 3 and display for each addition from our db.txt file. Frist value should be first value in column3, second in the list should be first value + second value and so on.
awk 'BEGIN{a=0}{a=a+$3;print a}' db.txt
Output:
78
134
172
259
289

In the above example we defined a variable a and initialized to zero in the BEGIN block and did arithmetic operation in main block.
Example2: I don’t want above output, I just want to print a final value once the sum of all value in 3rd column. We can use END block for doing a final print instead of printing per iteration.
awk 'BEGIN{a=0}{a=a+$3}END{print a}' db.txt
Output:
289
Example3: Print all the lines which contain numbers greater than VAR1 in its column 4

awk 'BEGIN{VAR1=57}(VAR1<$4){print $0}' db.txt
 
Output:

Jones 2143 78 84 77
Gondrol 2321 56 58 45
Edwin 253734 87 97 95


These are simple examples to start with Varaiables. In our next post we will see how get inputs from user.