[SED]: Remove repeated/duplicate words from a file in Linux

In this post we will see how to delete repeated words. There is a human tendency to write fast and and when we try to review our writing we will find repeated words side by side. If you observe I written "and" two times. This is human mind tendency to process before we write actual word. Its hard to read entire file for duplicate words if the file is big enough to skim the text. This even cause to skip some words. A better procedure is to use some tools like SED and Perl/Python to do this with the help of Regular Expressions.

I have a file abc.txt with following data.

cat abc.txt
Output:

This is is how it works buddy
What else else you want
 Remove repeated words with SED as given below.

sed -ri 's/(.*\ )\1/\1/g'  abc.txt

cat abc.txt

Output:

This is how it works buddy
What else you want
Let me explain sed command which we used.

-r option is for enabling Extended Regular Expression which have grouping option with () braces.
-i option for inserting the changes to original file, Be careful with this option as you can not get your original file once modified.
(.*\ ) for mentioning any group of characters and which is followed by same set of characters which is represented by \1. This concept is called back reference, where \1 can store first set of characters enclosed in first (). And these two things (.*\ )\1 is replaced by same word with \1 which is actual back reference to first (.*\ ).




 

Read More

AWK scripting: What is an AWK and how to use it?

This is our first post on AWK, in this we will see some of the basic stuff of AWK like history, advantages syntax and how it works etc.


1. A brief history about AWK

2. Advantages and disadvantages of AWK

3. AWK syntax

4. How AWK works?

AWK is a command/tool available in all the Linux/Unix flavors to do text filtering, manipulation etc. This tool is mainly meant for processing text files and reporting. AWK can be treated as a programming language due to its capabilities such as Arithmetic operations, Binary operations, conditions, loops, functions etc. AWK is an interpreter language. This tool/programming language was developed in 1977 by Alfred V. Aho, Peter J. Weinberger, Brian W. Kernighan. AWK got its name from its creates family names.

Due to its capabilities AWK earned its nick name as “Awk the Swiss army knife of the Unix toolkit”. This is true because it can do text processing with ease when compared to other text parsing tools available in Linux/Unix.

Below are some advantages and disadvantages of AWK tool which I come across when using it.

AWK Advantages

• Validate data
• Managing small db files
• Generating reports
• Parsing command outputs
• Parsing log files
• Can parse more than one file at a time.

AWK Disadvantages

• Many flavors are there: awk, nawk, gawk,mawk, tawk which makes portability issue.
• Cannot be full-pledge scripting language like PERL, Python and Ruby.
• Useful only for data processing most of the time.

The syntax for AWK command is as follows

 When executing an AWK command we are going to use one of the below syntax.

awk options 'awk-code' filename

Unix-command | awk options 'awk-code'

awk options -f awkscript-file filename

How AWK works?

AWK works line by line: As many assume that AWK works columns wise, but it’s not true. AWK works as SED works i.e. in horizontal manner, reading one line after the other.

Awk treats a file as group of columns: AWK when reading lines it will assign data to columns depending on the field separator.

Column number: AWK assigns each column with a number and they are represented as $1, $2, $3 and so on till the last column. The last column number is assigned with NF(Number of Fields). And entire line/record is represented with $0. NF is a built-in variable contains total number of columns. Awk knows the last column number and equals NF. Depend on different conditions AWK works on these columns to get desired output.

In our next post we will see about AWK inbuilt variable. These variable are very much important as they are frequently referred by AWK to do its activity.

Read More

Shell Script: Remove Spaces in file names in Linux/Unix

Removing/renaming multiple files containing spaces is a bit difficult task.  for, while loop, mv, cp commands can not understand spaces, we have to use \  for mentioning spaces some of the commands.

For example if we take "for loop" to convert all the files in a given directory we will face issue, because for loop will take space as a separator between given entries

for i in *
do
VAR1=$(echo "$i" | sed 's/\ /_/g')
mv "$i" "$VAR1"
done

The problem here is that
1)for look can not understand spaces
2)mv command do not understand spaces

Suppose I have a file name as "abc read.txt", but this commands will take as abc and read.txt as two files which is not right. one way to make mv, cp command to understand space is using "\ ". But for loop still don't have this feature.

One solution is to use rename command which supports Regexp, Batch remaining files etc.

Check if rename command, if its not installed install it.

In Debian based machines:

apt-get install rename

In Red hat based machines:

yum install rename

Once the command is installed go to the directory where you have files with spaces in it and execute below any command

rename 's/\ /_/g' *
or
rename ' ' '_' *

This is an excellent command which have inbuilt capabilities of Regexp, tr command, sed command etc.

Shell script to rename files in multiple directories which are in a parent directory


#!/bin/bash
for i in *
do
rename 's/\ /_/g' $i/*
done

if your directories falls under different folders then get that links in to a file and use below for loop for that

#!/bin/bash

for i in $(cat /path/to/dir.txt)

do

rename 's/\ /_/g' $i/*

done

These two scripts will help us in reduce time.

Read More

SED find and replace multiple search patterns

Q. I have a word abc and cde and I want to replace them both with xyz. How can I do that in Shell scripting?

We can not do this with out using sed command. Below are the ways you can do that.

Below example will try to replace abc or cde with xyz. I used regexp '|' to accomplish this task. -r is for enabling regexp and -i for inserting the changes in to the file.
sed -ri 's/abc|cde/xyz/g' filename

Some more examples on replacing multiple words/char. In below example we can replace either a or b with d

sed -i 's/[ab]/d/g' filename

I want to replace swapon and swapoff words with just word "free"

sed -r 's/swap(on|off)/free/g' filename
In order to understand () and | we have know about regexp please click here to know more about regexp

Below is bit odd way to do the multiple word changes using sed.

sed 's/abc/get/g;s/def/get/g' filename
or
sed -e 's/abc/get/g' -e 's/def/get/g' filename

Read More

Linux Shell script to convert numbers to star's in a file name

Q. One of the blog follower asked this question. Write a LINUX shell script that accepts an arbitrarily long list of file names as arguments and replaces all digits in them with a "*".

This is bit tricky and the solution I got is below script


#!/bin/bash
for i in $(ls | grep '[0-9]')
do
eval mv $i $(echo $i | sed 's/[0-9]/\\\*/g')
done


Some of the issues I faced are
1)In sed I tried to replace numbers with just * which created an issue
2)In sed I tried to replace numbers with \\* which is still not working.

So I tried to use eval to get this solution. I advantage of eval is that it will try to evaluate a shell code on the fly.
ls | grep [0-9] will give you the files which contain a single number in its filename, this output is feed in to for loop and each value is read in to i variable.

sed 's/[0-9]/\\\*/g' Will search for a number in the filename and replace it with \*.
eval command will convert mv $filename $(echo $i | sed 's/[0-9]/\\\*/g') to mv file1name file\*name

Hope this helps.

Read More

Delete last line from a file using Linux command or Shell script

Q. Delete or remove last line from a file.

This can be achieved  with SED command.

sed -i '$d' filename

Let me explain this command. SED is a stream editor which is useful for editing stream of characters. here -i option say's to insert the data in to the original file, ie modify original file. $ indicates last line and d for deleting that line and update to the file.

There are other methods to delete last lines

head -n -1 file1 > file2;mv file2 file1

Here I used head command to list all the lines but not the last line, redirected the output of this to a new file then moved file1 to file2.

Read More

Move all file description(first 5 lines) of a file to file

Q. I have a requirement to move all my shell script meta data such as interpreter, Author, date, purpose etc which occupies my first 5 lines.

With head

head -n5 *.sh > file.txt

With SED


sed -n '1,4 p' xyz.txt 

or

sed '5,$ d' xyz.txt

you can keep this one in for loop and execute sed command get your file content to other file.

Read More