Perl - Return a value from function depending on the context

 How to return a value from a function depending on the context? For example, if the return value is collected into an array, the function should return an array, if collected in a scalar, then the return value should be a specific scalar. Let us see how to handle this situation in Perl.

  The Perl script below uses a function "calc" which for a given array as input, calculates the sum of all elements in the array, and also finds the biggest and smallest  number in the array.

Tweet

Shell Script - How to dump a Oracle table into a CSV file?

How to dump /export the contents of an Oracle table into a CSV file? Let us write a shell script to dump the content of EMP table into a CSV file.

Tweet

How to print every nth line in a file in Linux?

How to print every nth line of a file?  The requirement is to print every 3rd line of the file.
  Let us consider a file with the following content.

$ cat file
AIX
Solaris
Unix
Linux
HPUX
Ubuntu

1. Using awk :

$ awk '!(NR%3)' file
Unix
Ubuntu

  NR variable contains the line number of the file in awk. If NR modulus 3 is equal to 0, it indicates the line is a multiple of 3. And only those lines are printed which modulus is 0.

Tweet

sed - 10 examples to print lines from a file

 In this article of sed series, we will see how to print a particular line using the print(p) command of sed.
  Let us consider a file with the following contents:

$ cat file
AIX
Solaris
Unix
Linux
HPUX

1. Print only the first line of the file:

$ sed -n '1p' file
AIX

  Similarly, to print a particular line, put the line number before 'p'.

2. Print only the last line of the file

$ sed -n '$p' file
HPUX

 $ indicates the last line.

Tweet

10 examples of initializing a Hash variable in Perl

 Hash'es  is one of the most important concepts in Perl. Hashes are associative arrays where the data is stored in the form of key-value pairs. The big advantage of a hash is faster look-up compared to arrays. In this article, we will see the how we can populate / initialize a hash variable in different scenarios:

1. Regular way of initializing a hash :

my %h1=("Jan" => 31, "Feb" => 28, "Mar" => 31);

  This is the normal way of populating hash where-in every set is a key-value pair. The operator separating the key-value pair '=>' is called Fat-comma operator.
  After the initialization, the data in the hash will be stored as shown below where 'Jan','Feb' and 'Mar' are the keys of the hash and values against it are their respective values associated.

'Jan' => 31
'Feb' => 28
'Mar' => 31

Tweet

How to remove the leading and trailing spaces in a file?

How to remove / delete the leading and trailing spaces in a file? How to replace a group of spaces with a single space?
    Let us consider a file with the below content:

$ cat file
 Linux  25
Fedora   40
Suse 36
    CentOS 50
LinuxMint 15

Using the -e option of cat, the trailing spaces can be noticed easily(the $ symbol indicates end of line)

$ cat -e file
 Linux  25$
Fedora   40$
Suse 36    $
    CentOS 50$
LinuxMint 15$

Tweet

25 examples of vi / vim substitution commands

 In this article, we will see the commonly used vi substitution commands. All the below commands are to be given in escape mode:

Common tricks:
1. To replace the 1st occurrence of 'a' with 'A' in every line:

:%s/a/A/

2. To replace the all the occurences of 'a' with 'A'

:%s/a/A/g

g is for global substitution, ie. substitution for all matches.
3. To replace 'e' with 'E' only in the current line:

:s/e/E/g

Tweet

awk - 10 examples to insert / remove / update fields of a CSV file

How to manipulate a text / CSV file using awk/gawk? How to insert/add a column between columns, remove columns, or to update a particular column? Let us discuss in this article.

Consider a CSV file with the following contents:

$ cat file
Unix,10,A
Linux,30,B
Solaris,40,C
Fedora,20,D
Ubuntu,50,E

1. To insert a new column (say serial number) before the 1st column

$ awk -F, '{$1=++i FS $1;}1' OFS=, file
1,Unix,10,A
2,Linux,30,B
3,Solaris,40,C
4,Fedora,20,D
5,Ubuntu,50,E

$1=++i FS $1 => Space is used to concatenate columns in awk. This expression concatenates a new field(++i) with the 1st field along with the delimiter(FS), and assigns it back to the 1st field($1). FS contains the file delimiter.

Tweet

7 examples of find command with xargs in Linux

 find command becomes very powerful when used with xargs. xargs is a beautiful command which actually builds a command and executes it. Though xargs command can be used standalone to work on standard input or on a file, it becomes all the more powerful when used with the find command. The man page definition of xargs is :"build and execute command lines from standard input"

Let us see some examples of find command with xargs to understand more:

Tweet

How to swap 2 columns in a file in Linux?

How to swap 2 columns or fields in a text or a CSV file?

Let us consider a file with the following contents. The requirement is to swap the first 2 fields:

$ cat file
Linux,25,1
Solaris,30,2
HPUX,15,3
AIX,28,4

1. Using awk:

$ awk -F, '{print $2,$1,$3}' OFS=, file
25,Linux,1
30,Solaris,2
15,HPUX,3
28,AIX,4

Using the print statement, simply the fields can be printed in the order in which it is needed.  -F is to specify the input field separator and OFS for the output field separator. However, this method will not be ideal for a file with large number of fields since it forces us to print all columns individually.

Tweet

How to retrieve or extract the tag value from XML in Linux?

How to fetch the tag value for a given tag from a simple XML file?
  Let us consider a simple XML file, cust.xml, with  customer details as below:

<?xml version="1.0" encoding="ISO-8859-1"?>
<CustDetails>
<CustName>Unix</CustName>
<CustomerId>999</CustomerId>
<Age>28</Age>
</CustDetails>

The requirement is to retrieve the tag value of "CustName" from the xml file. Let us see how to get this done in different ways:

1. Using the sed command:

$ sed -n '/CustName/{s/.*<CustName>//;s/<\/CustName.*//;p;}' cust.xml
Unix

Using sed substitution command, the pattern matching till the beginning of the opening tag is deleted. Another substitution is done to remove the pattern from the closing tag till the end. After this, only the tag value will be left in the pattern space.

Tweet

Perl - User defined function to simulate uc and lc

  uc and lc are Perl functions to convert strings into upper-case and lower-case respectively. In case, if the user wants to simulate the uc and lc functions by writing his own function, how to do it? Let us discuss in this article the different methods of how to simulate the uc and lc function?

Method 1:

  3 things should be known before going ahead with this one:

 1. The decimal values ranges of alphabets are:
       A - Z : 65 to 90
       a - z : 97 to 122
 2.  ord - Perl function which gives the decimal value of an ASCII character.
 3.  chr - Perl function which gives the ASCII character for a given value.

Tweet

Link count: File vs Directory

  One of the results of the ls -l command  is the link count. What is this link count? How it is useful? What is the difference between the link count of a file and a directory? Let us discuss in this article.

1. What is the link count of a file?
      The link count of a file tells the total number of links a file has which is nothing but the number of hard-links a file has. This count, however, does not include the soft-link count.
Note: The soft-link is not part of the link count since the soft-link's inode number is different from the original file.

2. How to find the link count of a file or directory?
   The link count can be seen by using the ls -l command :

$ ls -l test.c
-rwxr-xr-x 1 guru users 267 Jul 18 16:59 test.c

    By default, a file will have a link count of 1 i.e., any new file created will have a link count 1.

Tweet

How to remove a component of absolute path?

How to remove a component of absolute PATH?

Let us consider an absolute path "/export/home/guru/prog/perl".

$ x="/export/home/guru/prog/perl" 

The requirement is to remove the 4th component of the absolute path so that it becomes "/export/home/prog/perl".

1. cut command:

$ echo $x | cut -d"/" -f -3,5-
/export/home/prog/perl

cut is the straightforward option. Since we want to drop the 4th field, cut fields 1 to 3, and then from 5 onwards till the end.

Tweet

How to find the total count of a word / string in a file?

How to find/calculate the total count of occurences of a particular word in a file?

Let us consider a file with the following contents. The requirement is to find the total number of occurrences of the word 'Unix':

$ cat file
Unix is an OS
Linux is also Unix
Welcome to Unix, and Unix OS.

Tweet

10 examples of split command in Unix

split command in Unix is used to split a large file into smaller files. The splitting can be done on various criteria: on the basis of number of lines, or the number of output files or the byte count, etc. In an earlier article, we discussed how to split files into multiple files using awk. In this, we will discuss it using the split command itself:

 Let us consider a file with the following content:

$ cat file
Unix
Linux
AIX
Solaris
HPUX
Ubuntu
Cygwin

Tweet

How to find duplicate records of a file in Linux?

How to find the duplicate records / lines from a file in Linux?
  Let us consider a file with the following contents. The duplicate record here is 'Linux'.

$ cat file
Unix
Linux
Solaris
AIX
Linux

Let us now see the different ways to find the duplicate record.

1. Using sort and uniq:

$ sort file | uniq -d
Linux

   uniq command has an option "-d" which lists out only the duplicate records. sort command is used since the uniq command works only on sorted files. uniq command without the "-d" option will delete the duplicate records.

Tweet

How to remove duplicate records from a file in Linux?

How to remove / delete duplicate records / lines from a file?

  Let us consider a file with the following content. The duplicate record is 'Linux' with 2 entries :

$ cat file
Unix
Linux
Solaris
AIX
Linux

Tweet

grep vs awk : 10 examples of pattern search

"grep vs awk" does not mean comparison between awk and grep because there is no comparison to awk whatsoever from grep.  awk is a highly powerful programming language whereas grep is just a filtration tool. However, many of the things which is done using grep and few more commands can be done using a simple awk command. In this article, we will see the awk altenative for the frequently used grep commands.

Let us consisder a file with the following contents:

$ cat file
Unix
Linux
Solaris
AIX
Ubuntu
Unix

Tweet

$IGNOREEOF vs ignoreeof

How to prevent your terminal from getting closed OR prevent from getting the user logged out on pressing Control-D?
  The answer to this is IGNOREEOF / ignoreeof. Let us see what is this IGNOREEOF and the difference between IGNOREEOF and ignoreeof.

Control-D:
   Many of the times we might have pressed some key only to realize that the terminal got closed or he got logged out from his account. The user gets logged out whenever an EOF character is pressed which is Control-D.  So, if the user happens to be in his login shell, then logically the Ctrl-D ends up in getting the user's terminal closed as well.

Bash/Bourne shell:
    In Bash/Bourne shell, the log-out from the user account can be prevented by using the environment variable IGNOREEOF.

Tweet

8 examples to find sum of all columns / numbers in a line

How to find the sum of all numbers or columns in a line of a text / CSV file?
    In an earlier article, we discussed about different ways of how to sum all the numbers in a file. In this, we will see how to find the sum of all the numbers in a line, or in other words, sum of all the columns in a line.

Let us consider a file with the following contents:

$ cat file
20 30 21
33 33 32
12 21 12

Tweet

perl - 15 examples of substr command

substr is a perl function to extract substring from a string. substring in Perl is little special compared to other languages due to some additional functionalities. Let us discuss the substr function with examples.

Syntax:

 substr(STRING, OFFSET,  LENGTH, REPLACEMENT)

where STRING - the actual string from which the substring is to be extracted
          OFFSET - the starting position from where to extract
          LENGTH - the number of characters to be extracted from OFFSET (Optional)
          REPLACEMENT - the string to be replaced (Optional)

Tweet

ps command : what does the TIME indicate?

   One of the output fields of the ps commands is the TIME component. As per the man page of ps command, the TIME is: "CPU utilization of process or thread, incremented each time the system clock ticks and the process or thread is found to be running".  However, most of the times, when the user executes the ps command, he always sees the TIME as  00:00:00 as shown below(usually in development or practice environments). So, what exactly is the TIME? Let us see in this article.

$ ps
  PID TTY          TIME CMD
18315 pts/3    00:00:00 bash
18347 pts/3    00:00:00 ps

TIME:

   It is the total accumulated CPU utilization time for a particular process. The 00:00:00 against the bash process indicates no CPU time has been given at all for the bash process by the kernel till now. Is it confusing? How come the CPU utilization of the bash(login shell) is 0 when we are executing so many commands under bash?

Tweet

8 examples to change the delimiter of a file in Linux

How to change the delimiter of a file from comma to colon?

  Let us consider a file with the following contents:

$ cat file
Unix,10,A
Linux,30,B
Solaris,40,C
HPUX,20,D
Ubuntu,50,E

Tweet

What is a process in UNIX / Linux?

  A process is a program in execution in memory or in other words, an instance of a program in memory. Any program executed creates a process. A program can be a command, a shell script, or any binary executable or any application. However, not all commands end up in creating process, there are some exceptions. Similar to how a file created has properties associated with it, a process also has lots of properties associated to it.

Process attributes:
  A process has some properties associated to it:

Tweet

getopts - How to pass command line options to shell script?

   In an earlier article, we discussed how to pass command line arguments to shell script and access them using positional parameters. In this, we will see how to use the getopts command to pass command line options to shell scripts. Command line options are the options or switches passed to a command. For example, -l, -r, -t are some examples of the command line options passed to the ls command.

Types:
  Command line options can be classified into 3 types:

1. Options which does not take arguments:
• ls -l    #no arg for -l
2. Options which take arguments:
• cut -d","  file #arg "," for -d
3. Options + Additional command line arguments:
• ls -l file1 file2 file3 

Syntax:
The syntax of the getopts command is:

 getopts optstring name

-> optstring - the string which contains the list of options expected in the command line
-> name - the variable name which is used to read the command line options one by one.

Tweet

5 examples to extract substring in bash / ksh93 shell

Sub-string is nothing but extracting a part of a string. In an earlier article, we discussed the various ways of how to extract a sub-string from every line in a file. In this article, we will see how to extract sub-string exclusively in bash / ksh93 and its options. Moreover, this substring extraction is an internal command, and hence it is very good from performance perspective:

 The syntax of the sub-string extraction is:

${variable:offset:length}

where variable is the variable containing the string,
          offset specifies the position from where to start extracting the string,
          length specifies the length of characters to be extracted from the offset.

 Let us consider a variable containing a string "Solaris"

$ x="Solaris"

Tweet

Insert new line or text after every n lines in Linux

In one of our earlier articles, we discussed how to insert a line before or after a pattern. In this article, we will see how to insert a line after every n lines in a file.

Let us consider a file. The requirement is to insert a line with text "LINE" afte every 2 lines:

$ cat file
Unix
Linux
Solaris
AIX
Linux

Tweet

What is the use of sticky bit in Linux?

    In an earlier article, we discussed the purpose of SUID, an advanced file permision. In the same lines, Sticky bit is also an advanced file permission used in Unix/Linux or its flavors. This feature was originally meant for regular files and directories, though is now used only in directories in many Unix flavors. This permission, when applied on a directory, allows users to maintain 'their' files in directories which are shared with everybody.  

Tweet

15 examples of sort command in Linux

sort command is used to sort a file, arranging the records in a particular order. By default, the sort command sorts file assuming the contents are ascii. Using options in sort command, it can also be used to sort numerically. Let us discuss it with some examples:

File with Ascii data:
 Let us consider a file with the following contents:

$ cat file
Unix
Linux
Solaris
AIX
Linux
HPUX

1. sort simply sorts the file in alphabetical order:

$ sort file
AIX
HPUX
Linux
Linux
Solaris
Unix

All records are sorted alphabetically.

Tweet

7 examples to print few lines before a pattern match in Linux

In this article, we will discuss the different ways for how to print a few lines before a pattern match in log files. You will do well to know how to print a few lines AFTER a pattern match.

 Let us consider a sample file as below. The requirement is to print 2 lines before the pattern 'Linux':

$ cat file
Unix
AIX
Solaris
Linux
SCO

1. grep will do it for you if your grep is a GNU grep.

$ grep -B2 Linux file
AIX
Solaris
Linux

The option '-B' will give n lines before the pattern. Hence '-B2 Linux' gives 2 lines before the pattern 'Linux' including the line matching the pattern.

Tweet

awk - Passing awk variables to shell

In one of our earlier articles, we discussed how to access or pass shell variables to awk. In this, we will see how to access the awk variables in shell? Or How to access awk variables as shell variables ?  Let us see the different ways in which we can achieve this.

 Let us consider a file with the sample contents as below:

$ cat file
Linux 20
Solaris 30
HPUX 40

1. Access the value of the entry "Solaris" in a shell variable, say x:

$ x=`awk '/Solaris/{a=$2;print a}' file`
$ echo $x
30

This approach is fine as long as we want to access only one value. What if we have to access multiple values in shell?

Tweet

Move files and directories : Inodes Part 3

  In our earlier inode articles, we discussed about the inode basics and inode relations with the soft links and hard links and their differences. In this, we will discuss about the changes in inodes during the move(mv) of a file or directory.
  
1.  Does the mv command rename a file or move a file?
      It actually does both, depending on the context. If the source and target are both files, it does rename. If the source is a file and the target is a directory, it does move. However, once we understand the internal working of the mv command with respect to inodes, we will realize both are same.

Tweet

awk - 10 examples to read files with multiple delimiters

 In this article of awk series, we will see how to use awk to read or parse text or CSV files containing multiple delimiters or repeating delimiters. Also, we will discuss about some peculiar delimiters and how to handle them using awk.

Let us consider a sample file. This colon separated file contains item, purchase year and a set of prices separated by a semicolon.

$ cat file
Item1:2010:10;20;30
Item2:2012:12;29;19
Item3:2014:15;50;61

Tweet

10 examples of paste command usage in Linux

   In this article, we will see how to use the paste command with some examples. paste command, by definition of man page, is used to merge lines of files. It is very useful for merging a single file and also for merging set of files as well. This article is divided into 2 parts:

• paste command examples for single file handling
• paste command examples for multiple files handling

Tweet

find command: 15 examples to exclude directories using prune

   When we search for files in Unix/Linux using the find command, we might want to search for files only in some specific directories, OR to search in all directories except a particular directory and so on.  prune option in the find command is the one for excluding a directory within a directory tree. In this article of find command, we will use how to use the prune option of the find command to exclude directories with some examples.

The simple find command below lists all the files and directories in the directory tree. The . indicates the path to search for the files. We will see the usage of prune using this directory tree:

$ find .
.
./area.pm
./C
./C/temp
./C/f2.c
./test.c
./temp
./temp/a.c

The functionality of prune is simple: "If the file is a directory, do not descend or traverse into it." Sounds simple, isn't it? Let us see some examples:

Tweet

find2perl - 10 examples to find files without find command

    In our earlier articles on the find command, we discussed how to find files modified before X hours, X mins or X seconds. find command options mtime, mmin, newer, etc. are used to get the desired list of files. 

    How to find the files modified before X mins or X seconds in a system which does not support the 'newer' option or the 'mmin' options of the find command? 

     The answer for this find2perl. This is a tool which comes in-built along with perl installation. Hence, any Unix flavor in which perl is present should have the find2perl utility present in it. Perl has some modules for finding files which is as good as the Unix find. This utility creates a perl script which uses the File::Find module to find files, and gives an output which is nothing but a perl script equivalent of the Unix find command. On running this script, the desired list of files could be retrieved. 

Tweet

Read 2 files line by line alternatively

 In this article, we will see different ways in which we can read 2 files in parallel line by line, i.e, one line from file1 and one line from file2, next line from file 1, next line from file2, and so on. This way of reading produces a combined file output in which lines are alternated between the files.

Let us consider 2 sample files with the following contents:

$ cat file1
Unix
Linux
Solaris

$ cat file2
AIX
HPUX
Cygwin

Tweet

awk - 10 examples to split a file into multiple files

 In this article of the awk series, we will see the different scenarios in which we need to split a file into multiple files using awk. The files can be split into multiple files either based on a condition, or based on a pattern or because the file is big and hence needs to split into smaller files.

Sample File1:
Let us consider a sample file with the following contents:

$ cat file1
Item1,200
Item2,500
Item3,900
Item2,800
Item1,600

Tweet

sed - 25 examples to delete a line or pattern in a file

In this article of  sed tutorial series, we are going to see how to delete or remove a particular line or a particular pattern from a file using the sed command.

 Let us consider a file with the sample contents as below:

$ cat file
Cygwin
Unix
Linux
Solaris
AIX

1. Delete the 1st line or the header line:

$ sed '1d' file
Unix
Linux
Solaris
AIX

      d command is to delete a line. 1d means to delete the first line.

Tweet

A forum of The UNIX School

Dear Unix School Subscribers,

   Something which I have been thinking for quite some time, to have a dedicated forum for our blog is now ready, The UNIX School Forum.This forum is a place where-in you can get all your doubts clarified on Shell Scripting, sed, awk, perl, etc. All the users who till now have been sending mails to clarify their queries can simply post a question here and get the answer. So, what are you waiting for? Simply go the forum tab and post a question. 

Tweet

Insert a line before or after a pattern

  In this article, we will see the different ways in which we can insert a line before or after on finding a pattern.  On finding the pattern 'Fedora', the requirement is to insert a line:

• before the pattern 
• after the pattern.

Let us consider a file with the following contents:

$ cat file
Linux
Solaris
Fedora
Ubuntu
AIX
HPUX

Inserting before the pattern:

Tweet

Shell script to send birthday reminder e-mails

   In a professional work place, it is pretty common to celebrate the birthdays of your colleagues. Somebody in the team will have the birthdays maintained in a file and they will let the team know when the day arrives. The issue here is it is quite possible to miss out if one forgets to check the birthday file. How nice it will be if you can get an e-mail reminder of the person celebrating their birthday? Let us see in this article how to create a shell script which will send us an e-mail stating the names of the people celebrating their birthdays. Also, the same script with little modifications can be used to set reminder for any other purpose.

  This article discusses about 3 scripts:

• Shell Script to get reminder on the list of users having their birthday today.
• Shell  Script to get reminder on the list of users having their birthday the next day or some other day.
• Shell  Script to get reminder on the list of all users having their birthday over the next n days.

Tweet

Swap every 2 lines in a file

In an earlier article, we had discussed the different ways in which we can join every two lines in a file. In this article, we will see the different ways in which we can swap every two lines in a file.

Let us consider a file with the following contents:

$ cat file
Linux
Solaris
AIX
Ubuntu
Fedora

Tweet

10 tips to improve Performance of Shell Scripts

   If writing good Shell scripts is a skill, writing better performance shell scripts is an art.  Shell scripts are any Unix admin's asset. It is a major time saver for repetitive activities in Unix flavors. When a shell script is written, by default, the big target in-front of the developer is to make the script running, and of course, to give the right result. As long as these scripts are intended for some user account activities which does not involve huge data, it does not matter how the result is arrived at. However, the same cannot be said on those Shell scripts which are meant to be executed on production servers containing millions of customer records present in huge files.

 

     A Shell script can give the right result, but if the result does not arrive in a considerable time, it is an issue, and yes, it can become a nightmare for the support staff. Every second is important in a production environment. A script which can be improved by even some 10 seconds can be considered an improvement depending on the environment. In this article, we will discuss about some import things which a shell script developer should keep in mind while writing shell scripts to give better performance.

Tweet

5 ways to reverse the order of file content

 In this article, we will see the different methods in which we can print the file content in reverse order.

Let us consider a file with the following contents:

$ cat file
Linux
Solaris
AIX

1. tac command is the reverse of cat. It simply prints the file in reverse order.

$ tac file
AIX
Solaris
Linux

   Note: tac is not available in all Unix flavors.

Tweet

awk - 10 examples to group data in a CSV or text file

awk is very powerful when it comes for file formatting.  In this article, we will discuss some wonderful grouping features of awk. awk can group a data based on a column or field , or on a set of columns. It uses the powerful associative array for grouping. If you are new to awk, this article will be easier to understand if you can go over the article how to parse a simple CSV file using awk.

Let us take a sample CSV file with the below contents. The file is kind of an expense report containing items and their prices. As seen, some expense items  have multiple entries.

$ cat file
Item1,200
Item2,500
Item3,900
Item2,800
Item1,600

Tweet

Right pad a string or a number with zero

 In an earlier article, we saw the different ways in which we can left pad a string with zeros. In this, let us see the different ways in which we can right pad a string or a number with zeros.

  Assuming a number, say 234, which we want to right pad to 7 digits with zeros:

1. This sed way had been discussed in example 4 of left padding a string as well. The same explanation holds good here. Only difference being instead of having a '0' before &, it is after &.

$ echo 234 | sed -e :a -e 's/^.\{1,6\}$/&0/;ta'
2340000

     One thing to note is the number 6(one less than actual, which is 7). This is because once the length of the number reaches 7, sed quits. To understand better, put 7 in place of 6 and check the result.

Tweet

Shell - Read a text or CSV file and extract data

   In one of our earlier articles on awk, we saw how easily awk can parse a file and extract data from it. Shell also has properties with which we can handle text files: files with fields separated by white spaces or CSV files in which the fields are separated by a comma delimiter. Let us see in this article how to read and parse a file field by field or column by column and extract data from it using the while loop of shell. This article will also explain the usage of shell while loop.

Let us consider a file with sample contents as shown below. The file contains 3 fields: OS, the company and a random value.

$ cat file
Solaris Sun 25
Linux RedHat 30

Tweet

7 different ways to print the last line of a file in Linux

 In an earlier article, we saw the different ways in which we can print the first line of a file.  In this article, we will see the different ways in which we can print or display the last line or the trailer record of a file in Linux.

Let us consider a file with the following contents:

$ cat file
Unix
Solaris
Linux

1. The tail is the most common command used. tail command prints the last part of the files. -1 specifies to print one line from the last part.

$ tail -1 file
Linux

Tweet

Different ways to reverse a string in Linux

 In this article, we will see the different ways in which we can reverse a string in Linux. Let us take a string "welcome" and try to reverse this string in different ways:

1. The Linux rev command is used to reverse the lines in a file. This command can take standard input as well as shown below.

$ echo welcome | rev
emoclew

Note: The rev command is not present in all flavors of Unix.

Tweet

awk - Join or merge lines on finding a pattern

 In one of our earlier articles, we had discussed about joining all lines in a file and also joining every 2 lines in a file. In this article, we will see the how we can join lines based on a pattern or joining lines on encountering a pattern using awk or gawk.

Let us assume a file with the following contents. There is a line with START in-between. We have to join all the lines following the pattern START.

$ cat file
START
Unix
Linux
START
Solaris
Aix
SCO

1. Join the lines following the pattern START without any delimiter.

$ awk '/START/{if (NR!=1)print "";next}{printf $0}END{print "";}' file
UnixLinux
SolarisAixSCO

    Basically, what we are trying to do is:  Accumulate the lines following the START and print them on encountering the next START statement. /START/ searches for lines containing the pattern START.  The command within the {} will work only on lines containing the START pattern. Prints a blank line if the line is not the first line(NR!=1). Without this condition, a blank line will come in the very beginning of the output since it encounters a START in the beginning. 

Tweet

Different ways to print the next few lines after pattern match

  grep, awk  or a sed command is used to print the line matching a particular pattern. However,  at times, we need to print a few more lines following the lines matching the pattern. In this article, we will see the different ways in which we can get this done. The first part explains ways to print one line following the pattern along with the pattern, few examples to print only the line following the pattern and in the end we have ways to print multiple lines following the pattern.

  Let us consider a file with the following contents as shown below:

$ cat file
Unix
Linux
Solaris
AIX
SCO

1. The simplest is using the grep command. In GNU grep, there is an option -A which prints lines following the pattern.

$ grep -A1 Linux file
Linux
Solaris

  In the above example, -A1 will print one line following the pattern along with the line matching the pattern. To print 2 lines after the pattern, it is -A2.

Tweet

GNU dates and strings conversion in Linux

 In one of our articles, we discussed about the GNU date command and the powerful options it has. In this article, we will see about the -d option in detail. -d option in date command can take multiple forms. Basically, it  takes a STRING and converts it to a date. And the STRING can be given in many different forms which makes the dates computation much easier. Let us see in detail with examples.

1. To convert a string "08-May-2012" to a UNIX date:

$ date -d "08-May-2012"
Tue May  8 00:00:00 IST 2012

     The output will look like as if the date command is fired on 8-May 2012. The -d option can take the string with - separated format as well. As shown , the string here takes the form  MM-Mon-YEAR format and gives the Unix date as output..

2. To format the above date output to a time stamp:

$ date -d "08-May-2012" '+%y%m%d'
120508

     %y gives YY, %m is number of the month and %d gives day.

Tweet

awk - Match a pattern in a file in Linux

  In one of our earlier articles on awk series, we had seen the basic usage of awk or gawk. In this, we will see mainly how to search for a pattern in a file in awk. Searching pattern in the entire line or in a specific column.

  Let us consider a csv file with the following contents. The data in the csv file contains kind of expense report. Let us see how to use awk to filter data from the file.

$ cat file
Medicine,200
Grocery,500
Rent,900
Grocery,800
Medicine,600

1. To print only the records containing Rent:

$ awk '$0 ~ /Rent/{print}' file
Rent,900

     ~ is the symbol used for pattern matching.  The / / symbols are used to specify the pattern. The above line indicates: If the line($0) contains(~) the pattern Rent, print the line. 'print' statement by default prints the entire line. This is actually the simulation of grep command using awk.

Tweet

Different ways to print first few characters of a string in Linux

  In this article, we will see the different ways in which we can extract and print the first 3 characters of every line in a file. Assume a file with sample contents as below:

$ cat file
Linux
Unix
Solaris 

1.  This is the best of all since its purely internal.  The file is read in the file loop and the first 3 characters are retrieved using the shell.

$ while read line
> do
>  echo ${line:0:3}
> done < file
Lin
Uni
Sol

   ${x:0:3} means to extract 3 characters from position 0 from the variable x. In this way, the shell can also be used to extract sub-string from a variable. This is one of the most important features of the shell.

2. Using cut, we can cut the first 3 characters.

$ cut -c -3 file
Lin
Uni
Sol

   The same can also be achieved by having 1-3 in place of -3.

Tweet

Linux Interview Questions - Part 1

   This article contains a few  interview questions related to Unix or Linux Shell Scripting and command-line stuff. Some of them could be commonly asked, most of them are not so. For questions for which we have a detailed explanation, a link would be provided after the answer for detailed reference. The idea of this article is not just to provide an answer to a question, instead to know of the many different options which are present to solve a particular problem.

1. What is Shell Scripting ?

        Shell scripting, in Linux or Unix,  is programming with the shell using which you can automate your tasks. A shell is the command interpreter which is the interface between the User and the kernel. A shell script allows you to submit a set of commands to the kernel in a batch. In addition, the shell itself is very powerful with many properties on its own, be it for string manipulation or some basic programming stuff. 

 Ref: What is Shell Scripting?

Tweet

How to zero pad a number or a variable?

  In this article, we will see the different ways in which we can do zero padding of a number or a variable or in other words, to put leading zeros to a number. Let us assume a variable "x" whose value is 23, and try to zero pad it to a length of 4.

1. Using printf command:
This is the simplest of all, and efficient too since it uses an internal command printf. The format specifier 4d will align the variable towards the right by 4 digits. By putting 04d, the same number will be zero padded 4 places to the right.

$ x=23
$ printf "%04d\n" $x
0023

Tweet

What is umask?

  umask, as the man page says, stands for User file creation mask which is used for determining the default permission for a new file creation. The new file creation could either be a file creation through a normal process or a file copy. umask command is a shell built-in meaning it is an internal command.

  The three file permission attributes are read, write and execute. These 3 are mapped to octal values as shown below:

          read     - 4
         write     - 2
         execute - 1

Tweet

find files by name or part of name or extension

   In one of our earlier articles on find, we saw about finding files depending on their modification time. In this article, we will see about the finding files using the filename or using a part of the file name or using the extension.

1. To find all the files ending with .c :

$ find . -name "*.c"

      The '.' indicates current directory. The default behavior of find makes this command to search in the current directory and also in all the sub-directories under the current directory for the files ending with .c.

Tweet

here-doc(EOF) issue in sqlplus or FTP

   In one of our articles, we had seen the different ways in which we can connect to sqlplus from Unix. The first way discussed was using the heredoc or the here-document method(EOF). Many times users encounter a weird issue while using the here-doc in sqlplus or FTP. The issue is no statements after the closing EOF get executed .

  Let us take an example of an update query inside sqlplus:

#!/usr/bin/ksh
echo script started
sqlplus -s guru/unix11@XE <<EOF         
     UPDATE EMP SET FNAME='BLOGGER' WHERE EMPID=10;
     COMMIT;
     exit
     EOF
echo script ended 

Tweet

What is CDPATH ?

    Has it happened to you wherein you logged in to an account or to an environment and simply printed the command, say "cd log" and your directory gets changed to some log directory. The interesting part is: The log directory is not there in the directory from where you fired the cd command. It was somewhere also, however the cd command switched to it correctly. The question now is:  If the "log" directory is not present in the location from where we are giving the "cd" command, how did we reach the log directory?  The answer to this question is most probably  the CDPATH environment variable would have been set.

1. What is CDPATH?
       CDPATH is an environment variable. It looks exactly like the PATH variable containing with many different paths concatenated using ':'.

Tweet

Different ways to print the first line of a file

 In one of our earlier articles, we saw how to add a header or a trailer record to a file.  Let us see in this article how to print the first line or the header record of a file. Assuming a file with the sample contents as shown below:

$ cat file
Unix
Linux
Solaris

1. The default command which comes to our mind is the head command. head with the option "-1" displays the first line.

$ head -1 file
Unix

Tweet

cp - 10 different copy command examples

  cp command is used to copy files in Unix . There are some options in cp which add lot of value to this command. In this article, we will see about some of the rarely used options of the cp command which makes cp really powerful.

1. Copy a file
cp command has the arguments source file and destination file. Here, we are copying file1 to file2.

$ cp file1 file2
$ ls -l file*
-rw-r--r-- 1 guru users 163 Apr 11 15:36 file1
-rw-r--r-- 1 guru users 163 Apr 11 15:40 file2

   The same thing can also be achieved using the below command. It might look tricky if you are seeing this for the first time.

Tweet

What is sourcing a file?

   UNIX users would have heard the term "sourcing a file" pretty frequently. What exactly do we mean by sourcing a file? Which files do we source? What is the use of it? Let us see in this article about this in detail.

1. What is sourcing a file? 

  Sourcing a file is nothing but executing the file, more importantly making the changes of the file applicable in the current shell itself, the shell which we are working in. Usually, when we execute a command or a script, a new sub-shell is created and the script is executed in the sub-shell. Due to this, if the script or the command intends to make any change to the current shell, it will not happen. In order to make the changes applicable in the current shell itself, we "source" the file.

Tweet

Different ways to get the filename from absolute path

   In this article, we will see the different ways from which we can get the filename from an absolute path name. Assume a variable "y" which contains the absolute path. Let us see how to extract the filename alone from the absolute path.

$ echo $y
/home/guru/perl/file.pl

Tweet

date & GNU date

  date command in UNIX is used more often to create file names, rather than to check the system date. For any of our process, if we are in need of a unique file name, then one of the best options is to have the date and time as part of the file name.

 Let us see the usage of date command:

Tweet

10 good shell scripting practices

   Everybody working in UNIX can do a decent level of shell scripting. Depending on your expertise, the kind of commands you use, the way you look at a problem, the mode of arriving at a solution could be different. For people in their early stages of shell scripting, it is good to follow certain practices which will help you  in learning the art faster, cleaner and better. We discussed about the 5 important things to follow to  become good in shell scripting in one of our earlier articles. Along the same lines, the following are the 10 points in which we will discuss about the good practices:

1. Learn less, do more: You want to do some scripting in UNIX, be it in shell, perl or python. So, you took a book and started  studying. Some people study the entire thing first and start practising. This might be a good method for some, but I do not believe too much in this. Instead, just study the basic, the very basic with which you can start with something. Once done with the basic, start writing simple programs. Gradually, build your requirement. Go back and develop your program further. If you get stuck due to lack of knowledge, go back to the book. Read what you want. Come back and start developing. Build requirement further. Read a little more if stuck. Carry on. I believe in this way than to read the entire stuff first because how much ever we read, unless we start practicing, we cannot correlate lot of things, and it does not add much value to your study either. This learn-practice-stuck method helps me a lot.

Tweet

sed - Replace or substitute file contents - Part 2

 In one of our earlier articles, we saw about Replace and substitute using sed. In continuation to it, we will see a few more frequent search and replace operations done on files using sed.

Let us consider a file with the following contents:

$ cat file
RE01:EMP1:25:2500
RE02:EMP2:26:2650
RE03:EMP3:24:3500
RE04:EMP4:27:2900

Tweet

Internal vs External commands

  UNIX commands are classified into two types

• Internal Commands - Ex: cd, source, fg
• External Commands - Ex: ls, cat

Let us look at these in detail

Internal Command:

   Internal commands are something which is built into the shell. For the shell built in commands, the execution speed is really high. It is because no process needs to be spawned for executing it.    For example, when using the "cd" command, no process is created. The current directory simply gets changed on executing it.

External Command:

  External commands are not built into the shell. These are executables present in a separate file. When an external command has to be executed, a new process has to be spawned and the command gets executed. For example, when you execute the "cat" command, which usually is at /usr/bin, the executable /usr/bin/cat gets executed.

Tweet

Join every 2 lines in a file

  In this article, we will see the different ways in which we can join every two lines in a file. We will join them using comma as the delimiter.
    Assume a file with the following contents as shown below. The file below shows stats for countries. The only issue in the file is: the stats are not directly against the country names, instead they are in the next line.

$ cat file
USA
442
India
249
UK
50

Tweet

Soft Links & Hard Links - All about Inodes - Part 2

 In one of our earlier articles, we discussed about the inode structure and its properties under the title All about Inodes. In continuation to the same, in this article, we will see about the links in UNIX: hard links and soft links, and their relationship with Inodes. We will learn them in the way of questions and answers.

Tweet

Different ways to capitalize the contents of a file

   In this article, we will see the different ways in which we can capitalize the contents of the file and also the other way around. In other words, this will tell how to convert the lowercase characters or string into upper case and upper case into lower case.

Let us take a sample file, "file1", with the following contents:

$ cat file1
hello
welcome
string

Tweet

Retrieve table names from Oracle queries

    Sometimes, you might have a requirement where you have a file full of Oracle sql queries, and you want to filter out the table names alone. In this article, we are going to look into a Perl script which is for this very purpose.  Given a file with sql queries which could be comprising of SELECT, UPDATE, INSERT or DELETE statements, the script will parse the sql query file and list out the table names alone.

Tweet

typeset - arithmetic operations

   typeset is one of those many beautiful commands which is highly underused. In simple words, typeset is the one which is used to declare variables. Interesting!! We do not usually declare variables in Shell, isn't it? Let us see in the article the use of typeset.
   typeset command has different options to use with, but we are going to discuss only the "-i" option in this article.

1. Assignment

$ typeset -i x 

Tweet

Disk Usage and Free Space- Housekeeping

  File system and directories should be kept clean and tidy, else they can anytime messup. Say, you get a mail from your system admin saying the filesystem is full, and asking you to clean up the files. Or say your TL comes and stands next to you asking you to free some space in your account so as to get some free space.How do we do it?  In this article, we will see about these house keeping stuffs.

1. How do you find out the file system capacity or free space?

The df command lists all the filesystems in the system. Infact, df -h gives more readable output:

$ df -h
Filesystem   Size  Used   Avail Use%  Mounted on
/dev/dev1    97G   6.0G   86G   7%    /
/dev/dev2    97G   96G     1G   99%   /home

Tweet

15 different ways to display the contents of a file

  Looking at the file contents is one such activity which a UNIX developer might be doing in sleep also. In this article, we will see the umpteen different ways in which we can display or print the contents of a file.

Let us consider a file named "a", with the following  contents:

a
b
c

1. The first is the most common cat command:

$ cat a
a
b
c

Tweet

Different ways to split the file contents

   In one of our earlier articles, we saw different ways to join all lines in a file. In this article, we will see the different ways in which we can split the file contents of a file.  Some even refer to this as converting rows into columns of a file.

Let us assume a file, say "a" with the following contents:

$ cat a
a,b,c,d

Tweet

Login to a UNIX account from Windows without password

   UNIX users login to their unix boxes umpteen times in a day. Every time, while logging in, we give username and passwords which, takes quite a bit of time, if you happen to login many times in a day. In this article, we will see how we can login to a UNIX account without providing username and password using ssh.

Logging into an UNIX account usually happens in 2 ways:

1. User is already in a UNIX box or server. He tries to login to another UNIX server from the first one.
2. User logs into the UNIX box using putty installed in his Windows system.

Tweet

C program to multiply, divide and find reminder for x by 4 without arithmetic operators?

  One of my friends asked me an interesting interview question. The question is like this: How to multiply, divide and find reminder of a number X by 4 without using arithmetic operators?

   Since we are not supposed to use arithmetic operators, none of the +,-,*,/% can be used. Binary operators always help us in this regard.

Tweet

join command

   join command is used to join  two files based on a key field present in both the files. The input files can be separated by white space or any delimiter. join can be looked at like the join functionality in any RDBMS(database). However, the only difference being in the database join we get only the columns which we give in the select clause, in case of unix join, you get all the columns of both the files. And yes, the join can be done using only one key field, not multiple.

1. Let us consider 2 sample files a1 and a2. a1 containing bank account numbers and the balance in the account. a2 containing bank account numbers and the account names.

Tweet