Tuesday, June 26, 2012

sed - 25 examples to delete a line or pattern in a file



In this article of  sed tutorial series, we are going to see how to delete or remove a particular line or a particular pattern from a file using the sed command.

 Let us consider a file with the sample contents as below:
$ cat file
Cygwin
Unix
Linux
Solaris
AIX
1. Delete the 1st line or the header line:
$ sed '1d' file
Unix
Linux
Solaris
AIX
      d command is to delete a line. 1d means to delete the first line.

     The above command will show the file content by deleting the first line. However, the source file remains unchanged. To update the original file itself with this deletion or to make the changes permanently in the source file, use the -i option. The same is applicable for all the other examples.
sed -i '1d' file
    Note: -i option in sed is available only if it is GNU sed. If not GNU, re-direct the sed output to a file, and rename the output file to the original file.

2. Delete a particular line, 3rd line in this case:
$ sed '3d' file
Cygwin
Unix
Solaris
AIX
3. Delete the last line or the trailer line of the file:
$ sed '$d' file
Cygwin
Unix
Linux
Solaris
       $ indicates the last line.

4. Delete a range of lines, from 2nd line till 4th line:
$ sed '2,4d' file
Cygwin
AIX
       The range is specified using the comma operator.

5. Delete lines other than the specified range, line other than 2nd till 4th here:
$ sed '2,4!d' file
Unix
Linux
Solaris
       The ! operator indicates negative condition.

6. Delete the first line AND the last line of a file, i.e, the header and trailer line of a file.
$ sed '1d;$d' file
Unix
Linux
Solaris
     Multiple conditions are separated using the ';' operator. Similarly, say to delete 2nd and 4th line, you can use: '2d;3d'.

7. Delete all lines beginning with a particular character, 'L' in this case:
$ sed '/^L/d' file
Cygwin
Unix
Solaris
AIX
      '^L' indicates lines beginning with L.

8. Delete all lines ending with a particular character, 'x' in this case:
$ sed '/x$/d' file
Cygwin
Solaris
AIX
      'x$' indicates lines ending with 'x'. AIX did not get deleted because the X is capital.

9. Delete all lines ending with either x or X,  i.e case-insensitive delete:
$ sed '/[xX]$/d' file
Cygwin
Solaris
    [xX] indicates either 'x' or 'X'. So, this will delete all lines ending with either small 'x' or capital 'X'.

10. Delete all blank lines in the file
$ sed '/^$/d' file
Cygwin
Unix
Linux
Solaris
AIX
      '^$' indicates lines containing nothing and hence the empty lines get deleted. However, this wont delete lines containing only some blank spaces.

11. Delete all lines which are empty or which contains just some blank spaces:
$ sed '/^ *$/d' file
Cygwin
Unix
Linux
Solaris
AIX
      '*' indicates 0 or more occurrences of the previous character. '^ *$' indicates a line containing zero or more spaces. Hence, this will delete all lines which are either empty or lines with only some blank spaces.

12. Delete all lines which are entirely in capital letters:
$ sed '/^[A-Z]*$/d' file
Cygwin
Unix
Linux
Solaris
      [A-Z] indicates any character matching the alphabets in capital.

13. Delete the lines containing the pattern 'Unix'.
$ sed '/Unix/d' file
Cygwin
Linux
Solaris
AIX
     The pattern is specified within a pair of slashes.

14. Delete the lines NOT containing the pattern 'Unix':
$ sed '/Unix/!d' file
Unix
15. Delete the lines containing the pattern 'Unix' OR 'Linux':
$ sed '/Unix\|Linux/d' file
Cygwin
Solaris
AIX
     The OR condition is specified using the | operator. In order not to get the pipe(|) interpreted as a literal, it is escaped using a backslash.

16. Delete the lines starting from the 1st line till encountering the pattern 'Linux':
$ sed '1,/Linux/d' file
Solaris
AIX
    Earlier, we saw how to delete a range of lines. Range can be in many combinations: Line ranges, pattern ranges, line and pattern, pattern and line.

17. Delete the lines starting from the pattern 'Linux' till the last line:
$ sed '/Linux/,$d' file
Cygwin
Unix
18. Delete the last line ONLY if it contains the pattern 'AIX':
$ sed '${/AIX/d;}' file
Cygwin
Unix
Linux
Solaris
      $ is for the last line. To delete a particular line only if it contains the pattern AIX, put the line number in place of the $. This is how we can implement the 'if' condition in sed.

19. Delete the last line ONLY if it contains either the pattern 'AIX' or 'HPUX':
$ sed '${/AIX\|HPUX/d;}' file
Cygwin
Unix
Linux
Solaris
20. Delete the lines containing the pattern 'Solaris' only if it is present in the lines from 1 to 4.
$ sed '1,4{/Solaris/d;}' file
Cygwin
Unix
Linux
AIX
      This will only delete the lines containing the pattern Solaris only if it is in the 1st four lines, nowhere else.

21. Delete the line containing the pattern 'Unix' and also the next line:
$ sed '/Unix/{N;d;}' file
Cygwin
Solaris
AIX
     N command reads the next line in the pattern space. d deletes the entire pattern space which contains the current and the next line.

22. Delete only the next line containing the pattern 'Unix', not the very line:
$ sed '/Unix/{N;s/\n.*//;}' file
Cygwin
Unix
Solaris
AIX
     Using the substitution command s, we delete from the newline character till the end, which effective deletes the next line after the line containing the pattern Unix.

23. Delete the line containing the pattern 'Linux', also the line before the pattern:
$ sed -n '/Linux/{s/.*//;x;d;};x;p;${x;p;}' file | sed '/^$/d'
Cygwin
Solaris
AIX
    A little tricky ones. In order to delete the line prior to the pattern,we store every line in a buffer called as hold space. Whenever the pattern matches, we delete the content present in both, the pattern space which contains the current line, the hold space which contains the previous line.
   Let me explain this command: 'x;p;' ; This gets executed for every line. x exchanges the content of pattern space with hold space. p prints the pattern space. As a result, every time, the current line goes to hold space, and the previous line comes to pattern space and gets printed. When the pattern /Linux/ matches, we empty(s/.*//) the pattern space, and exchange(x) with the hold space(as a result of which the hold space becomes empty) and delete(d) the pattern space which contains the previous line. And hence, the current and the previous line gets deleted on encountering the pattern Linux. The ${x;p;} is to print the last line which will remain in the hold space if left.
  The second part of sed is to remove the empty lines created by the first sed command.

24. Delete only the line prior to the line containing the pattern  'Linux', not the very line:
$  sed -n '/Linux/{x;d;};1h;1!{x;p;};${x;p;}' file
Cygwin
Linux
Solaris
AIX
     This is almost same as the last one with few changes. On encountering the pattern /Linux/, we exchange(x) and delete(d). As a result of exchange, the current line remains in hold space, and the previous line which came into pattern space got deleted.
    1h;1!{x;p;} -  1h is to move the current line to hold space only if it first line. Exchange and print for all the other lines. This could easily have been simply: x;p . The drawback is it gives an empty line at the beginning because during the first exchange between the pattern space and hold space, a new line comes to pattern space since hold space is empty.

25. Delete the line containing the pattern 'Linux', the line before, the line after:
$ sed -n '/Linux/{N;s/.*//;x;d;};x;p;${x;p;}' file | sed '/^$/d'
Cygwin
AIX
     With the explanations of the last 2 commands, this should be fairly simple to understand.

69 comments:

  1. Nice collection of commands !! Keep posting such Good articles !! :)

    ReplyDelete
  2. Thank you for this great article. I will use as a reference! I have a question about number 24. How to delete line before Linux only if it is empty or has only white spaces? Thank you again!

    ReplyDelete
    Replies
    1. This should do James:

      $ sed -n '/Linux/{x;/^ *$/d;x;};1h;1!{x;p;};${x;p;}' file

      Delete
  3. Thank you Guru! You are very good! I am new to Shell scripting, only been at it for a few months. I bookmarked this article for future reference.

    ReplyDelete
  4. Excellent article!

    How can I do to delete all lines between 2 parameters?

    i.e.: I want to remove all lines between 'Linux' and 'AIX' from the file called 'test' and I don't know how many lines are between tehy, so I need to use two parameters.

    Thank you in advance

    ReplyDelete
    Replies
    1. almost same as in example 17..just replace $ with /AIX/

      Delete
  5. great article with examples Guru, appreciate it.

    ReplyDelete
  6. Nice article......... can u please post commond for
    how to get top 10 records and last 10 records of a file.

    ReplyDelete
    Replies
    1. Use the head(to get top 10) and tail(to get last 10) commands for this.

      Delete
  7. Good examples.Can you tell me to delete a record if set of columns are null/blank.Say total 10 columns in which 2,5,7,8 columns are coming as null/blank for some records.Those should be removed from the file.I am using awk command but how will be the performance if millions of records are there in a file.

    ReplyDelete
    Replies
    1. Something like this(not tested):
      awk '{if ($2 =="" && $5 == "")next}1' file
      It might take a few seconds to execute it, but its ok. Whenever a file is big, just make sure every command you use for per record processing should be justified to be there.
      For any technical questions, please post the question in our forum: forum.theunixschool.com

      Delete
  8. Hi, Very nice read... however, I was not able to delete a line from a csv file which contained the pattern ',Pub,Totals:, and so, on,' So i used the below command.

    sed '/^,Pub,Totals:/d' $filename... any help is greatly appreciated.

    Thanks
    Dhruuv

    ReplyDelete
    Replies
    1. The command you used is correct one. However, only if your file content is beginning with ',Pub,...', use the ^, else drop it.

      Delete
  9. Hi,

    very good example and easy to understand. But i have a question. How to delete lines and matched patterns between 2 patterns without deleting the second pattern? i used this:

    sed '/pattern1/,/pattern/{//p;d;}' inputfile

    It deleted everything between the patterns but remain the matched patterns. I want to delete pattern1 too. appreciate your help on this.

    Thanks.

    ReplyDelete
    Replies
    1. sed '/pattern1/,/pattern/{/pattern1/d;p;}' inputfile

      Delete
  10. It is do helpful! I wonder if there is a way to delete more than 1 line before a pattern?

    ReplyDelete
  11. hi,

    how to delete last 10 lines from a file.

    thanks,

    ReplyDelete
    Replies
    1. x=$(wc -l <file)
      x=$(expr $x - 10)
      sed -n "1,$x p" file

      Delete
  12. Hi Guru, great article. I know this might be really straightforward, but I don't seem to get the right syntax. I want to delete the line below a pattern (not the pattern) only if it is blank (you answered this for the line above, but I can't make it work for my case). Thank you.

    ReplyDelete
    Replies
    1. To delete a line after the pattern if it is blank:

      sed '/Unix/{n;/^$/d;}' file

      Delete
  13. Hi

    How I delele only second and fourth matching line , and replace only second an fourth matching line.

    Thank you

    ReplyDelete
  14. Hi

    how do i Delete only the 3rd next line containing the pattern 'Unix', not the very line.

    Suppose i have a file-->

    tr
    Ansh
    Anshuman
    Raj
    L
    M
    M
    N

    So i want to remove only the third line after tr.

    ReplyDelete
  15. Very good collection of sed commands!

    ReplyDelete
  16. Hi Guru, Nice Article!

    Two doubts:-
    I have one XML having multiple header segments. I need to delete multiple segments(Part of header) from a large XML file(around 150 MB):
    The header segment consist of following segments:
    Transaction-834
    Internal-Properties
    Segment-ST
    Segment-BGN

    I used the following command:
    sed -e '/Transaction-834/d;/Internal-Properties/d;/Segment-ST/d;/Segment-BGN/d' Input_File > New_File

    With this command I am able to delete these multiple segments for a small XML file, but when I am trying to delete from this huge file (150 MB size) I am getting following error: "input line too long".

    Any idea how to read and delete the segments from this huge XML file using sed or any other command?
    Also is there any limit upto which sed command is able to read the file and perform action?

    Thanks in advance!

    ReplyDelete
    Replies
    1. Older sed has these problems. Use GNU sed. Try inserting a new line character after the long line and try once.

      Delete
  17. Hi Guru,
    This was so helpful. I have a quick question.
    I need to match a string and delete the previous lines (not the matched line) from a CSV file. My file would look like:
    6,ZYX 0006/11262013,SLK,SLK,ABN,10:00:45.302765,19:00:00.000000,19:00:00.000000,19:00:00.000000,0,0,0,0,0,0,0,0,0,0,0,0,N
    7,ZYX 0007/11262013,SLK,SLK,ADI,10:00:45.302774,19:00:00.000000,19:00:00.000000,19:00:00.000000,0,0,0,0,0,0,0,0,0,0,0,0,N
    8,ZYX 0008/11262013,SLK,SLK,ADI,10:00:45.302780,19:00:00.000000,19:00:00.000000,19:00:00.000000,0,0,0,0,0,0,0,0,0,0,0,0,N
    920,ABY 4531/11262013,#DM02,SLK,KEY,10:28:36.221893,10:28:36.234889,19:00:00.000000,19:00:00.000000,0,0,0,0,0,0,0,0,0,0,0,0,N
    921,AAA 0110/11262013,#DM02,SLK,MDZ,10:28:36.233397,10:28:36.311391,19:00:00.000000,19:00:00.000000,0,0,0,0,0,0,0,0,0,0,0,0,N

    ReplyDelete
  18. Hi guru, i want to delete after and before character and get the number in between.
    2h31m45s
    5h05m44s
    want to get only number 31 or 05 between h and m

    ReplyDelete
    Replies
    1. $ echo 2h31m45s | sed -r 's/.*h([^m]+)m.*/\1/'
      31

      Delete
  19. These are great! Is there a way to delete a pattern itself (without deleting the line it occurs on) in a file? I have a csv file that has filenames in the third column. I need to strip off the extension ONLY, and keep everything else. (e.g. remove .flv) from each line that has it in the column. Thanks!

    ReplyDelete
    Replies
    1. To just delete a pattern, check this link:
      http://www.theunixschool.com/2014/08/sed-examples-remove-delete-chars-from-line-file.html

      Delete
  20. hi Guru,

    I am trying to get the contents between two instances of a word.

    Basically I want to delete all lines before the 3rd instance of the word "cheese" and I want to remove all lines (including the containing one) of the 4th instance of the word cheese. So I am left with what is in between only.

    Is this possible using just sed?

    ReplyDelete
    Replies
    1. Please post this in Q&A(tab before contact). Also give an example input file and output expected.

      Delete
  21. Hello,

    How to delete line number 3,6,9,12,15,18,21,24 and so on? The pattern is line number +3.

    Thanks

    ReplyDelete
  22. Hi,

    How do I delete all entries in a file A from another file B that contains some IDs?
    For instance, file A has data like
    A1 1 2 3
    A2 2 3 4
    A3 3 4 5

    And file B has entries like
    A1
    A2

    How do I delete all lines from file A whereever ids match to file B?

    ReplyDelete
  23. I Want to delete the record if I find a word in the fixed length(5-7). How can I do this?

    ReplyDelete
    Replies
    1. awk is better for this. To get lines whose length is not 5:
      $ awk 'length($0)!=5' file

      Delete
  24. How to delete a line till the time pattern is matched. This line contains pattern more than twice.
    e.g.
    The main class is going to start after class 4.

    Result should be : "class is going to start after class 4."

    I am using sed command : sed -e 's/.*\(class.*\)/\1/'
    This is giving incorrect result "class 4"

    Can you please share how can i get the desired result using sed.

    ReplyDelete
    Replies
    1. $ x="The main class is going to start after class 4"
      $ echo $x | sed 's/[^c]*\(class.*\)/\1/'
      class is going to start after class 4
      $

      Delete
  25. Hi Guru,

    I need delete the line containing the pattern 'Linux', the 2 lines before and the line after

    Before:
    AIX
    HP
    Linux
    Solaris
    Tru64

    After command:
    Tru64

    Can you help me?
    Thanks

    ReplyDelete
  26. Hi Guru,

    I have the following text in one file
    pattern1
    line1
    line2
    pattern2
    line3
    line4
    line5
    pattern1
    line6
    line7
    pattern2
    line8
    line9
    pattern1
    line10
    pattern2

    I want to replace the first occurrence of the block between pattern1 and pattern2 with some-text and delete the remaining 2 occurrences of the matching block. So the output should look like this:
    some-text
    line3
    line4
    line5
    line8
    line9

    Help please. Thanks!

    ReplyDelete
  27. I have a csv file which contains 65000 lines (Size approximately 28 mb). In each of the lines certain path in the beginning is given e.g. "c:\abc\bcd\def\123\456". Now let say the path "c:\abc\bcd\" is common in all the lines and rest of the content is different. What i have to do is to remove the common part (In this case "c:\abc\bcd\") from all the lines.

    Can any of you please help me out with this ? Also apologies in advance if i have mistakenly not followed any of the forum rules to ask this question.

    ReplyDelete
  28. Anurag,
    A simple sed command like the following can accomplish this. First I created a short sample file with two matches and also one similar path starting with "d" instead of "c".

    $ more inputfile
    This is a sample file with heading
    c:\abc\bcd\rest\of\path
    c:\abc\bcd\more
    d:\abc\bcd\more
    d:\abc\def\ghi

    My sed command:
    sed 's&c:\\abc\\bcd\\&&' inputfile > outputfile

    The resulting output is:
    $ more outputfile
    This is a sample file with heading
    rest\of\path
    more
    d:\abc\bcd\more
    d:\abc\def\ghi

    - The "&" is used as a sed delimiter to make it more readable, since your pattern includes backslashes.
    - The double backslashes "\\" are necessary since \ is used to escape special characters.
    Hope this helps!
    R

    ReplyDelete
    Replies
    1. Thanks for the help. I'll try this and get back.

      Delete
  29. This is gold information . Thank you for sharing it with us.

    ReplyDelete
  30. Totally agree with Ivan's comment above.Thank you very much Guru.

    I have a.txt as follows and I am looking for b.txt
    Header
    Page: 1
    column1 column2
    - - - - - - - -
    data11 data21
    data12 data22
    data13 data23
    data14 data24
    Header
    Page: 2
    column1 column2
    - - - - - - - -
    data15 data25
    data16 data26
    data17 data27
    data18 data28
    Header
    Page: 3
    column1 column2
    - - - - - - - -
    data19 data29
    data20 data30

    Expected output b.txt

    Header
    Page: 1
    column1 column2
    - - - - - - - -
    data11 data21
    data12 data22
    data13 data23
    data14 data24
    data15 data25
    data16 data26
    data17 data27
    data18 data28
    data19 data29
    data20 data30

    I tried with sed '/Header/,/---/'d a.txt >b.txt
    but in that process I am losing first occurance of header as well.
    I wish to start above command from line 5, however I could not build one.
    Tried sed '5,/Header/,/---/'d a.txt >b.txt, but its seems wrong syntax.

    any inputs, help?

    ReplyDelete
    Replies
    1. To get from 5th line, you can do like this:

      sed '5,${/Header/,/- - -/d;}' a.txt

      Alternatively, if the content of a.txt is something you got out of a sql query, by adding the "set pagesize 0", automatically you will get the header only once..

      Delete
    2. Hi Guruprasad,

      I have another issue referencing same example above, only change is every input file has Trailer info at the end appearing only once (showing the filtering/selection criteria).

      I wish to retain first header, deleting all in headers appearing on subsequent pages. The solution provided by you perfectly works in that case. however in the process trailer (end of report) information appearing at the end also gets removed. Do we have any way to retain it .

      Thank you very much.

      Delete
  31. Hi Guru,

    Help me out in this please.

    i have text file abc.txt
    L1137Q1 shantanu 040013.15 L Prashant .00
    L134AQ1 Prashant .00 L shantanu 040013.15
    L1337Q1 shantanu 040013.15 L Prashant .00

    i need to delete line containing pattern " Prashant" in 2nd coloumn only,

    output must be

    L1137Q1 shantanu 040013.15 L Prashant .00
    L1337Q1 shantanu 040013.15 L Prashant .00

    Thank you in advance.

    ReplyDelete
    Replies
    1. This is a typical candidate for awk:

      awk '$2 !="Prashant"' file

      Delete
  32. what is the command to find a word in a line, then deleting (that line and previous line)

    example, I want to search for the word JDK, delete the JDK line and the previous line.

    INPUT
    -------------------------
    Version 5.1
    Version 5.1.1.12
    ID BASE
    Version 1.4.2
    ID JDK
    Version 5.1
    Version 1.4.2
    ID JDK
    Version 5.1.1.12
    ID ND


    OUTPUT
    ---------------------------
    Version 5.1
    Version 5.1.1.12
    ID BASE
    Version 5.1
    Version 5.1.1.12
    ID ND

    ReplyDelete
  33. How do you "sed" out multiple particular lines?

    Say I want to sed line # 10,11,16,17,38,37 and 40?

    ReplyDelete
  34. I'm facing a issue in deleting a line having 'NUL' special character.
    I used the below command
    sed '/^@/d' test.csv
    but this is removing only the 'NUL' character and not the line it contains. Can someone help on this.

    ReplyDelete
  35. 23. doesn't work in ksh on AIX :( man! I really need it!

    ReplyDelete
  36. One of the best article which explains useful sed command with examples

    ReplyDelete
  37. Thanks for the examples. Could you show us how to, as a variation of 25?

    Delete the line containing the pattern 'Linux', the line before, 2 lines after.

    ReplyDelete
  38. Hi, how do i delete the last n lines? i mean.. i have

    1111
    2222
    3333
    4444
    5555

    and i want to remove from 3 till the end and get this

    1111
    2222

    Thanks in advance

    ReplyDelete
    Replies
    1. $ y=$(wc -l<file)
      $ let x=y-3+1
      $ sed "$x,$y d" file



      Delete
  39. Hi, how do I delete everything between two patterns, including the patterns. The start and end of pattern could be on different lines.
    E.g. if input is
    start aaa end bbb ccc
    aaa start bb end cc
    aa bb cc
    dd start ee
    ff end gg

    The output should be
    bbb ccc
    aaa cc
    aa bb cc
    dd
    gg

    Thanks!

    ReplyDelete
  40. That was really awesome stuff I learnt today!!!! Thanks for the detailed explanation

    ReplyDelete
  41. Hello, ur tutorial is really being helpfull for beginners.........
    in my case i need to check every 17th col of the lines (row)in a file and if a particular character is found (say B) i need to delete the whole line...... please help

    ReplyDelete
  42. Hello.

    How can i delete lines where all the points in the line are same

    line 1: 40 50 60 70 80
    line 2: 30 50 50 90 100
    line 3: 50 50 50 50 50
    line 4: 20 15 20 32 45

    How can i delete only lines like line 3 where all the points are 50. Line 1 and 2 also has 50 but not for all the points.

    Thanks

    ReplyDelete