Linux uniq Command

Linux 命令大全Complete Linux Commands

The Linux uniq command is used to check and remove duplicate lines in text files, and is usually used together with the sort command.

uniq can check for duplicate lines in text files.

Syntax

uniq [-cdu][-f<栏位>][-s<字符位置>][-w<字符位置>][--help][--version][输入文件][输出文件]

Parameters：

  • -c or --count: Display the number of times each line repeats next to that line.
  • -d or --repeated: Only display duplicate lines.
  • -f<fields> or --skip-fields=<fields>: Ignore the specified fields during comparison.
  • -s<chars> or --skip-chars=<chars>: Ignore the specified characters during comparison.
  • -u or --unique: Only display lines that appear once.
  • -w<chars> or --check-chars=<chars>: Specify the characters to compare.
  • --help: Display help.
  • --version: Display version information.
  • [input file]: Specify the sorted text file. If not specified, read data from standard input;
  • [output file]: Specify the output file. If not specified, display the content to the standard output device (display terminal).

Examples

In the file testfile, lines 2, 3, 5, 6, 7, and 9 are identical. To delete duplicate lines using the uniq command, use the following command:

uniq testfile

The original content of testfile is:

$ cat testfile      #原有内容  
test 30  
test 30  
test 30  
Hello 95  
Hello 95  
Hello 95  
Hello 95  
Linux 85  
Linux 85

After using the uniq command to delete duplicate lines, the output is as follows:

$ uniq testfile     #删除重复行后的内容  
test 30  
Hello 95  
Linux 85

Check the file, delete duplicate lines, and display the number of occurrences of each line at the beginning of the line. Use the following command:

uniq -c testfile

The output result is as follows:

$ uniq -c testfile      #删除重复行后的内容  
3 test 30             #前面的数字的意义为该行共出现了3次  
4 Hello 95            #前面的数字的意义为该行共出现了4次  
2 Linux 85            #前面的数字的意义为该行共出现了2次
When duplicate lines are not adjacent, the uniq command does not work. That is, if the file content is as follows, the uniq command will not take effect:
$ cat testfile1      # 原有内容 
test 30  
Hello 95  
Linux 85 
test 30  
Hello 95  
Linux 85 
test 30  
Hello 95  
Linux 85

In this case, we can use sort:

$ sort  testfile1 | uniq
Hello 95  
Linux 85 
test 30

Count the number of times each line appears in the file:

$ sort testfile1 | uniq -c
   3 Hello 95  
   3 Linux 85 
   3 test 30

Find duplicate lines in the file:

$ sort testfile1 | uniq -d
Hello 95  
Linux 85 
test 30

Linux 命令大全Complete Linux Commands

Other Extensions