處理文字檔時，去除重複行是極高頻需求。uniq 是 Linux 標準工具箱中專為「相鄰行去重」設計的利器，但許多使用者忽略其「僅檢測相鄰行」的核心限制，導致結果不符預期。本文系統梳理 uniq 核心機制、完整參數表、實戰範例與常見陷阱，助您精準駕馭文字去重任務。
uniq 不會 掃描整份檔案尋找重複，它只比較「當前行」與「緊鄰前一行」。因此，前置排序（sort）是去除全域重複的必要步驟。若輸入已有序，可直接使用 sort -u 一步到位。
| 短參數 | 長參數 | 功能說明 |
|---|---|---|
| -c | --count | 每行前加上出現次數計數器 |
| -d | --repeated | 僅輸出重複行（每組僅顯示一行） |
| -D | --all-repeated[=分隔模式] | 輸出所有重複行（含重複行本身），可選 separate/prepend/append 分隔組 |
| -f N | --skip-fields=N | 忽略前 N 個欄位（以空白分隔）再比較 |
| -i | --ignore-case | 比較時不區分大小寫 |
| -s N | --skip-chars=N | 忽略每行前 N 個字元再比較 |
| -u | --unique | 僅輸出不重複行（去重後的唯一行） |
| -w N | --check-chars=N | 僅比較每行前 N 個字元 |
| -z | --zero-terminated | 以 NUL (\0) 而非換行符作為行結束符（配合 sort -z、find -print0） |
測試檔 uniqtest.txt 內容：
this is a test
this is a test
this is a test
i am tank
i love tank
i love tank
this is a test
whom have a try
WhoM have a try
you have a try
i want to abroad
those are good men
we are good men
$ sort uniqtest.txt | uniq
i am tank
i love tank
i want to abroad
those are good men
we are good men
whom have a try
WhoM have a try
you have a try
this is a test
或直接 sort -u uniqtest.txt。
$ sort uniqtest.txt | uniq -c
1 i am tank
2 i love tank
1 i want to abroad
1 those are good men
1 we are good men
1 whom have a try
1 WhoM have a try
1 you have a try
4 this is a test
$ sort uniqtest.txt | uniq -d
i love tank
this is a test
$ sort uniqtest.txt | uniq -D
i love tank
i love tank
this is a test
this is a test
this is a test
this is a test
$ sort -f uniqtest.txt | uniq -i
i am tank
i love tank
i want to abroad
those are good men
we are good men
whom have a try
you have a try
this is a test
whom 與 WhoM 合併為一行。
$ sort uniqtest.txt | uniq -f 1 # 忽略第一欄（如 "this" "i" "whom"）
$ sort uniqtest.txt | uniq -s 4 # 忽略前 4 字元
應用場景：日誌首欄為時間戳，僅比對訊息內容時極實用。
$ sort uniqtest.txt | uniq -w 2
僅比對 "th"、"i "、"wh"、"yo"、"iw"、"th"、"we"，前兩字相同視為重複。
$ sort uniqtest.txt | uniq -u
i am tank
i want to abroad
those are good men
we are good men
whom have a try
WhoM have a try
you have a try
i love tank、this is a test 因重複被完全移除。
uniq file 僅能去除連續重複，務必 sort file | uniq 或 sort -u file。-i 或配合 sort -f。-f 預設以空白（空格/Tab）分欄，若資料含 CSV、固定寬度格式，請先用 cut/awk 預處理或改用 sort -k、awk 邏輯。sort -S 50% --parallel=4 限制記憶體並啟用多執行緒，或改用 awk '!seen[$0]++' 串流去重（不保證順序但常數記憶體）。find -print0 | sort -z | uniq -z 全程 NUL 分隔。awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -20(head -1 data.csv; tail -n +2 data.csv | sort -u) > dedup.csvcat -n file | sort -k2 | uniq -f1 -Duniq 雖小，配合 sort、awk、cut 卻能解決 90% 文字去重與統計需求。牢記「先排序、後去重」，善用 -c/-d/-u/-f/-s/-i/-w 組合，即可高效處理日誌分析、資料清洗、報表彙總等場景。
