CSV Deduplication Tool

Supports single-file and batch CSV data deduplication, matching by specified columns, custom retention strategies, and exporting unique and duplicate data.

Bug report

Browser execution mode: Your data is processed in your browser and is not uploaded to the server.

Speed and Stability: Processing speed depends on your device and browser. For large batch work, the desktop version may be more stable.

Loading tool, please wait...

If the online tool fails to load or run, try the desktop tool.https://tools.yikeaigc.com/

Tool Usage

Return to old version

Set up first, then dedupe CSV data

Supports pasted data, single files, and batch deduping. Find duplicates by all columns or selected columns, and export unique data, duplicates, or a batch ZIP.

CSV / TXT Match selected columns Batch ZIP Savable settings

Import CSV data

In single file mode, upload one file or paste CSV directly. In batch mode, select multiple files at once. The list shows only the first 20, but all files will be processed.

Select or drop CSV files Supports .csv and .txt. File content is read locally only.

Selected files 0 files

Data preview -

Preview and download results

For a single file, download unique and duplicate data separately. Batch files keep original names, append numbers for duplicates, and are packed as a ZIP.

0Files

0Original rows

0Rows after dedup

0Duplicate rows

No results yet. Import data, confirm settings, then click “Start deduping”.

Single-file results -

Unique data preview-

Duplicate data preview-

Removal reason-

Batch Results -

File Name	Original row	Deduped	Duplicate rows	Duplicate rate	Status

Instructions

Software Usage Instructions

Import CSV data: Select single-file mode to upload a CSV/TXT file, or paste CSV content directly; in batch mode, you can select multiple CSV/TXT files. The interface only displays the first 20 files, but all files will be included during processing.
Parse data: Set the input delimiter, file encoding, whether the first row is a header, and whether to trim leading and trailing spaces from cells, then click “Parse Preview” to view column names and data rows.
Set deduplication criteria:
- Match all columns: Rows are considered duplicates when all fields in the entire row are identical.
- Match specified columns: Determine duplicates based on key columns such as email, phone number, or customer ID; in batch mode, you can enter column numbers.
Select a retention strategy:
- Keep first occurrence: Keeps the earlier row in each duplicate group.
- Keep last occurrence: Keeps the later row in each duplicate group.
- Delete all duplicates: Moves all data rows in duplicate groups to the duplicate data results.
Adjust advanced options: You can set options such as case handling, merging consecutive whitespace, ignoring punctuation, empty value handling, output delimiter, whether to export duplicate data, and whether to add a UTF-8 BOM.
Save settings: For frequently used rules, click “Save Settings” to automatically restore them the next time you open the tool.
Start processing and download: Click “Start Deduplication” to view statistics and preview results; for a single file, you can download the deduplicated results and duplicate data separately, while for batch files, you can download a ZIP archive.

FAQ

A: Match all columns requires every field in the entire row to be identical to count as a duplicate; Match specified columns only compares the selected key columns, such as determining whether customers are duplicates based only on the email column.

A: Keep first occurrence is suitable for retaining original records; keep last occurrence is suitable for retaining updated records; delete all duplicates is suitable when you only want to keep data with no duplicate relationships.

A: The processed file names will be kept as close to the original file names as possible. If multiple files have the same name, the system will append a number to the file name to prevent files in the ZIP package from overwriting each other.

A: Yes. In the parsing settings, change the file encoding to “GBK / GB18030”, then parse the preview again. If garbled text appears, switch the encoding and parse again.

A: It depends on the “Case handling” setting. When “Case-insensitive” is checked, ABC and abc will be compared as the same content; when unchecked, they will be treated as different content.

A: After processing is complete, you can switch to the “Duplicate Data” preview, or click “Download Duplicate Data” to export a CSV. In batch mode, you need to check “Also export duplicate data”.