URL Batch Filtering and Processing Tool

Batch extract URLs from TXT text, filter and deduplicate by domain, suffix, and protocol.

Browser execution mode: Your data is processed in your browser and is not uploaded to the server.

Speed and Stability: Processing speed depends on your device and browser. For large batch work, the desktop version may be more stable.

Loading tool, please wait...

If the online tool fails to load or run, try the desktop tool.https://tools.yikeaigc.com/

Tool Usage

Return to old version

Complete filter and output settings before processing. File content is read, parsed, and packaged only on this page. For batch selection, the UI shows the first 20 files; processing covers all selected files.

No data selected Rules set to default Output set to default

1Select TXT or text files

Click to choose files, or drag them here Supports TXT, CSV, LOG, HTML, JSON, and other text files; folders can also be selected for batch reading.

If multiple files are selected, the preview list shows only the first 20; processing and packaging include all files.

2File Overview

Selected Files0

File Size0 KB

Shown0

Source ModeFile

Select files, paste URL text, or load sample data.

3Paste URL List

4Sample Data

1Parse Method and Output

Extraction Mode

For mixed text, choose “Extract all URLs”.

Text Encoding

Protocol Filter

Root Domainexample.com Full Domainwww.example.com Protocol + Domainhttps://www.example.com Full URLKeep path; parameters configurable Original FragmentExtracted original text Domain Suffixcom / com.cn

2Domain Suffix and Host Rules

Suffix Filter Mode

Custom Suffix

Common suffixes

Require domain suffixFilter local hosts like localhost Keep IP addressesSupports IPv4 / IPv6 hosts Remove leading www.Normalize domain output

3Keywords, paths, and deduplication

Domain includes keyword

Domain excludes keyword

Path includes keyword

Deduplication scope

Deduplicate by

Result sorting

1Output file structure

Output mode

Per-file output keeps original names; duplicates get numbers added automatically.

Result format

Multi-column separator

Output Protocol

Full URL Parameters

Empty result files

2ZIP and Extra Files

ZIP file name

Merged result file name

Generate statistics reportCSV format, records count per file Generate unrecognized fragmentsHelps troubleshoot invalid URLs Add BOM to CSVHelps spreadsheet software detect Chinese

Make sure data sources, filter settings, and output settings are complete. After you start, all selected files will be processed on this page and a ZIP download will be generated.

Waiting to process0%

Scanning files0

URLs found0

Matched0

Deduplicate0

Output file0

PreviewResults

Not processed

No.	Source file	Output content	Suffix	Status
Not processed yet. Complete the settings, then click Start.

Instructions

Software Usage Instructions

Select a data source: On the “Data Source” page, select text files such as TXT, CSV, or LOG. You can also select a folder, paste a URL list, or load sample data to try the workflow.
Set the parsing method:
- Extract all URLs from text: Suitable for mixed content such as logs, webpage source code, and JSON snippets.
- Process one URL per line: Suitable for already organized URL lists.
Select output content: Choose result types such as root domain, full domain, protocol + domain, full URL, original snippet, and domain suffix.
Configure filtering rules:
- Suffix filtering: Keep only or exclude domain suffixes such as .com, .cn, .com.cn, and .co.uk.
- Protocol filtering: Allow any protocol, or keep only HTTP or HTTPS.
- Keyword filtering: Filter by domains that contain keywords, domains to exclude, or paths containing keywords.
- Deduplication settings: Supports global deduplication, deduplication within each file, or keeping duplicate results.
Set the output method: Output by individual file while keeping the original file names, output into folders by result type, or merge into a single result file.
Start processing and download: Go to the “Process & Download” page and click “Start Processing”. After completion, click “Download ZIP Results” to save the processed files.

FAQ

A: A root domain is the main domain, such as example.com; a full domain keeps subdomains such as www, blog, and shop; a protocol domain keeps http or https, such as https://www.example.com.

A: Common reasons include incomplete formatting, insufficient domain levels, abnormal characters, a protocol other than HTTP/HTTPS, or filtering by suffix, keyword, or protocol rules. Check “Generate unrecognized snippets” to view the troubleshooting file.

A: You can directly select common suffixes, or enter .com, .net, .com.cn, etc. in custom suffixes. Separate multiple suffixes with commas, semicolons, or line breaks.

A: When outputting files one by one, the original file names will be preserved. If multiple files have the same name, the system will append a number to the file name, such as list.txt or list(2).txt, to prevent results from overwriting each other.

A: Choose “global deduplication” when you need to clean multiple files together; choose “deduplicate within each file” when you want each file to keep its own set of results; choose “no deduplication” when you need to preserve frequency counts or original duplicate records.

A: No. File content is read, filtered, and packaged on the current page and will not be uploaded to the server. It is suitable for processing data containing internal links, log snippets, or site lists.

A: By default, it includes the processed results generated using the original file names and a statistics report. If “Unrecognized Fragments” is enabled, an additional troubleshooting file will also be generated, recording URL fragments that could not be parsed and the reasons.