Alat Penyaring Kemiripan Nama File Teks

Mengelompokkan file TXT/HTML berdasarkan kemiripan nama file, lalu secara otomatis mempertahankan file ideal dalam setiap grup berdasarkan metrik konten seperti jumlah kata dan jumlah kalimat, untuk deduplikasi yang efisien.

Laporan bug

Mode eksekusi browser: Data Anda diproses di browser dan tidak diunggah ke server.

Kecepatan dan stabilitas: Kecepatan pemrosesan bergantung pada perangkat dan browser Anda. Untuk pekerjaan batch besar, versi desktop mungkin lebih stabil.

Memuat alat, harap tunggu...

Jika alat online tidak dapat dimuat atau dijalankan, coba alat desktop. https://tools.yikeaigc.com/

Penggunaan Alat

Kembali ke versi lama

Tips pemrosesan:Konfirmasi ambang kemiripan, algoritma, dan aturan simpan di “Pengaturan Filter” sebelum mulai. File hanya dibaca dan dikemas di halaman ini, tidak diunggah ke server.

1Pilih file teks

Klik pilih, atau seret file TXT / HTML ke sini

Mendukung pilihan massal; hanya 20 pertama dipratinjau, semua file diproses saat filter

2Data contoh

File dipilih

Ukuran total

0 KB

Jenis teks

Jumlah pratinjau

3Pratinjau file

Belum ada file dipilih. Anda dapat memuat data contoh dahulu, lalu memproses file sendiri setelah memeriksa pengaturan.

1Aturan grup

Ambang kemiripan nama file

76%

Semakin tinggi nilainya, nama file harus makin mirip untuk masuk grup yang sama.

Algoritma kemiripan

Algoritma gabungan cocok untuk nama file campuran berbahasa Mandarin, Inggris, dan bernomor.

Metode pengelompokan

Pengelompokan tautan lebih longgar, file wakil lebih stabil.

2Aturan simpan

Dasar simpan dalam grup

Metrik hanya dibandingkan dalam grup mirip dan tidak mengubah file asli.

Penanganan skor sama

Digunakan saat skor dasar simpan sama.

File konten kosong

Cocok untuk mengecualikan file placeholder yang teksnya kosong setelah dibaca.

3Baca & normalisasi

Encoding teks

Deteksi otomatis memprioritaskan BOM dan encoding Tionghoa umum.

Nama file ZIP

Kosongkan untuk memakai nama default.

Baris pratinjau hasil

Hanya memengaruhi tampilan halaman, bukan hasil paket.

Abaikan ekstensi saat membandingkan Abaikan huruf besar/kecil Samakan spasi, tanda hubung, dan tanda baca Lemahkan penanda duplikat, copy, dan versi Kelompokkan langsung jika isi teks sama persis Tambahkan laporan CSV dan JSON dalam ZIP

1Progres & hasil

Menunggu proses. Pilih file dan atur parameter dulu. 0%

Setelah selesai, grup, file tersimpan, file terfilter, dan tombol unduh akan tampil di sini.

2Pratinjau grup mirip

Belum ada hasil grup.

3Pratinjau file tersaring

Belum ada file tersaring.

4Log proses

Menunggu mulai.

', 86400000 * 5), createFile('product_detail_page 副本.htm', '

商品详情页

页面包含规格参数和购买须知。

', 86400000 * 4), createFile('product-detail-page-2026.html', '

商品详情页

页面包含规格参数、适用场景、售后说明、购买须知、常见问题和配送说明。

', 86400000 * 2), createFile('about-us.html', '

关于我们

公司简介、服务范围和联系方式。

', 86400000) ]; const mixed = [ createFile('Report_A_2026_Final.txt', 'Report A. Revenue: 126000. Cost: 84200. Notes: checked by finance team.', 86400000 * 8), createFile('report-a-2026-final copy.txt', 'Report A. Revenue: 126000. Cost: 84200.', 86400000 * 7), createFile('Report A 2026 v3.txt', 'Report A. Revenue: 126000. Cost: 84200. Notes: checked by finance team. Added channel split and summary.', 86400000 * 2), createFile('客户名单_北京.txt', '张三\n李四\n王五\n赵六\n钱七', 86400000 * 3), createFile('客户名单-北京-副本.txt', '张三\n李四\n王五', 86400000 * 2), createFile('客户名单上海.txt', '陈一\n周二\n吴三\n郑四', 86400000) ]; if (kind === 'html') setFiles(html); else if (kind === 'mixed') setFiles(mixed); else setFiles(article); appendLog('已载入示例数据，可以切换到筛选设置进行试算。'); switchTab('tf-tab-settings'); } function normalizeFilename(name, options) { let value = options.ignoreExt ? fileBase(name) : name; try { value = value.normalize('NFKC'); } catch (e) { // Older browsers may not support normalize; keep original text. } if (options.normalizeCase) value = value.toLowerCase(); if (options.removeCopy) { value = value .replace(/\b(copy|copied|final|draft|new|old)\b/gi, ' ') .replace(/(副本|复件|拷贝|复制|修订版|定稿|草稿|新版|旧版)/g, ' ') .replace(/[\s_-]*(v|ver|version|版本)\s*\d+(\.\d+)?/gi, ' ') .replace(/[\s_-]*$?\d{4}[-_.年]?\d{1,2}?[-_.月]?\d{0,2}日?$?/g, ' ') .replace(/[\s_-]*[$（]\d+[$）]/g, ' '); } if (options.normalizeSymbols) { value = value .replace(/[·•,，.。;；:：!！?？'"“”‘’`~@#$%^&*+=|\\/<>《》【】\[\]{}()（）]/g, ' ') .replace(/[-_—–]+/g, ' ') .replace(/\s+/g, ''); } else { value = value.replace(/\s+/g, ' ').trim(); } return value.trim(); } function levenshteinSimilarity(a, b) { if (a === b) return 100; const lenA = a.length; const lenB = b.length; if (!lenA || !lenB) return lenA === lenB ? 100 : 0; let previous = new Array(lenB + 1); let current = new Array(lenB + 1); for (let j = 0; j <= lenB; j++) previous[j] = j; for (let i = 1; i <= lenA; i++) { current[0] = i; for (let j = 1; j <= lenB; j++) { const cost = a[i - 1] === b[j - 1] ? 0 : 1; current[j] = Math.min(previous[j] + 1, current[j - 1] + 1, previous[j - 1] + cost); } const temp = previous; previous = current; current = temp; } return (1 - previous[lenB] / Math.max(lenA, lenB)) * 100; } function lcsSimilarity(a, b) { if (a === b) return 100; const lenA = a.length; const lenB = b.length; if (!lenA || !lenB) return lenA === lenB ? 100 : 0; let prev = new Array(lenB + 1).fill(0); let curr = new Array(lenB + 1).fill(0); for (let i = 1; i <= lenA; i++) { for (let j = 1; j <= lenB; j++) { curr[j] = a[i - 1] === b[j - 1] ? prev[j - 1] + 1 : Math.max(prev[j], curr[j - 1]); } const temp = prev; prev = curr; curr = temp.fill(0); } return (2 * prev[lenB] / (lenA + lenB)) * 100; } function charSetSimilarity(a, b) { const setA = new Set(Array.from(a)); const setB = new Set(Array.from(b)); if (!setA.size && !setB.size) return 100; let intersection = 0; setA.forEach((char) => { if (setB.has(char)) intersection++; }); const union = new Set([...setA, ...setB]).size; return union ? intersection / union * 100 : 0; } function cosineSimilarity(a, b) { const count = (text) => { const map = new Map(); Array.from(text).forEach((char) => map.set(char, (map.get(char) || 0) + 1)); return map; }; const mapA = count(a); const mapB = count(b); const chars = new Set([...mapA.keys(), ...mapB.keys()]); if (!chars.size) return 100; let dot = 0; let magA = 0; let magB = 0; chars.forEach((char) => { const av = mapA.get(char) || 0; const bv = mapB.get(char) || 0; dot += av * bv; magA += av * av; magB += bv * bv; }); if (!magA || !magB) return 0; return dot / (Math.sqrt(magA) * Math.sqrt(magB)) * 100; } function filenameSimilarity(a, b, algorithm) { if (a === b) return 100; if (!a || !b) return 0; if (algorithm === 'levenshtein') return levenshteinSimilarity(a, b); if (algorithm === 'sequence') return lcsSimilarity(a, b); if (algorithm === 'jaccard') return charSetSimilarity(a, b); if (algorithm === 'cosine') return cosineSimilarity(a, b); const lev = levenshteinSimilarity(a, b); const seq = lcsSimilarity(a, b); const jac = charSetSimilarity(a, b); const cos = cosineSimilarity(a, b); return lev * 0.42 + seq * 0.28 + jac * 0.14 + cos * 0.16; } function hashText(text) { let hash = 5381; for (let i = 0; i < text.length; i++) { hash = ((hash << 5) + hash) ^ text.charCodeAt(i); hash >>>= 0; } return hash.toString(16); } function stripHtml(raw, name) { const looksHtml = /\.html?$/i.test(name) || /<\/?[a-z][\s\S]*>/i.test(raw); if (!looksHtml) return raw; try { const doc = new DOMParser().parseFromString(raw, 'text/html'); doc.querySelectorAll('script,style,noscript,template,svg,canvas').forEach((node) => node.remove()); return (doc.body ? doc.body.textContent : doc.documentElement.textContent) || raw.replace(/<[^>]*>/g, ' '); } catch (e) { return raw.replace(/<[^>]*>/g, ' '); } } function detectEncoding(buffer, preferred) { if (preferred && preferred !== 'auto') return preferred; const bytes = new Uint8Array(buffer); if (bytes[0] === 0xef && bytes[1] === 0xbb && bytes[2] === 0xbf) return 'utf-8'; if (bytes[0] === 0xff && bytes[1] === 0xfe) return 'utf-16le'; if (bytes[0] === 0xfe && bytes[1] === 0xff) return 'utf-16be'; if (window.jschardet && bytes.length > 0) { try { let probe = ''; const limit = Math.min(bytes.length, 8000); for (let i = 0; i < limit; i++) probe += String.fromCharCode(bytes[i]); const detected = window.jschardet.detect(probe); const name = String(detected.encoding || '').toLowerCase(); if (/gb|gbk|gb2312|gb18030/.test(name)) return 'gb18030'; if (/big5/.test(name)) return 'big5'; if (/utf-16le/.test(name)) return 'utf-16le'; if (/utf-16be/.test(name)) return 'utf-16be'; if (/utf-8|ascii/.test(name)) return 'utf-8'; } catch (e) { return 'utf-8'; } } return 'utf-8'; } function decodeBuffer(buffer, encoding) { const candidates = [encoding, 'utf-8', 'gb18030']; for (const item of candidates) { try { return new TextDecoder(item, { fatal: false }).decode(buffer); } catch (e) { // Try next candidate. } } return new TextDecoder().decode(buffer); } function countHtmlTagRatio(raw) { if (!raw) return 0; const matches = raw.match(/<[^>]+>/g) || []; const tagLength = matches.reduce((sum, item) => sum + item.length, 0); return tagLength / raw.length; } function analyzeText(raw, file) { const clean = stripHtml(raw, file.name) .replace(/\u00a0/g, ' ') .replace(/\r\n?/g, '\n') .trim(); const noPunc = clean.replace(/[\s.,;:!?，。！？；：“”‘’"'、()（）《》<>【】\[\]{}\-_=+\\/|`~@#$%^&*…·]/g, ''); const charCount = Array.from(noPunc).length; const cjkUnits = clean.match(/[\u3400-\u9fff]{1,2}/g) || []; const latinWords = clean.match(/[A-Za-z0-9]+(?:[._-][A-Za-z0-9]+)*/g) || []; const sentenceParts = clean.split(/[。！？!?；;\n]+/).map((s) => s.trim()).filter(Boolean); const paragraphs = clean.split(/\n\s*\n+/).map((s) => s.trim()).filter(Boolean); const digits = clean.match(/\d/g) || []; const wordCount = cjkUnits.length + latinWords.length; return { rawLength: raw.length, cleanText: clean, char_count_no_punc: charCount, word_count_segmented: wordCount, sentence_count: sentenceParts.length, paragraph_count: paragraphs.length || (clean ? 1 : 0), avg_sentence_length: sentenceParts.length ? charCount / sentenceParts.length : 0, digit_ratio: charCount ? digits.length / charCount : 0, html_tag_ratio: countHtmlTagRatio(raw), content_hash: hashText(clean.replace(/\s+/g, '')) }; } async function readAndAnalyzeFile(file, options, index) { const buffer = await file.arrayBuffer(); const encoding = detectEncoding(buffer, options.encoding); const raw = decodeBuffer(buffer, encoding); const metrics = analyzeText(raw, file); return { index, file, name: file.name, ext: fileExt(file.name), size: file.size || 0, lastModified: file.lastModified || 0, encoding, normalizedName: normalizeFilename(file.name, options), metrics }; } function enrichQualityScores(filesInfo) { const max = { chars: Math.max(1, ...filesInfo.map((info) => info.metrics.char_count_no_punc)), words: Math.max(1, ...filesInfo.map((info) => info.metrics.word_count_segmented)), sentences: Math.max(1, ...filesInfo.map((info) => info.metrics.sentence_count)), paragraphs: Math.max(1, ...filesInfo.map((info) => info.metrics.paragraph_count)) }; filesInfo.forEach((info) => { const m = info.metrics; const contentPart = (m.char_count_no_punc / max.chars) * 42 + (m.word_count_segmented / max.words) * 24 + (m.sentence_count / max.sentences) * 18 + (m.paragraph_count / max.paragraphs) * 10; const htmlPenalty = Math.min(10, m.html_tag_ratio * 20); const emptyPenalty = m.char_count_no_punc === 0 ? 24 : 0; m.quality_score = Math.max(0, contentPart + 6 - htmlPenalty - emptyPenalty); }); } function sleepFrame() { return new Promise((resolve) => requestAnimationFrame(resolve)); } function makePairKey(a, b) { return a < b ? `${a}:${b}` : `${b}:${a}`; } function getSimilarityValue(infoA, infoB, options, pairScores) { const key = makePairKey(infoA.index, infoB.index); if (pairScores && pairScores.has(key)) return pairScores.get(key); if (options.exactContent && infoA.metrics.content_hash === infoB.metrics.content_hash && infoA.metrics.cleanText) { if (pairScores) pairScores.set(key, 100); return 100; } const score = filenameSimilarity(infoA.normalizedName, infoB.normalizedName, options.algorithm); if (pairScores) pairScores.set(key, score); return score; } async function buildConnectedGroups(filesInfo, options) { const n = filesInfo.length; const adj = Array.from({ length: n }, () => []); const pairScores = new Map(); const total = n * (n - 1) / 2 || 1; let done = 0; for (let i = 0; i < n; i++) { for (let j = i + 1; j < n; j++) { if (cancelRequested) throw new Error('处理已取消'); const score = getSimilarityValue(filesInfo[i], filesInfo[j], options, pairScores); if (score >= options.threshold) { adj[i].push(j); adj[j].push(i); } done++; if (done % 120 === 0 || done === total) { setProgress(45 + done / total * 35, `正在比较文件名相似度：${done}/${total}`); await sleepFrame(); } } } const visited = new Array(n).fill(false); const groups = []; for (let start = 0; start < n; start++) { if (visited[start]) continue; const queue = [start]; const members = []; visited[start] = true; while (queue.length) { const item = queue.shift(); members.push(item); adj[item].forEach((next) => { if (!visited[next]) { visited[next] = true; queue.push(next); } }); } groups.push({ memberIndexes: members }); } return { groups, pairScores }; } async function buildRepresentativeGroups(filesInfo, options) { const pairScores = new Map(); const groups = []; let done = 0; const total = Math.max(1, filesInfo.length * Math.max(1, filesInfo.length - 1) / 2); for (let i = 0; i < filesInfo.length; i++) { if (cancelRequested) throw new Error('处理已取消'); let placed = false; for (const group of groups) { const rep = filesInfo[group.memberIndexes[0]]; const score = getSimilarityValue(filesInfo[i], rep, options, pairScores); done++; if (score >= options.threshold) { group.memberIndexes.push(i); placed = true; break; } } if (!placed) groups.push({ memberIndexes: [i] }); if (i % 20 === 0 || i === filesInfo.length - 1) { setProgress(45 + Math.min(35, done / total * 35), `正在按代表文件分组：${i + 1}/${filesInfo.length}`); await sleepFrame(); } } return { groups, pairScores }; } function parseRetention(value) { const direction = value.endsWith('_asc') ? 'asc' : 'desc'; const metric = value.replace(/_(asc|desc)$/, ''); return { metric, direction }; } function metricValue(info, metric) { if (metric === 'file_size') return info.size || 0; return Number(info.metrics[metric] || 0); } function tieBreakCompare(a, b, options) { if (options.tiebreak === 'modified_new') return (b.lastModified || 0) - (a.lastModified || 0); if (options.tiebreak === 'modified_old') return (a.lastModified || 0) - (b.lastModified || 0); if (options.tiebreak === 'name_short') return a.name.length - b.name.length; if (options.tiebreak === 'size_large') return (b.size || 0) - (a.size || 0); return a.index - b.index; } function chooseKeptFile(memberInfos, options) { const parsed = parseRetention(options.retention); const sorted = [...memberInfos].sort((a, b) => { const av = metricValue(a, parsed.metric); const bv = metricValue(b, parsed.metric); if (Math.abs(av - bv) > 0.000001) { return parsed.direction === 'asc' ? av - bv : bv - av; } return tieBreakCompare(a, b, options); }); return sorted[0]; } function averageSimilarity(memberInfos, pairScores) { if (memberInfos.length <= 1) return 100; let sum = 0; let count = 0; for (let i = 0; i < memberInfos.length; i++) { for (let j = i + 1; j < memberInfos.length; j++) { const score = pairScores.get(makePairKey(memberInfos[i].index, memberInfos[j].index)); if (typeof score === 'number') { sum += score; count++; } } } return count ? sum / count : 0; } function decideRetention(filesInfo, rawGroups, pairScores, options) { const groups = []; const retained = []; const discarded = []; let groupNumber = 1; rawGroups.forEach((rawGroup) => { const memberInfos = rawGroup.memberIndexes.map((idx) => filesInfo[idx]); const emptyMembers = options.emptyAction === 'discard' ? memberInfos.filter((info) => info.metrics.char_count_no_punc === 0) : []; const eligibleMembers = options.emptyAction === 'discard' ? memberInfos.filter((info) => info.metrics.char_count_no_punc > 0) : memberInfos; let keep = null; if (eligibleMembers.length) { keep = chooseKeptFile(eligibleMembers, options); retained.push({ info: keep, groupId: groupNumber }); } memberInfos.forEach((info) => { if (!keep || info.index !== keep.index) { const emptyReason = info.metrics.char_count_no_punc === 0 && options.emptyAction === 'discard'; discarded.push({ info, groupId: groupNumber, reason: emptyReason ? '空内容排除' : '同组筛选' }); } }); groups.push({ id: groupNumber, members: memberInfos, keep, discarded: memberInfos.filter((info) => !keep || info.index !== keep.index), avgSimilarity: averageSimilarity(memberInfos, pairScores) }); groupNumber++; }); return { groups, retained, discarded }; } function uniqueZipName(originalName, usedNames) { const sanitized = String(originalName || 'file.txt').replace(/[\\/:*?"<>|]+/g, '_'); const dot = sanitized.lastIndexOf('.'); const base = dot > 0 ? sanitized.slice(0, dot) : sanitized; const ext = dot > 0 ? sanitized.slice(dot) : ''; let candidate = sanitized || 'file.txt'; let counter = 2; while (usedNames.has(candidate.toLowerCase())) { candidate = `${base}_${counter}${ext}`; counter++; } usedNames.add(candidate.toLowerCase()); return candidate; } function buildReportRows(result) { const rows = []; result.groups.forEach((group) => { group.members.forEach((info) => { const kept = group.keep && info.index === group.keep.index; rows.push({ group_id: group.id, status: kept ? '保留' : '筛选', filename: info.name, normalized_name: info.normalizedName, group_size: group.members.length, avg_similarity: group.avgSimilarity.toFixed(2), size: info.size, encoding: info.encoding, chars_no_punc: info.metrics.char_count_no_punc, words: info.metrics.word_count_segmented, sentences: info.metrics.sentence_count, avg_sentence_length: info.metrics.avg_sentence_length.toFixed(2), digit_ratio: (info.metrics.digit_ratio * 100).toFixed(2) + '%', html_tag_ratio: (info.metrics.html_tag_ratio * 100).toFixed(2) + '%', quality_score: info.metrics.quality_score.toFixed(2) }); }); }); return rows; } function rowsToCsv(rows) { const headers = [ 'group_id', 'status', 'filename', 'normalized_name', 'group_size', 'avg_similarity', 'size', 'encoding', 'chars_no_punc', 'words', 'sentences', 'avg_sentence_length', 'digit_ratio', 'html_tag_ratio', 'quality_score' ]; const lines = [headers.join(',')]; rows.forEach((row) => { lines.push(headers.map((key) => escapeCsv(row[key])).join(',')); }); return '\ufeff' + lines.join('\n'); } function buildJsonReport(result) { return JSON.stringify({ generatedAt: new Date().toISOString(), options: result.options, summary: result.summary, groups: result.groups.map((group) => ({ id: group.id, avgSimilarity: Number(group.avgSimilarity.toFixed(2)), kept: group.keep ? group.keep.name : null, members: group.members.map((info) => ({ filename: info.name, status: group.keep && info.index === group.keep.index ? 'keep' : 'discard', metrics: { charsNoPunc: info.metrics.char_count_no_punc, words: info.metrics.word_count_segmented, sentences: info.metrics.sentence_count, avgSentenceLength: Number(info.metrics.avg_sentence_length.toFixed(2)), digitRatio: Number((info.metrics.digit_ratio * 100).toFixed(2)), htmlTagRatio: Number((info.metrics.html_tag_ratio * 100).toFixed(2)), qualityScore: Number(info.metrics.quality_score.toFixed(2)) } })) })) }, null, 2); } function renderEmptyResult() { latestResult = null; setProgress(0, '等待处理。请先选择文件并设置参数。'); els.resultPlaceholder.style.display = 'block'; els.resultContent.style.display = 'none'; els.groupsEmpty.style.display = 'block'; els.groupsTable.style.display = 'none'; els.groupsBody.innerHTML = ''; els.groupsMore.textContent = ''; els.discardEmpty.style.display = 'block'; els.discardTable.style.display = 'none'; els.discardBody.innerHTML = ''; els.discardMore.textContent = ''; els.downloadZip.disabled = true; els.downloadCsv.disabled = true; els.downloadJson.disabled = true; } function renderResults(result, mode) { const options = result.options; const similarGroups = result.groups.filter((group) => group.members.length > 1); const ungrouped = result.groups.filter((group) => group.members.length === 1).length; const previewLimit = options.previewLimit; els.resultPlaceholder.style.display = 'none'; els.resultContent.style.display = 'block'; els.resTotal.textContent = result.summary.totalFiles; els.resGroups.textContent = similarGroups.length; els.resKeep.textContent = result.retained.length; els.resDrop.textContent = result.discarded.length; els.resTime.textContent = `${(result.summary.elapsedMs / 1000).toFixed(2)}s`; els.resultPills.innerHTML = ''; [ `算法：${els.algorithm.options[els.algorithm.selectedIndex].text}`, `阈值：${options.threshold}%`, `保留：${metricLabels[options.retention] || '自定义指标'}`, `单文件组：${ungrouped}` ].forEach((text) => { const pill = document.createElement('span'); pill.className = 'tf-pill'; pill.textContent = text; els.resultPills.appendChild(pill); }); els.downloadZip.disabled = mode === 'preview' || result.retained.length === 0; els.downloadCsv.disabled = false; els.downloadJson.disabled = false; const shownGroups = result.groups.slice(0, previewLimit); els.groupsBody.innerHTML = ''; if (shownGroups.length) { shownGroups.forEach((group) => { const tr = document.createElement('tr'); const keepName = group.keep ? group.keep.name : '无保留文件'; const cells = [ group.id, keepName, group.members.length, `${group.avgSimilarity.toFixed(1)}%`, metricLabels[options.retention] || options.retention ]; cells.forEach((value, idx) => { const td = document.createElement('td'); td.textContent = value; if (idx === 1) td.className = 'tf-mono'; tr.appendChild(td); }); els.groupsBody.appendChild(tr); }); els.groupsEmpty.style.display = 'none'; els.groupsTable.style.display = 'block'; els.groupsMore.textContent = result.groups.length > previewLimit ? `还有 ${result.groups.length - previewLimit} 个分组未在页面展示，可在报告中查看。` : ''; } else { els.groupsEmpty.style.display = 'block'; els.groupsTable.style.display = 'none'; els.groupsMore.textContent = ''; } const shownDiscarded = result.discarded.slice(0, previewLimit); els.discardBody.innerHTML = ''; if (shownDiscarded.length) { shownDiscarded.forEach((item) => { const tr = document.createElement('tr'); const nameTd = document.createElement('td'); nameTd.textContent = item.info.name; nameTd.className = 'tf-mono'; const groupTd = document.createElement('td'); groupTd.textContent = item.groupId; const statusTd = document.createElement('td'); const badge = document.createElement('span'); badge.className = 'tf-badge-drop'; badge.textContent = item.reason; statusTd.appendChild(badge); tr.append(nameTd, groupTd, statusTd); els.discardBody.appendChild(tr); }); els.discardEmpty.style.display = 'none'; els.discardTable.style.display = 'block'; els.discardMore.textContent = result.discarded.length > previewLimit ? `还有 ${result.discarded.length - previewLimit} 个筛选文件未在页面展示，可在报告中查看。` : ''; } else { els.discardEmpty.style.display = 'block'; els.discardTable.style.display = 'none'; els.discardMore.textContent = ''; } } async function runProcessing(mode) { if (busy) return; if (!selectedFiles.length) { appendLog('请先选择文件或加载示例数据。'); switchTab('tf-tab-files'); return; } if (selectedFiles.length < 2) { appendLog('文件数量不足，至少需要 2 个文件才能进行相似度筛选。'); return; } const options = getOptions(); const startTime = performance.now(); cancelRequested = false; setBusy(true); latestResult = null; els.log.textContent = ''; switchTab('tf-tab-results'); setProgress(2, '开始读取文件内容...'); appendLog(`准备处理 ${selectedFiles.length} 个文件。`); try { const filesInfo = []; for (let i = 0; i < selectedFiles.length; i++) { if (cancelRequested) throw new Error('处理已取消'); const info = await readAndAnalyzeFile(selectedFiles[i], options, i); filesInfo.push(info); setProgress(5 + (i + 1) / selectedFiles.length * 35, `正在读取：${info.name}`); if (i % 8 === 0) await sleepFrame(); } enrichQualityScores(filesInfo); appendLog('文件内容统计完成。'); setProgress(43, '开始计算文件名相似度...'); const groupBuild = options.groupMode === 'representative' ? await buildRepresentativeGroups(filesInfo, options) : await buildConnectedGroups(filesInfo, options); if (cancelRequested) throw new Error('处理已取消'); setProgress(83, '正在按保留规则筛选...'); const decided = decideRetention(filesInfo, groupBuild.groups, groupBuild.pairScores, options); const elapsedMs = performance.now() - startTime; const reportRows = buildReportRows(decided); const result = { ...decided, options, reportRows, csv: rowsToCsv(reportRows), json: '', summary: { totalFiles: selectedFiles.length, retainedFiles: decided.retained.length, discardedFiles: decided.discarded.length, similarGroups: decided.groups.filter((group) => group.members.length > 1).length, elapsedMs } }; result.json = buildJsonReport(result); latestResult = result; setProgress(100, mode === 'preview' ? '分组预览完成。' : '筛选完成，可以下载结果。'); appendLog(`处理完成：保留 ${result.retained.length} 个文件，筛选 ${result.discarded.length} 个文件。`); renderResults(result, mode); } catch (error) { setProgress(0, error.message === '处理已取消' ? '处理已取消。' : '处理失败。'); appendLog(error.message || '处理失败。'); } finally { setBusy(false); } } async function downloadZip() { if (!latestResult || !latestResult.retained.length) return; if (!window.JSZip) { appendLog('JSZip 组件未加载，暂时无法打包下载。'); return; } els.downloadZip.disabled = true; appendLog('正在生成 ZIP 文件...'); const zip = new JSZip(); const usedNames = new Set(); latestResult.retained.forEach((item) => { const outputName = uniqueZipName(item.info.name, usedNames); zip.file(outputName, item.info.file); }); if (latestResult.options.includeReport) { zip.file('筛选报告.csv', latestResult.csv); zip.file('筛选报告.json', latestResult.json); } try { const blob = await zip.generateAsync({ type: 'blob' }, (metadata) => { setProgress(metadata.percent, `正在打包 ZIP：${metadata.percent.toFixed(1)}%`); }); let name = latestResult.options.zipName || '文本文件名相似度筛选结果.zip'; if (!/\.zip$/i.test(name)) name += '.zip'; downloadBlob(blob, name); setProgress(100, 'ZIP 已生成。'); appendLog('ZIP 下载已触发。'); } catch (error) { appendLog(`ZIP 生成失败：${error.message || error}`); } finally { els.downloadZip.disabled = false; } } function downloadCsv() { if (!latestResult) return; downloadBlob(new Blob([latestResult.csv], { type: 'text/csv;charset=utf-8' }), '文本文件名相似度筛选报告.csv'); } function downloadJson() { if (!latestResult) return; downloadBlob(new Blob([latestResult.json], { type: 'application/json;charset=utf-8' }), '文本文件名相似度筛选报告.json'); } els.tabs.forEach((tab) => tab.addEventListener('click', () => switchTab(tab.dataset.tab))); els.selectFiles.addEventListener('click', () => els.fileInput.click()); els.dropzone.addEventListener('click', () => els.fileInput.click()); els.dropzone.addEventListener('keydown', (event) => { if (event.key === 'Enter' || event.key === ' ') { event.preventDefault(); els.fileInput.click(); } }); els.fileInput.addEventListener('change', () => setFiles(els.fileInput.files)); els.clearFiles.addEventListener('click', () => { selectedFiles = []; els.fileInput.value = ''; renderFilePreview(); renderEmptyResult(); appendLog('已清空文件列表。'); }); els.toSettings.addEventListener('click', () => switchTab('tf-tab-settings')); els.backFiles.addEventListener('click', () => switchTab('tf-tab-files')); els.goSettings.addEventListener('click', () => switchTab('tf-tab-settings')); els.threshold.addEventListener('input', () => { els.thresholdValue.textContent = els.threshold.value; }); els.previewGroups.addEventListener('click', () => runProcessing('preview')); els.start.addEventListener('click', () => runProcessing('process')); els.cancel.addEventListener('click', () => { cancelRequested = true; appendLog('正在尝试取消处理...'); }); els.downloadZip.addEventListener('click', downloadZip); els.downloadCsv.addEventListener('click', downloadCsv); els.downloadJson.addEventListener('click', downloadJson); $$('.tf-sample-btn').forEach((button) => { button.addEventListener('click', () => loadSample(button.dataset.sample)); }); ['dragenter', 'dragover'].forEach((eventName) => { els.dropzone.addEventListener(eventName, (event) => { event.preventDefault(); event.stopPropagation(); els.dropzone.classList.add('dragover'); }); }); ['dragleave', 'drop'].forEach((eventName) => { els.dropzone.addEventListener(eventName, (event) => { event.preventDefault(); event.stopPropagation(); els.dropzone.classList.remove('dragover'); }); }); els.dropzone.addEventListener('drop', (event) => { const files = event.dataTransfer ? event.dataTransfer.files : []; setFiles(files); }); renderFilePreview(); renderEmptyResult(); })();

Petunjuk

Petunjuk Penggunaan Perangkat Lunak

Pilih file atau muat contoh:Klik “Pilih file” untuk memilih banyak file teks seperti .txt, .html, .htm, atau gunakan data contoh terlebih dahulu untuk pengujian. Halaman hanya menampilkan pratinjau 20 file pertama, tetapi semua file akan dihitung saat diproses.
Atur pengelompokan kemiripan:
- Ambang kemiripan:Semakin tinggi nilainya, semakin mirip nama file yang diperlukan agar masuk ke grup yang sama.
- Algoritma kemiripan:Dapat memilih algoritma gabungan, jarak edit Levenshtein, rasio pencocokan berurutan, Jaccard, atau kosinus frekuensi karakter.
- Metode pengelompokan:Pengelompokan rantai kemiripan cocok untuk file dengan perubahan versi yang berurutan; pengelompokan file perwakilan cocok untuk merapikan file berdasarkan nama acuan.
Pilih dasar penyimpanan:Dapat menentukan file yang dipertahankan dalam grup berdasarkan skor kualitas konten gabungan, jumlah karakter teks murni, jumlah kata, jumlah kalimat, panjang kalimat rata-rata, persentase angka, persentase tag HTML, atau ukuran file.
Konfigurasikan pembacaan dan output:Dapat mengatur encoding teks, nama file ZIP, jumlah baris pratinjau, serta memilih apakah mengabaikan ekstensi, huruf besar/kecil, penanda salinan, penanda nomor versi, dan apakah menambahkan laporan CSV dan JSON ke dalam ZIP.
Mulai penyaringan:Setelah mengklik “Mulai penyaringan”, alat akan membaca file, mengekstrak isi utama, menghitung kemiripan nama file, membuat grup, dan mempertahankan file sesuai aturan.
Lihat dan unduh:Setelah pemrosesan selesai, Anda dapat melihat pratinjau grup, pratinjau file hasil penyaringan, dan log pemrosesan. Klik “Unduh file yang dipertahankan ZIP” untuk mendapatkan hasil. Jika ada file dengan nama yang sama dalam hasil, angka akan ditambahkan di belakang nama file untuk membedakannya.

Pertanyaan Umum

J: Disarankan mulai menguji dari ambang default. Jika ambang lebih tinggi, hanya file dengan nama yang mirip yang akan dikelompokkan; jika ambang lebih rendah, file dengan perbedaan nama yang lebih besar juga mungkin masuk ke grup yang sama. Anda dapat menggunakan “Pratinjau grup” terlebih dahulu untuk melihat hasilnya, lalu melakukan penyaringan resmi.

J: Levenshtein terutama mengukur jarak edit nama file, cocok untuk file dengan perubahan nama kecil; algoritma gabungan menggabungkan jarak edit, pencocokan berurutan, kumpulan karakter, dan frekuensi karakter, sehingga lebih fleksibel untuk nama file yang mencampur bahasa Mandarin, bahasa Inggris, nomor urut, dan penanda salinan.

Jawab: Ini untuk menjaga tampilan halaman tetap rapi dan pengoperasian tetap lancar. Jumlah pratinjau tidak memengaruhi cakupan pemrosesan; semua file yang dipilih secara massal akan ikut dalam perhitungan kemiripan dan pengemasan ZIP.

Jawab: File yang dipertahankan akan sebisa mungkin menggunakan nama file asli. Jika ada file dengan nama yang sama di dalam ZIP, sistem akan menambahkan angka di belakang nama file asli, misalnya 文章_2.txt, untuk menghindari penimpaan.

Jawab: File HTML akan terlebih dahulu mengekstrak teks utama yang terlihat dan menghapus konten seperti skrip serta gaya, lalu menghitung metrik seperti jumlah karakter, jumlah kata, dan jumlah kalimat. Laporan juga akan mencatat proporsi tag HTML untuk memudahkan penilaian kepadatan konten halaman.

Jawab: Tidak. Pembacaan file, statistik konten, pengelompokan kemiripan, dan pengemasan ZIP semuanya dilakukan di halaman lokal; server tidak akan menerima konten file Anda.

Jawab: Tidak. File asli tidak akan dipindahkan, diganti namanya, atau dihapus. Penyaringan hanya menentukan file mana yang masuk ke paket hasil unduhan; file yang tidak dipertahankan akan dicantumkan dalam laporan dan pratinjau halaman.