near-duplicate-detection

Image similarity in Golang. Version 4 (LATEST)

ISCC: International Standard Content Code

Test-contamination detector - finds train/eval overlap via deterministic shingle matching, per-source rollups, CI-ready exit codes. Standard library only.

A Simple Image Clustering Script using CLIP and Hierarchial Clustering

Fast image similarity search with hash tables (Golang). Version 2 (LATEST)

Python library for detecting near duplicate texts in a corpus at scale.

Simple library for finding duplicate and near-duplicate text documents in massive sets/libraries/databases

Fast image similarity search with hash tables (Golang). Version 1

Multi module project focused on near-duplicate search for images.

Holds code for near-duplicate image parser using optimized image classifiers.

Scalable near-duplicate image detection using Vision Transformers, Siamese Networks, Triplet Loss, Deep Hashing, and FAISS.

an application for comparing images using various image hashing algorithms

Bachelor's Thesis on Near-Duplicate Image Detection. This repo contains all resources, code, and documentation developed during the process.

Code similarity search by compression - find near-duplicate files, clone families, and where a pasted snippet came from. No embeddings, no model, no vector database.

Find duplicated files including permutations where underscores replace spaces. Allow a tolerance and use a signature for audio files to ignore metadata variations.

Analytics pipeline for detecting coordinated inauthentic behavior on social media. Fuses six evidence channels into a weighted graph, then applies Leiden community detection to surface candidate account networks for analyst review. Detects coordination, never automates attribution.

Exploiting the PyTerrier library to build a Search Engine and resolve the Near Duplicate Detection tasks.

Charikar SimHash near-duplicate detection in pure Go — 64-bit fingerprints plus a banded-LSH index for near-neighbor lookup and clustering. Zero dependencies.

Discount corpus wide boilerplate before near duplicate scoring, without the duplicates suppressing their own text

Perceptual video deduplication on Backblaze B2: hash a video library with videohash, cluster near-duplicates by Hamming distance, and reclaim storage. Local, open-source, keyless compute — B2 holds the library, the hash index, and every cluster report.