How to select similar sets in SQL
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.
Introduction
Selecting similar sets in SQL means finding groups of records that share common elements, such as orders with overlapping products, users with shared interests, or documents with matching tags. The core techniques are self-joins for pairwise comparison, Jaccard similarity for measuring overlap, and set intersection/difference operators. The key challenge is performance: naive approaches create O(n^2) comparisons, so indexing and pre-filtering are critical.
Setup: Example Data
Finding Exact Matching Sets
Orders that contain exactly the same products:
GROUP_CONCAT (MySQL) or STRING_AGG (PostgreSQL) creates a string signature of the sorted set. Identical signatures mean identical sets.
PostgreSQL Version
Counting Shared Elements (Intersection Size)
The self-join on product_id finds all shared products. HAVING COUNT(*) >= 2 filters for pairs with at least 2 common items.
Jaccard Similarity
Jaccard similarity = |intersection| / |union|. Ranges from 0 (no overlap) to 1 (identical sets):
Finding Sets That Contain a Given Subset
Which orders contain ALL of products {101, 102}?
The HAVING COUNT = N ensures the order contains all N required products (not just any one).
Finding Sets Similar to a Specific Set
Find orders most similar to order 1:
Set Operations (PostgreSQL)
PostgreSQL supports array operations for set comparisons:
Performance: Indexing
For large datasets, pre-compute set signatures:
Common Pitfalls
- O(n^2) comparisons: Self-joining all pairs is expensive for large tables. Pre-filter by requiring at least one shared element (the self-join on
product_idalready does this) and add a minimum intersection threshold. - Duplicates in sets: If
order_itemscan have duplicate(order_id, product_id)rows, useCOUNT(DISTINCT product_id)instead ofCOUNT(*). - NULL handling:
GROUP_CONCATandARRAY_AGGskip NULLs. Ifproduct_idcan be NULL, filter it explicitly or the set signature will be wrong. GROUP_CONCATlength limit: MySQL'sgroup_concat_max_lendefaults to 1024 bytes. For sets with many elements, increase it:SET SESSION group_concat_max_len = 1000000.- Hash collisions: MD5 signatures for exact matching can collide (extremely rare). For critical applications, compare the full sorted array as a secondary check.
Summary
- Use self-joins on shared elements to find overlapping sets and count intersection sizes
- Use
GROUP_CONCAT/ARRAY_AGGto create set signatures for exact matching - Calculate Jaccard similarity as |intersection| / |union| for measuring set overlap
- Use
HAVING COUNT(DISTINCT) = Nto find sets containing a required subset - Index the element column (
product_id) for fast joins - Pre-compute set hashes in materialized views for large-scale exact matching
Related reading
- How to select strategy to reduce overfitting?
- How to set a single, main title above all the subplots
- how to set camera position for 3d plots using python/matplotlib?
- How to set class_weight in keras package of R?
- How to select the nth row in a SQL database table?
- How to selectively replicate private and shared portions of a CouchDB database?
- How to send a structure across multiple processes using MPI_Allreduce()?
- how to SET and DELETE using a single UpdateExpression with Dynamo

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.