Is there any good open-source or freely available Chinese segmentation algorithm available?

Chinese segmentation

open-source software

text processing

natural language processing

language tools

Is there any good open-source or freely available Chinese segmentation algorithm available?

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Natural Language Processing Computer Vision

Introduction

Chinese text does not use spaces between words, making word segmentation a fundamental NLP task for Chinese language processing. Several excellent open-source libraries are available: jieba is the most popular Python library for its simplicity, pkuseg offers higher accuracy from Peking University, THULAC comes from Tsinghua University, and spaCy supports Chinese through trained pipelines. For deep learning approaches, HanLP and LAC (Baidu) provide state-of-the-art segmentation. This article compares the major tools with installation instructions, code examples, and accuracy/speed tradeoffs.

jieba (Most Popular)

jieba is the most widely used Chinese segmentation library, supporting three modes:

python

1# Installation
2# pip install jieba
3
4import jieba
5
6text = "我来到北京清华大学"
7
8# Precise mode (default) — most accurate
9words = jieba.lcut(text)
10print(words)  # ['我', '来到', '北京', '清华大学']
11
12# Full mode — finds all possible words
13words_full = jieba.lcut(text, cut_all=True)
14print(words_full)  # ['我', '来到', '北京', '清华', '清华大学', '华大', '大学']
15
16# Search engine mode — finer granularity for search indexing
17words_search = jieba.lcut_for_search(text)
18print(words_search)  # ['我', '来到', '北京', '清华', '华大', '大学', '清华大学']

Custom Dictionary

python

1import jieba
2
3# Add custom words
4jieba.add_word("深度学习")
5jieba.add_word("自然语言处理")
6
7text = "深度学习在自然语言处理中应用广泛"
8print(jieba.lcut(text))
9# ['深度学习', '在', '自然语言处理', '中', '应用', '广泛']
10
11# Load custom dictionary file
12# Format: word frequency part-of-speech (one per line)
13# 深度学习 5 n
14jieba.load_userdict("custom_dict.txt")

POS Tagging

python

1import jieba.posseg as pseg
2
3text = "我在北京大学学习计算机科学"
4words = pseg.lcut(text)
5for word, flag in words:
6    print(f"{word}/{flag}", end=" ")
7# 我/r 在/p 北京大学/nt 学习/v 计算机/n 科学/n

pkuseg (Higher Accuracy)

Developed by Peking University, pkuseg achieves higher accuracy than jieba through domain-specific models:

python

1# Installation
2# pip install pkuseg
3
4import pkuseg
5
6# Default model
7seg = pkuseg.pkuseg()
8text = "我来到北京清华大学"
9print(seg.cut(text))  # ['我', '来到', '北京', '清华大学']
10
11# Domain-specific models
12seg_news = pkuseg.pkuseg(model_name="news")     # News articles
13seg_web = pkuseg.pkuseg(model_name="web")        # Web text
14seg_med = pkuseg.pkuseg(model_name="medicine")   # Medical text
15
16# POS tagging
17seg_pos = pkuseg.pkuseg(postag=True)
18print(seg_pos.cut(text))
19# [('我', 'r'), ('来到', 'v'), ('北京', 'ns'), ('清华大学', 'nt')]

THULAC (Tsinghua University)

python

1# Installation
2# pip install thulac
3
4import thulac
5
6thu = thulac.thulac()
7text = "我来到北京清华大学"
8result = thu.cut(text, text=True)
9print(result)
10# 我_r 来到_v 北京_ns 清华大学_ni
11
12# Segmentation only (no POS tags)
13thu_seg = thulac.thulac(seg_only=True)
14print(thu_seg.cut(text, text=True))
15# 我 来到 北京 清华大学

HanLP (Deep Learning)

HanLP provides state-of-the-art NLP capabilities including segmentation:

python

1# Installation
2# pip install hanlp
3
4import hanlp
5
6# Load tokenizer
7tokenizer = hanlp.load(hanlp.pretrained.tok.COARSE_ELECTRA_SMALL_ZH)
8
9text = "我来到北京清华大学"
10print(tokenizer(text))
11# ['我', '来到', '北京', '清华大学']
12
13# Fine-grained tokenization
14fine_tokenizer = hanlp.load(hanlp.pretrained.tok.FINE_ELECTRA_SMALL_ZH)
15print(fine_tokenizer(text))
16# ['我', '来到', '北京', '清华', '大学']
17
18# Full NLP pipeline (tokenize + POS + NER + dependencies)
19HanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH)
20result = HanLP(text)
21print(result["tok/coarse"])
22print(result["pos/ctb"])

spaCy (Multi-Language Framework)

python

1# Installation
2# pip install spacy
3# python -m spacy download zh_core_web_sm
4
5import spacy
6
7nlp = spacy.load("zh_core_web_sm")
8doc = nlp("我来到北京清华大学学习自然语言处理")
9
10for token in doc:
11    print(f"{token.text}\t{token.pos_}\t{token.dep_}")
12
13# Segmented words
14words = [token.text for token in doc]
15print(words)

LAC (Baidu)

python

1# Installation
2# pip install lac
3
4from LAC import LAC
5
6# Segmentation
7lac = LAC(mode="seg")
8text = "我来到北京清华大学"
9print(lac.run(text))
10# ['我', '来到', '北京', '清华大学']
11
12# Segmentation + POS tagging
13lac_pos = LAC(mode="lac")
14print(lac_pos.run(text))
15# [['我', '来到', '北京', '清华大学'], ['r', 'v', 'LOC', 'ORG']]

Comparison

Library	Accuracy	Speed	Deep Learning	POS Tagging	Custom Dict
jieba	Good	Fast	No	Yes	Yes
pkuseg	Very Good	Moderate	No	Yes	Limited
THULAC	Very Good	Moderate	No	Yes	Yes
HanLP	Excellent	Slower	Yes	Yes	Yes
spaCy	Good	Fast	Yes	Yes	Yes
LAC (Baidu)	Excellent	Moderate	Yes	Yes	Yes

Common Pitfalls

Not loading a custom dictionary for domain-specific text: Out-of-the-box models struggle with specialized vocabulary (medical terms, brand names, slang). Always load a custom dictionary with jieba.load_userdict() or equivalent for your domain.
Confusing jieba's three modes: Precise mode is for general text, full mode finds all possible words (useful for indexing), and search mode provides finer granularity. Using full mode for regular text processing produces excessive, overlapping segments.
Not initializing jieba before timed benchmarks: jieba loads its dictionary on first use, which takes 1-2 seconds. Call jieba.initialize() before benchmarking to exclude startup time from measurements.
Assuming all segmenters handle Traditional Chinese: Most tools default to Simplified Chinese. For Traditional Chinese text, convert to Simplified first with opencc or use a model trained on Traditional Chinese (HanLP supports both).
Ignoring encoding issues: Chinese text files may use UTF-8, GBK, or GB2312 encoding. Always specify encoding when reading files: open("text.txt", encoding="utf-8"). Encoding errors produce garbage segmentation results.

Summary

Use jieba for quick setup and good-enough accuracy — it is the most popular and easiest to use
Use pkuseg for higher accuracy on specific domains (news, web, medicine)
Use HanLP or LAC for state-of-the-art deep learning-based segmentation
Use spaCy if you need Chinese segmentation integrated with a multi-language NLP pipeline
Always add domain-specific words via custom dictionaries for best results in specialized applications

Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course

Basis Lab

No course covers this one yet

Basis Lab writes one for you from a sentence about what you want to be able to do, then teaches it and checks you understood. Your first course is free.

Build my course

Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Is there any good open-source or freely available Chinese segmentation algorithm available?

ML System Design practice on Codemia

Introduction

jieba (Most Popular)

Custom Dictionary

POS Tagging

pkuseg (Higher Accuracy)

THULAC (Tsinghua University)

HanLP (Deep Learning)

spaCy (Multi-Language Framework)

LAC (Baidu)

Comparison

Common Pitfalls

Summary

Related reading

Tackling System Design Interview Problems

No course covers this one yet

Track what you have practised

ML System Design practice on Codemia