jp-difficulty

A small machine learning project that classifies Japanese sentences as either simple or hard, based on the vocabulary and kanji they use.

The idea came out of my own Japanese study routine. When you're mining sentences for flashcards, most of the work is deciding whether a sentence is worth adding at your current level. This project is a first attempt at automating that judgement.

What it does

You give it a Japanese sentence. It tells you whether the sentence looks like beginner material or something aimed at a more advanced reader.

私は学生です        → simple
文化的多様性の尊重が社会の安定に寄与する  → hard

How it works

The pipeline is deliberately simple, since the point was to get an end-to-end text classifier working on Japanese rather than to chase accuracy:

1. Tokenization — Japanese doesn't put spaces between words, so the usual whitespace splitting that scikit-learn does by default is useless here. janome handles the morphological analysis and splits each sentence into words.
2. Vectorization — TfidfVectorizer turns the token lists into TF-IDF features. The custom tokenizer is passed in directly, and token_pattern is set to None so scikit-learn doesn't try to apply its own regex on top.
3. Classification — LogisticRegression fits on the training split and predicts on the held-out sentences.
4. Evaluation — classification_report prints precision, recall and F1 for both classes.
5. Persistence — the fitted model and vectorizer are saved with joblib, so serving them doesn't mean retraining on every start.

The tokenizer lives in its own module rather than in the training script. That isn't a style choice: joblib stores a reference to the function by module name, so a tokenizer defined in the training script would be unresolvable when loaded from anywhere else.

Project layout

tokenizer.py    Japanese tokenization, shared by everything else
train.py        Trains the model, prints the report, writes the artifacts
app.py          Gradio interface
server.py       FastAPI server and JSON API
static/         The hand-written web page: HTML, CSS, JavaScript

Two interfaces

The same model is served two ways, which was deliberate.

Gradio (app.py) builds the whole page from a function signature. It takes a few lines and produces a working demo, but nothing about the web layer is visible or under my control.

FastAPI (server.py) serves a page I wrote myself: HTML for structure, CSS for layout, and JavaScript that sends the input to POST /predict with fetch and writes the returned label back into the page. More work, but every part of the request cycle is explicit.

The FastAPI version also exposes the model as a plain JSON API, with interactive documentation generated at /docs.

Dataset

The dataset lives inline in train.py as two Python lists: 16 sentences labelled simple and 16 labelled hard, for 32 in total.

The simple sentences use everyday vocabulary and basic kanji — the kind of thing you meet in the first months of study (猫が好きです, 毎日日本語を勉強します). The hard sentences are compound, abstract and full of Sino-Japanese compounds, closer to newspaper or academic register (文化的多様性の尊重が社会の安定に寄与する).

I wrote all 32 sentences by hand rather than scraping a corpus, so the labels are consistent but the sample is very small.

Results

Run python train.py and the script prints a classification_report for the test split.

              precision    recall  f1-score   support

        hard       1.00      1.00      1.00         5
      simple       1.00      1.00      1.00         5

    accuracy                           1.00        10
   macro avg       1.00      1.00      1.00        10
weighted avg       1.00      1.00      1.00        10

Note on the numbers: with 32 sentences and a 70/30 split, only about 10 sentences end up in the test set. Whatever score comes out of that is dominated by which particular sentences happened to land there. The two classes also use almost entirely non-overlapping vocabulary, which is exactly the signal TF-IDF picks up most easily. In other words, a high score here mostly confirms that the pipeline is wired up correctly — it is not evidence that the model would generalize to unseen Japanese text.

Getting started

git clone https://github.com/IAmBeli/jp-difficulty.git
cd jp-difficulty
pip install -r requirements.txt
python train.py

Training writes model.joblib and vectorizer.joblib, which both interfaces load at startup. Then pick one:

python app.py            # Gradio demo on http://127.0.0.1:7860
fastapi dev server.py    # FastAPI and the hand-written page on http://127.0.0.1:8000

Requires Python 3.9 or newer.

Limitations

• Sample size. 32 hand-written sentences is a demo, not a dataset. Any evaluation on it is noisy.
• Binary labels. Real difficulty is a spectrum. A JLPT-style scale (N5 through N1) would be far more useful than a simple/hard split.
• Vocabulary memorization. Because the model only sees TF-IDF weights over tokens it met during training, an unfamiliar word carries no information at all. The classifier is closer to a lookup table over known vocabulary than to a model of difficulty.
• No answer for unknown input. Text with no familiar tokens produces an all-zero vector, so the prediction collapses to the model's intercept — the same label every time, regardless of what was typed.
• No structural features. Sentence length, kanji-to-kana ratio, grammar patterns and JLPT vocabulary level are all much stronger signals of difficulty than raw token frequency, and none of them are used yet.

Roadmap

• Replace the inline lists with a proper dataset file and grow it to a few hundred sentences
• Add hand-engineered features: kanji count, kanji-to-kana ratio, average token length, sentence length
• Look up tokens against a JLPT vocabulary list and use the level distribution as a feature
• Move from a binary label to a graded JLPT-style scale
• Return a confidence score alongside the label, and flag input the model has no basis to judge
• Add tests for the tokenizer and the prediction endpoint

Stack

Python · janome · scikit-learn · pandas · joblib · FastAPI · Gradio · HTML/CSS/JavaScript

License

MIT