Two queries for the same place. One works, one doesn't.
The reason is not subtle once you look. The string "Ueno station" does not appear anywhere in the corpus. Product titles say Ueno. Region fields say Ueno. Nothing says Ueno station, because a hotel is not a station.
In Korean this is worse than it sounds, because the suffix attaches directly with no space. 우에노역 is one token to a naive matcher, and it matches nothing.
Queries ending in a facility suffix (station, beach, airport, terminal, market, township) are 4.3% of typed query kinds and 5.5% of search volume. I retried each one with the suffix removed.
| zero results | mean results | place name present at rank 1 | |
|---|---|---|---|
| suffix left on | 75 kinds, 29.6% | 3.75 | 52 kinds, 49.2% |
| suffix stripped | 21 kinds, 5.4% | 5.54 | 81 kinds, 67.4% |
Nearly a third of this class returned nothing. In Korean the suffix is one or three characters; removing it recovered most of them.
Measuring one suffix at a time
The obvious implementation is a list of suffixes and one A/B run. I did it per suffix instead, and that decision is most of what this post is about.
| suffix | improved | regressed |
|---|---|---|
| station | 23.2% | 0.0% |
| beach (long form) | 4.2% | 0.3% |
| beach (short form) | 2.7% | 0.0% |
| township | 1.4% | 0.0% |
| airport | 1.4% | 0.2% |
| market | 0.4% | 0.0% |
| terminal | 0.1% | 0.0% |
| district | 0.8% | 0.6% |
| park | 1.0% | 1.0% |
Percentages are of search volume within this query class.
The last two rows are why per-suffix mattered. Their net effects are +0.1% and −0.0%, which vanishes inside a +32% total. Individually they are doing real damage.
District. Korea has a 면 suffix for rural townships. Strip it from a query naming a township in one province and you get a bare name that is also a city in a different province, three hundred kilometres away. The query moved from a correct local guesthouse to a hotel in the wrong province.
Park. 어린이대공원 is a proper noun, a specific park in Seoul. Strip 공원 and you get 어린이대, which is not a word in any language. The retry returns whatever the vector index makes of a nonsense string.
Neither of those is visible in a combined number. If I had run the whole list at once and seen +32%, I would have shipped both.
The morphological analyser does not do this for you
Korean has good morphological analysers, and my first assumption was that one would segment these for free. It does, inconsistently:
Inferred: the station names that survive as single tokens are ones the analyser has in its dictionary as proper nouns, and the ones it splits are not. I did not read its dictionary to confirm this. I am reading the pattern off the output.
Either way it settles the design question. Morphological boundaries alone cannot drive this rule, because the analyser treats the most common station names as atomic. A suffix list is required.
A retry that is only allowed to widen
Query relaxation can make things worse by returning a different, smaller set. So the retry only replaces the original when it does not shrink it.
| replacement rule | fired | improved | regressed |
|---|---|---|---|
| always replace | 110 kinds | 33.4% | 0.4% |
| replace only if not fewer results | 107 kinds | 33.4% | 0.0% |
One condition, three fewer firings, and the regressions go to zero without costing any of the gain. Relaxation is an operation whose whole purpose is to widen; if it narrowed, it failed, and the cheapest test for that is counting the results.
When to fire it
The retry costs a second search, so it should not run on queries that are already fine. I swept the trigger: retry only when the original returns N results or fewer.
| trigger | fired | zero results fixed | rank-1 corrected | lost | net |
|---|---|---|---|---|---|
| 0 | 72 | 23.8% | 0.0% | 0.0% | 23.8% |
| 3 | 94 | 23.8% | 1.4% | 0.0% | 25.2% |
| 7 | 110 | 23.8% | 8.5% | 0.0% | 32.2% |
| 10 | 168 | 23.8% | 9.0% | 0.5% | 32.3% |
The net column is computed before rounding, so subtracting the displayed figures disagrees with it by a tenth in the row for 7.
Firing only on zero results is the safe reading and it is what I expected to land on. It costs 8.5 percentage points of rank-1 corrections: queries like "Ueno station" that returned five wrong things rather than nothing, which a zero-only trigger never touches.
Ten is effectively unconditional, since the pipeline returns at most ten. It buys 0.1 more points of gain and introduces the only losses in the table.
Seven is where the curve flattens. Everything from 0 to 7 has exactly zero measured loss, so the choice inside that range is free, and 7 takes the most.
What I would check first in someone else's version of this
The suffix list is a linguistic resource, and lists like it tend to grow by intuition. The rule I ended up with is that a suffix earns its place by its own measurement, not by belonging to a category that mostly works.
Station, beach, airport, terminal, market, township all describe places. So do district and park. Seven of the nine I measured are worth stripping, two are not, and the difference is not in the linguistics. It is in whether the remaining stem happens to collide with another place or stop being a word.
There is no principle I could have reasoned my way to. I had to run nine measurements and throw two away.