Every model call the bot makes is traced: prompt, tokens, cost, the tool calls inside. The traces go to a self-hosted Langfuse, which stores them in ClickHouse, which runs in Docker on the same machine as everything else. It is the only way I know what the bot actually did, and it had been quietly working for weeks.
On 2026-09-07 the machine's disk image was 144 GB. The ClickHouse volume was 137.9 GB of that, larger than my entire source tree (76 GB). I opened it expecting a lot of traces.
What was in the volume
| table | size | what it is |
|---|---|---|
system.trace_log | 117.4 GiB | ClickHouse sampling its own query stacks, 5.3 billion rows |
system.text_log, part_log, metric_log, ... | 9.1 GiB | ClickHouse's other self-observation logs |
default.events_full + events_core | 3.2 MB | the Langfuse traces, 7,865 rows |
The observability store was almost entirely the database observing itself, and 0.002% observations. I use Langfuse; I do not tune ClickHouse; I will never open trace_log. It had been sampling stacks on every query since the container first started, and Langfuse's own queries against a 3 MB table gave it plenty to sample.
The config that did nothing for 18 days
This was not the first time I had found it. On 2026-08-20 I had written the config that turns those logs off:
I had put it inside the running container, in config.d/, and not restarted ClickHouse. Configuration in config.d is read at startup. So for eighteen days the file sat there, correct and inert, while another 117 GB accumulated. It took effect on 09-07 only because the machine rebooted for an unrelated reason.
And a file written inside a container is one docker compose up away from not existing. The version that survives is a bind mount in the compose file:
Two rules I now treat as one: a config change to a container is not done until the container has restarted with it, and it is not durable until it is in the compose file.
remove="1" is a request, not a command
After the restart, four of the eight logs stayed off: trace_log, text_log, metric_log, asynchronous_metric_log. The other four came back on their own: part_log, error_log, processors_profile_log, opentelemetry_span_log. Together they are 89 KB, so I left them. But if the one you care about is in the second group, remove alone will not save you, and you will not find out until you look at system.parts again.
query_log I kept on purpose, with a seven-day TTL. It is 17 MB and "what queries ran" is occasionally the question.
You cannot drop a table that big
Turning the log off does not delete 117 GB. DROP TABLE system.trace_log returned error 359: the table exceeds the 50 GB safety limit for drops. So does DETACH. The limit exists so that nobody deletes production data with a typo, and it applies with equal force to data nobody wants.
Three ways through:
The disk image went from 144 GB to 16 GB. Host free space went from 433 GiB to 563 GiB. Fifteen days later, on 2026-09-22, the volume looks the way it should have all along: events_full 43.0 MiB, events_core 34.5 MiB, query_log 17.7 MiB.
How to notice it before it is 117 GB
docker system df lists local volumes with a RECLAIMABLE column, and a volume attached to a running container shows as 0% reclaimable regardless of size. That column answers "what can prune free", not "what is big". Read the SIZE column, and when a volume is bigger than you expect, open it with the system.parts query above.
Frequently Asked Questions
Why not just cap the log tables with a TTL instead of removing them?
A TTL would have held trace_log to a few days of samples, which on this workload is still gigabytes per day of data I will never read. The logs are for people tuning ClickHouse. If that ever becomes me, the config is one file to edit and one restart to apply, which I now know how to do.
Is this a Langfuse problem?
No. Langfuse's data was 3.2 MB and correct. The ClickHouse image ships with self-profiling on, and any self-hosted product built on it inherits that default. The same volume growth would happen under any workload that issues queries continuously, which is what a tracing backend does.
Why did the 50 GB drop limit not stop the growth instead of the cleanup?
Because it guards deletion, not insertion. Nothing in ClickHouse's defaults limits how large a system log may grow; the safety mechanism only appears once you try to remove the result. That asymmetry is worth knowing about before the table crosses the line.
Did any traces get lost during the cleanup?
No. The Langfuse tables were never touched; only system.* tables were dropped, and those are ClickHouse's own. Langfuse kept serving traces throughout, and the two event tables are still growing at the rate the bot produces them.