Ceylon Development Log — September 17, 2026: When a Crash Needs to Leave Evidence
A real fault exposed an important distinction today: Ceylon can detect and contain a crashed component, but that is not enough. If the machine is going to recover from failures, it also needs to remember exactly what happened.
Project Ceylon Nightly: When a Crash Needs to Leave Evidence
Today started with something that, on the surface, looked encouraging.
One of Ceylon's isolated execution components hit a real illegal-instruction fault.
The system detected it.
The fault stayed contained.
The rest of the machine did not fall over with it.
That is exactly the kind of behavior a componentized operating system is supposed to provide.
But the fault also exposed a hole.
Ceylon knew what had happened well enough to print useful information to the debug serial console, yet the same information never entered the structured logging path we have been building for the operating system.
In other words, the architecture worked well enough to catch the failure, but the evidence stopped at the equivalent of somebody yelling across the workshop.
That is not good enough for the system Ceylon is intended to become.
The difference between a debug message and an incident record
Serial output is incredibly useful while an operating system is under construction.
It is simple.
It is immediate.
When something goes wrong at a very low level, there is a good chance the serial console is still alive.
I do not want to get rid of it.
But serial output is not a durable incident history.
If a component faults, gets reclaimed, and is restarted, I eventually want to be able to answer questions such as:
What fault happened?
Which exact component instance failed?
Which build was running?
What instruction or address was involved?
What did the supervisor decide to do?
Was the component reclaimed?
Was it restarted?
Did the restart succeed?
Did any diagnostic information get dropped along the way?
A line of text on a debug console cannot answer those questions reliably after the fact.
A structured incident record can.
The good news: the logging architecture is not broken
The useful discovery today is that we do not need to throw away Ceylon's logging system and start over.
The existing design already has most of the right ingredients.
Ceylon has bounded shared-memory error transport.
It has explicit drop accounting.
It has a collector boundary separate from the producer.
It has a path for structured records to reach durable storage.
That part is important because I do not want a faulting or supervising component trying to perform filesystem I/O while it is dealing with a failure.
The intended path remains:
component or supervisor
→ bounded shared-memory record
→ collector
→ durable local log
→ management/query tools
The problem is much narrower.
The execution-manager fault path never enters it.
Right now that path effectively looks like:
fault
→ supervisor
→ debug serial
What it needs to become is:
fault
→ supervisor
→ structured fault event
→ bounded error mailbox
→ collector
→ durable incident record
Serial can remain as a secondary human-readable path.
It simply cannot be the only record.
One fault should become one coherent story
Once I started looking at this, it became obvious that simply logging "component crashed" is not enough either.
Ceylon is being designed around restartable, generation-safe resources.
That means one execution slot may host one component instance, be reclaimed, and later be reused for another instance.
So a useful fault report needs to identify more than a slot number.
It needs to identify the exact generation of the thing that failed.
It also needs a correlation identifier so we can connect the complete lifecycle:
FAULT OBSERVED
↓
CONTAINMENT
↓
RECLAIM
↓
RESTART ATTEMPT
↓
NEW GENERATION
↓
RECOVERY SUCCEEDED / FAILED / QUARANTINED
That turns a collection of unrelated log lines into an incident.
If Ceylon is going to recover automatically from failures someday, that distinction is critical.
The most dangerous kind of "reliable" system would be one that silently restarts things while losing the evidence explaining why they were restarted.
No silent telemetry loss
There is another rule that came out of today's work.
If the fault-reporting path itself is full or unavailable, that failure has to be visible too.
The system should never be allowed to claim that everything was recorded merely because nobody noticed the log buffer overflowing.
So the design now treats loss as part of the telemetry model.
If the error mailbox fills, count it.
If the collector is unavailable, expose it.
If storage cannot accept records, make that visible.
If a record is malformed, record the rejection.
The goal is not to pretend diagnostic infrastructure can never fail.
The goal is to make sure failure of the diagnostic infrastructure does not disappear silently.
That is a very Ceylon kind of rule: bounded resources are fine. Unexplained behavior is not.
Research before freezing the next interfaces
A significant part of today's work was research rather than implementation.
That is becoming normal as Ceylon grows.
I spent time comparing how mature systems separate fault capture from durable logging, how they preserve component identity across restart, and how diagnostic data can remain useful without turning the fault path into another source of failure.
The same research habit is also being applied to graphics, application integration, compatibility, and device architecture.
On the graphics side, one important working assumption is deliberately pessimistic: we should design Ceylon so it remains useful even if hardware vendors never write native drivers for us.
That pushes us toward a native Ceylon presentation architecture that can learn from open drivers, Mesa, firmware interfaces, virtual devices, and other operating systems without making Ceylon dependent on their architecture.
The same pattern applies elsewhere.
Study what already exists.
Understand why it works.
Keep the durable lessons.
Reject assumptions that do not fit Ceylon.
I would rather spend a day understanding why a mature design works than spend a month rediscovering the same failure mode in our own code.
The machine should be able to explain itself
This has become one of the themes I like most about Ceylon.
I do not just want the system to attempt recovery.
I want it to be able to explain what it did.
That applies to memory ownership.
It applies to device authority.
It applies to application execution.
And now it applies to faults.
The eventual management tools should be able to reconstruct something like:
At 14:22:31.482, application worker generation 17 faulted with an illegal instruction.
The supervisor contained the fault.
The worker was reclaimed.
No fault telemetry was dropped.
A new generation was created.
The application resumed.
That is much more useful than:
"Something crashed earlier, but it seems fine now."
This is also part of making Ceylon supportable
The public-facing side of the project is becoming more important as A1 gets closer.
An operating system that only its author can diagnose is not much of a platform.
Eventually somebody should be able to send a bounded support record or diagnostic bundle that identifies:
the Ceylon build;
the boot/session;
the component;
the fault;
the recovery sequence;
and whether the telemetry itself remained healthy.
That does not require a mandatory cloud service.
The architecture remains local-first.
But local-first does not mean opaque.
It means the machine should keep useful evidence under the user's control.
Where this leaves the work
Today's conclusion was reassuring.
The structured logging architecture is viable.
The fault-containment architecture is viable.
The missing piece is that one important supervisor path bypasses the structured system.
So the next work is targeted rather than revolutionary:
publish supervisor-observed faults into the existing bounded mailbox;
preserve exact identity and generation;
correlate the recovery sequence;
prove that healthy qualification runs lose no fault records;
and make the degraded cases visible when the logging path itself is under stress.
The larger lesson from the research side is equally important.
Ceylon is reaching the stage where every major subsystem should be designed with enough historical and modern context that we understand the tradeoffs before we freeze the interface.
That may look slower on a daily changelog.
It should make the project much faster over the long run.
— Mike