Monday, August 4

Why Object-Oriented Programmers Should Understand Category Theory (Part 1) - LINQ, Lists and Optionals

Table of Content

  • Part 1 (this part)
    • Functional Ideas
    • Category theory
    • LINQ, Lists and Optionals
  • Part 2 (coming)
    • Duality and RX Extensions
    • Monads and async await
  • Part 3 (coming)
    • Abstract Data Types
  • Part 4 (coming)
    • Making Invalid States Unrepresentable

Y

ou’ve Been Using Functional Ideas All Along—Now Understand Them

Many of the most important and widely adopted programming features of the last two decades have their roots in functional programming—and deeper still, in category theory. These ideas have spread across languages often underpinning popular features without developers realizing their origins. For object-oriented programmers, understanding these ideas can radically reshape how we think about types, structure, and behavior in software.

The Value of Abstraction

Category theory gives us a mathematical language for thinking about types, how they relate, and how they can be transformed. Once you see this, you can't unsee it. Patterns across libraries and languages start to click. Abstractions become more coherent. Features you once struggled with begin to make sense—not because you've gotten smarter, but because you're seeing the bigger picture.

Often I see programmers struggle with new features, when there are older features that have similar properties and behaviour that they have no problem with. Understanding is only a slight shift in perspective away.


My First Steps: LINQ and Composability

My first brush with category theory came with C# 3.0 and the introduction of LINQ. At the time, I was refactoring a codebase and trying to make libraries more reusable. Lists and "optionals" were giving me grief. Iteration, filtering, paging, and threading all felt like they required too much boilerplate. My object-oriented toolset—design patterns, interfaces, inheritance hierarchies—wasn’t helping. 

Then came LINQ, and suddenly, things started to fall into place.

Lists, Optionals, and What They Really Are

In category theory, structures like List<T> or IEnumerable<T> are more than just objects—they’re functors and (often) monads. These terms may sound esoteric, but they describe powerful and consistent ways to work with data. 

Imperative programmers often obsess about whether to use  T[], List<T>, Dictionary<TKey, TValue> or HashSet<T> and while this is be important, it is better to consider the shape of the information at a higher level of abstraction before descending into the weeds.

In C#  optionals were a mess. Calling nullable value types and reference types "optionals" is being generous. They held the same information,  but the functionality fell far short of what you would expect from a Maybe or Optional type. 

Nullable value types (int?, DateTime?, etc.) had ugly ergonomics: HasValue and Value were clunky, and misuse was rampant. Reference types were even worse—they were always nullable, with no type-level way to opt out to give an honest object.

You could use the new extension methods on null reference types and it helped with some things, by allowing you to sneak in null checks into some of your library components and letting you filter out null object or objects with null prosperities out. but this was trying to empty the ocean with a teaspoon. My main goto to get around null references was to use the Null Object software design pattern. However trying to do this for every reference type that might possibly be null, was a non-starter.

This wasn’t just a C# problem—it was inherited from C, which inherited it from even older languages going back to ALGOL W in 1965. Tony Hoare, the inventor of the null reference, famously called it his “billion dollar mistake.”

The crux of the problem: you’re asking humans to manually track where null might sneak in. Humans are bad at that. The nullable reference type was a massive hole in the static type system. These systems are supposed to catch errors at compile time, but here was a whole class of errors—null reference exceptions—escaping to runtime.

 
Microsoft was too afraid of breaking backward compatibility to fix the problem. When C# 8.0 finally introduced non-nullable reference types in 2019 (opt-in, with compiler trickery), it was a long-overdue fix for a expensive source of errors.


LINQ: Category Theory in Disguise

Though designed for querying databases, LINQ also worked on in-memory collections. And to make LINQ composable and consistent, Microsoft had to bake in ideas straight out of functional programming and category theory—whether they admitted it or not.  

If you define two extension methods—Select and SelectMany—with the right signatures, you can make almost anything LINQ-able. That’s the power of monads.

  • Select (a.k.a. map in FP circles) is for transforming values inside a context. Think: turning a List<int> into a List<string>, or a Nullable<string> into a Nullable<bool>.

  • SelectMany (a.k.a. flatMap, bind, or >>=) flattens nested contexts: a List<List<T>> becomes a List<T>, a Nullable<Nullable<T>> becomes a Nullable<T>. This same idea powers async/await and promise chaining in JavaScript. Something similar is happening when a do try catch block intercepts an error from several layers down in the stack (This flattening behaviour is more obvious with languages with checked exceptions like Java or Swift)

These operations are so essential that in C# 6.0, Microsoft introduced the null-conditional operator (?.)—a syntactic sugar for optional objects that provides a limited form of mapping for honest properties and binding for optional properties. On the upside it works for both Nullables and reference types.
 
The elvis operator  (?.) was so popular it made it into many other languages under an assortment of names.

And of course the optional situation was such that one of the first things people tried was getting LINQ to work Nullable types or to create their own Maybe types.

 

Functional Programming's Influence Grows

Around the same time, other functional staples were entering the mainstream. Hadoop brought map, reduce, and filter into the spotlight. C# had higher-order functions from the start, but lambdas in C# 3.0 finally made them usable without drowning in boilerplate.

LINQ’s use of extension methods also meant you could tack on new functionality to types you didn’t own—a clever trick that helped functional patterns sneak into OOP-heavy ecosystems. Without a coat of paint functional concepts are often resisted as being too procedural and not O-O enough.

Even the type system itself was changing. Parameterized types (aka generics) have roots in abstract algebra and were first seen in functional languages like ML (1973), then Ada (1977), then C++ templates (1991). Category theory takes this further with higher-kinded types—types that take types that take types.

Languages like Haskell and Scala can express these abstractions natively. C# can’t, but you can still use the ideas—just in a less elegant way.

The Big Realization

The real epiphany for me came when I saw that monadic operations acted on containers—like List<T> and Nullable<T>—as whole things. They respected encapsulation. I didn’t need to peek inside and mess with internals. I could compose operations, transform values, and chain computations without writing glue code every time.

Contrast that with traditional imperative code, which often tears through internal structures and violates encapsulation to “get things done.” Functional concepts gave me a cleaner, more composable way to think.


Conclusion

You don’t need to become a Haskell expert or abandon object-oriented programming to benefit from category theory. It might feel foreign at first. But it will:

  • Change how you think about types and data
  • Help you recognize deeper patterns across languages and libraries
  • Reduce boilerplate and improve composability
  • Let you write more robust and expressive code

Gaining familiarity with these concepts can dramatically improve your ability to design clean, reusable, and maintainable code—especially as languages like C# continue to adopt features born in functional programming.

If you're serious about mastering modern software development, understanding the abstractions behind LINQ, monads, and functors is no longer optional—it's essential.


Related Posts

More Information

Videos

2007 Era

Modern Era


Books

Category Theory for Programmers by Milewski
Category Theory by Awodey

Sunday, August 3

Developing Useful Applications (Part 3) UX & DX

Table of Contents

 

U

X & DX: Why you need both

  • UX makes users actually want to use your app.
  • DX makes developers actually want to build and maintain your app.

One creates value. The other keeps your team sane and your costs under control.


What Makes a Difference

Whether you're building for users or developers, remember:

  • People make mistakes—design for the unhappy path.
  • People are trying to get things done—don’t block them.
  • The more frequently something is done, the easier and more visible it should be.
  • If information is required to make a decision, make sure it’s visible at the moment of decision, not buried somewhere else.

These principles apply equally to user interfaces and developer tooling.



UX & DX: Neither Comes Free

A great user experience (UX) and a joyful developer experience (DX) don’t appear by accident. They take deliberate design, thoughtful architecture, and—at some point—someone insisting that the quick-and-dirty way isn’t good enough.


It’s easy to throw together a working prototype without considering UX or DX. But here’s the trap: the moment stakeholders see that it "works," they’ll want to ship it.


If that prototype becomes the product, things fall apart fast. The flaws that were acceptable in a demo become frustrating in daily use. Stakeholders, forgetting their own impatience, will blame the developers. And if the team pushes forward anyway? Time they could have used to drive the project forward they are now spending ensuring they are not going backwards. They are throwing together patchwork fixes to minimise the damage of an application that is now the face of the team, an application never intended to support real users or real workloads.


The Rule of Three

(a.k.a. Why This Will Take Way Longer Than You Think)

There are two versions of the Rule of Three—and because software people love recursion, each version has two parts.

Both point to the hidden costs of building software that is reliable, usable, and maintainable.


Version 1: The Mythical Man-Month Edition

This version deals with whole applications:

  • If you’re going to use a piece of software more than once, it’ll take three times longer to build than a throwaway script.
  • If more than one person needs to use it, multiply that by another three.

That’s a 9x effort multiplier—before you've even polished the UX or refined the API.

Version 2: Facts and Fallacies of Software Engineering

This one focuses on reusable components:

  • Making a component reusable takes three times more effort than building it for a single use.
  • To prove it's truly reusable, you need to use it in three different apps. Until then, your relying on hope rather than evidence

Skipping UX and DX might save time up front—but it guarantees pain later.

Networking: It’s Fine, Until It Isn’t

If the application is using network resources. There are a host of issues around latency and connectivity. 
  • Do you want to support offline work?
  • Do you want to reconcile when you reconnect?
  • Do you cache?
  • If so how much?
The technology is well established, so its not a big deal, but it is still something that needs to be managed.

Third Parties: You’re Not in Control

  • Their schedule isn’t your schedule.
  • Their uptime isn’t your uptime.
  • Their API changes don’t ask for permission.

SLAs and contracts help—but they're not magic.

In Conclusion

Toy programs teach, they prove concepts, they delight.

Real applications? They endure. They support users, survive entropy, and evolve with time.

To bridge the gap, you need to:

  • Respect complexity
  • Align intent, concept, and implementation
  • Plan for persistence
  • Care about UX and DX
  • Expect change—and design for it

Because software that works once is a demo.
Software that keeps working? That’s the job.
 

Books

Facts and Fallacies of Software Engineering by Robert Glass, Paul Becker, John Fuller

Saturday, August 2

Developing Useful Applications (Part 2) Persistence

 Table of Contents


P

ersistence: How Information Sticks Around and What Happens if it Doesn't

Software is all about manipulating information. Numbers, text, images and sounds, its all ones and zeroes. But when that information sticks around—when it persists—things start to get tricky.

Suddenly, you have to think about all sorts of headaches:

  • Where is the data stored?
  • Who can access it—and when?
  • What happens when two people try to change it at once?
  • Should we lock it? Merge it? Roll it back?
  • Are we aiming for ACID (strong consistency) or BASE (eventual consistency)?
  • What if the data isn’t what we expected? It needs validation and correction.
  • How do we migrate it safely when the APIs or internal schema change?

And once you've solved those, you're only a third of the way there.


Code is Data Too

One of the more profound and useful realisations in computer science is that code is data. It can be stored, moved, diffed, merged, and—most importantly—it can break or be misplaced if it is not managed.

That means everything you worry about with persistent data? You get to worry about it again when dealing with the source code. Version control isn't optional; it's survival. And not just for your code files. Think bigger.

You’ll want to track:

  • Sounds and images
  • Test datasets
  • Code generators
  • Build scripts
  • Image generation scripts

   

Entropy Never Sleeps

Even if you get all this right, there are other dependencies your software relies on.

Operating systems evolve (or vanish), programming languages shift, frameworks deprecate things without warning, and that obscure build tool you relied on might be abandoned.

Left unattended, even well-built software starts to rot. Not because the logic broke—but because the world moved on without it. 


Analysing Version Control Data

There is information in your version control system that goes beyond your current build.
Are there files that get changed frequently? Those components and modules probably have too many responsibilities, or maybe they need to be more extensible.

Or maybe when you look back, the code changes, then changes back, looping back and forth. You have two sets of customers with two difference preferences, and both groups complain when they don't get their way. The problem will not go away, until both groups are satisfied.

Have you run static analysis on your code base. How much duplication is there? Are there code smells or unreachable code? 

During a retrospective I raised the results of static analysis. When they didn't want to tackle item one, I said fine, each team and their circumstances are different. When there was push back on the top ten, I gave them homework. They needed to pick one of the top ten and give suggestions on improving the code base, using the issue as jumping off point. The suggestions where due in the next retrospective in two weeks. 

If you are not improving, you are moving backward.

Guess What, Your Project has Data Too

Every project management methodology that I am aware of breaks down work, schedules that work, and tracks progress.
PMBOK has rolling wave planning and work breakdown structure (WBS), while Scrum has progressive elaboration and backlog refinement.

Whether you call them tickets, work items, stories or tasks you need to track work. Whether you use Jira, Trello or something else, that data needs to be treated with the same respect that you give your application data, or your version control data.

Analysing Work Item Data

Its worthwhile analysing your project management data. Agile has velocityPMBOK has earned value management (EVM). You need to know where you are going and where you are in your journey to get there.

While you need metrics, you need to remember that all metrics are proxies for what you really what to measure. That is because there is zero overlap between what is easy or even possible to measure and what will make a real impact over the long period.

And anytime you tie metrics to performance, your metrics become worthless. The people evaluated will game the system in the most counter-productive way possible. This is just human nature.

The data you obtain is always subject to interpretation, therefor you need to run experiments. The experiments need to be simple, quick and cheap, because that will determine how often you can run them.



Garbage In, Garbage Out

I worked with a team that was constantly re-estimating their work items, and they were not including the QA time or time for rework. They were also experiencing all the symptoms of stories that were too large. By the time stories were finished their estimates were 0 points.  This made estimating velocity impossible. 

To get any half-way decent analysis out of the tracking software, I had assume all the stories were the same size, and that the differences would come out in the wash. To complicate things my manager would close items from 3 or 4 years ago, without backdating them. These were items that had been finished years ago, but had not been marked as finished in the tracking software. You would think that could not be that many of those, but no,  somehow he seem to find 2 to 7 of them each sprint.

 
My manager insisted the team productivity had dropped, while I thought it had risen. I held my nose, rolled up sleeve and got too work on the data. The first thing I noticed, was that the issues that they with project management before I arrived, were a lot worse than I was led to believe. There was a change in the data when I arrived. Before I started, the data was disorganised and incomplete. Of course, they could have been doing a lot more than what they had recorded in the tool, and they pretty much would have had too.

After cleaning the data as best I could, it showed two dips  in productivity, two to three months apart, about two years before I started. The team had never fully recovered from what ever happen at that time, through there had been improvement since I started. Still, with my increased awareness of what had happen before I joined, I was surprised the improvement was not more dramatic.

My analysis and conclusions were only as good as the data, and the quality of the data, even after I cleaned it, was not good.

Still, I saw enough in the data, to encourage me to push harder on on changes I knew were needed.

Starting with this:
  • Stop sabotaging your workflow tracking system.
  • Stop erasing estimates.
  • Stop injecting misleading data that makes it look like you’re shipping four-year-old work every sprint.

Because if your planning data is garbage, then all your decisions based on it will be garbage too.


Note: My manager did not want to tell me what had happen two years before I started.

In Summary

Persistence isn’t just about databases. It’s everywhere: in your code, your tooling, your workflows, and your team’s memory. If you don’t treat all of that data with care, your systems will degrade—even if no one touches a line of code.


Books

PMBOK Guide
 

Friday, August 1

Developing Useful Applications (Part 1) Complexity and Intent vs Implementation

Table of Contents


Introduction

There’s a world of difference between a demo app you knock together in an afternoon and a production-grade system that delivers value over time.   

Real applications must deal with:

  • Complexity
  • Persistence
  • User Experience (UX)
  • Developer Experience (DX)
  • Network Communication
  • Third Parties

Managing Complexity 

Breaking It Down

To deal with complexity effectively:

  • Separate Intent, Concepts, and Implementation
  • Decompose functionality into modules, types, and functions.
  • Use concepts, guarantees, and encapsulation to reason about parts in isolation.
  • Keep your implementation aligned with the concepts it's supposed to represent.
  • And keep your concepts aligned with what the system is supposed to do.

Misalignment between intent, concepts, and implementation creates friction.


Intent vs Implementation

Why Separate Intent, Concepts, and Implementation?

They are differentiated by

  • Impact on complexity
  • Expressiveness
  • Impact on costs
  • Speed of change
  • Utilitisation over the development/product lifecycle.

Because

  • Intent is constrained by user needs and stakeholder politics.
  • Implementation is constrained by technology, time, and budget.
  • Intent evolves slowly. Implementation changes constantly.
  • High-level concepts reduce cognitive load and communicate goals and benefits, clarify intent and encapsulate technical detail.
  • Low-level concepts structure and organise the implementation.  

Relationship with Costs and Benefits

Real-world software has both functional (value-generating) and non-functional (cost-related) requirements.  

  • Intent describes why the system exists (the benefits) and the goals it is trying to achieve.
  • Concepts describe what the system is made of (the mental models) and give us a language to reason about Intent and Implementation
  • Implementation describes how it works (the technical details) and determines the costs in resources.   

Cost Trade Offs in Implementation

How an application is implemented is an exercise in juggling resource use such as

  • Memory
  • Performance
  • Developer Effort 
  • Usability

Reducing one cost may increase another.  


Speed of Change

Intent tends to change slowly—often painfully so. That’s because:

  • Improvements in benefits need to be sought out and chased with customer research, focus groups and feedback from beta testers.
  • Everyone has an opinion on what “value” means.
  • Aligning stakeholders takes time.
  • Introducing a new benefit usually means negotiating a new compromise.

Implementation changes quickly. That’s because:

  • Improvements in costs come knocking at the development teams door as tools improve, libraries mature, and developers gain experience.
  • Everyone loves it when “costs” go down. 
  • There is less delays because of co-ordination and communication costs as everyone is on the same page.

As time goes on, the team: 

  • Understands the domain better.
  • Learns the codebase and its quirks.
  • Gets better at using the platform and tools.
  • Sees more opportunities for improvement.

Frameworks evolve. Languages adopt better abstractions. Tools reduce boilerplate and automate the boring parts. All of these shift the implementation landscape, making things that were once hard, easy—and giving you reasons to change how things are built. 




The Source of the Concepts

A key idea in Domain-Driven Design is ubiquitous language: shared concepts and terminology built collaboratively with stakeholders.

Over time, a team builds a refined, shared vocabulary—one that captures both problem-domain ideas and the corresponding solution space. These concepts live in:

  • Conversations with subject matter experts
  • Tooling and platforms used
  • Reusable components and internal libraries
Good concepts help reduce complexity and improve communication.

 

Development and Product Lifecycle

Most Software Development Life Cycle (SDLC) models include:

  • Planning – Define goals, scope, and requirements
  • Analysis – Gather and clarify needs using domain knowledge
  • Design – Create structure and architecture
  • Implementation – Build the system
  • Testing – Validate functionality and correctness
  • Deployment – Ship to users
  • Maintenance – Ongoing updates, fixes, and improvements

Each stage benefits from continuous feedback and refinement. Ubiquitous language plays a crucial role throughout, especially during analysis, prototyping, and maintenance. Never underestimate peoples ability to mis-communicate.

At some stages Intent will be the focus in others Implementation.


A Practical Breakdown

I like to breakdown the Design and Implementation phases into
  1. Make It Work in Principle
    • Spike or proof-of-concept
    • Explore feasibility, intent, and high-level trade-offs
  2. Make It Work in Practice
    • Build a Minimum Viable Product (MVP)
    • Balance implementation detail with early feedback
  3. Make It Easy to Use (UX)
    • Reduce cost and effort for users
    • Design for mistakes:
      • Use poka-yoke (mistake-proofing)
      • Support undo/reversal
      • Provide clear, actionable warnings
      • Avoid alert fatigue
  4. Make It Easy to Maintain (DX)
    • Reduce cost and cognitive load for developers
    • Design for mistakes:
      • Use unit tests for fast feedback
      • Favor simple code paths
      • Use sensible defaults
      • Build reusable components
      • Apply restrictions and constraints intentionally

At each step: test, get feedback, iterate.



Decomposing Functionality: It’s Not Just Breaking Things Apart

Decomposing functionality is one of those deceptively simple ideas.

Reductionism is a common strategy to deal with complexity, its a cornerstone of science for example, so breaking big things into smaller things is an obvious approach. However, if components are separated at the boundaries of weak vague concepts, or the boundaries themselves are ill defined, then complexity will worsen instead of improve.


Naming: Where Concepts Meet Clarity

  • Clear, expressive names reflect clean, well-bounded concepts.
  • Good names improve readability and signal design quality.

If you're reaching for vague names like Manager, Processor, or Helper, it might be worth asking: do I really understand what this thing is doing?

To decompose effectively:

  • Boundaries should reflect real concepts, not just “this was getting long.”
  • Encapsulation should hide irrelevant details and expose clear responsibilities. 

Coherence and Coupling: Testing as a Design Tool

Test-First Design (or at least test-soon design) can reveal how well the code is decomposed. If you can tease out a component from your program well enough to test it in isolation, then it is probably nicely self-contained, and you have good encapsulation.

Other signs of good encapsulation are:

  • Interfaces hide internal details
  • External code doesn’t need to "poke around" internals

Watch out for:

  • Excessive chatter between modules
  • Leaky abstractions
  • Violations of SOLID principles


Concepts, Restrictions, Guarantees and Encapsulation

A human can only hold four pieces of information in working memory at a time. We deal with complexity by grouping things into higher-level concepts. This is called chunking, and it’s the mental compression algorithm that keeps our brains from crashing.

This works beautifully in software if your encapsulations are tight, and your components behave as advertised.

To help with that, use restrictions and guarantees that strengthen your boundaries such as:

  • Interfaces and protocols
  • Type inference and compiler checks
  • IDE validation and annotations (red squiggles)
  • Unit tests

These warn you if you are veering off in the wrong direction, and help tighten your encapsulation, and define the edges of your concepts.


When Intent, Concept, and Implementation Align

When a component’s name, idea, and behavior match up, the system feels intuitive.

When they drift apart, things fall apart:

  • Encapsulation leaks
  • Abstractions crumble
  • Complexity creeps back in

In short: if a component doesn’t do what its name or interface implies, that’s a problem. Not just for the code, but for anyone trying to understand or maintain it.


Decomposition Done Right

Good decomposition isn't about cutting things up. It's about shaping things that:

  • Represent strong concepts
  • Fit together cleanly
  • Communicate clearly

Strong concepts with meaningful names. Components with encapsulation that reveals intent, and conceals implementation.  

That’s where the real simplicity lives.


Related Posts


Further Information


Books

Domain-Driven Design: Tackling Complexity in the Heart of Software