Radical Simplicity in Knowledge Engineering | by Cai Parry-Jones | Jul, 2024

Not too long ago, I’ve had the fortune of talking to quite a few information engineers and information architects in regards to the issues they face with information of their companies. The principle ache factors I heard time and time once more have been:

• Not figuring out why one thing broke
• Getting burnt with excessive cloud compute prices
• Taking too lengthy to construct information options/full information tasks
• Needing experience on many instruments and applied sciences

These issues aren’t new. I’ve skilled them, you’ve in all probability skilled them. But, we will’t appear to discover a resolution that solves all of those points in the long term. You would possibly suppose to your self, ‘effectively level one may be solved with {insert information observability software}’, or ‘level two simply wants a stricter information governance plan in place’. The issue with these fashion of options is that they add further layers of complexity, which trigger the ultimate two ache factors to extend in seriousness. The mixture sum of ache stays the identical, only a totally different distribution between the 4 factors.

This text goals to current a opposite fashion of drawback fixing: radical simplicity.

TL;DR

• Software program engineers have discovered large success in embracing simplicity.
• Over-engineering and pursuing perfection can lead to bloated, slow-to-develop information programs, with sky excessive prices to the enterprise.
• Knowledge groups ought to contemplate sacrificing some performance for the sake of simplicity and pace.

A Lesson From These Software program Guys

In 1989, the pc scientist Richard P. Gabriel wrote a comparatively well-known essay on pc programs paradoxically known as ‘Worse Is Higher’. I gained’t go into the small print, you possibly can learn the essay right here in case you like, however the underlying message was that software program high quality doesn’t essentially enhance as performance will increase. In different phrases, on events, you possibly can sacrifice completeness for simplicity and find yourself with an inherently ‘higher’ product due to it.

This was an odd concept to the pioneers of computing throughout the 1950/60s. The philosophy of the day was: a pc system must be pure, and it might probably solely be pure if it accounts for all doable eventualities. This was probably resulting from the truth that most main pc scientists on the time have been lecturers, who very a lot wished to deal with pc science as a tough science.

Lecturers at MIT, the main establishment in computing on the time, began engaged on the working system for the subsequent era of computer systems, known as Multics. After almost a decade of improvement and hundreds of thousands of {dollars} of funding, the MIT guys launched their new system. It was unquestionably essentially the most superior working system of the time, nonetheless it was a ache to put in because of the computing necessities, and have updates have been gradual because of the dimension of the code base. Consequently, it by no means caught on past just a few choose universities and industries.

Whereas Multics was being constructed, a small group supporting Multics’s improvement grew to become pissed off with the rising necessities required for the system. They ultimately determined to interrupt away from the challenge. Armed with this expertise they set their sights on creating their very own working system, one with a elementary philosophy shift:

The design should be easy, each in implementation and interface. It’s extra essential for the implementation to be easy than the interface. Simplicity is an important consideration in a design.

— Richard P. Gabriel

5 years after Multics’s launch, the breakaway group launched their working system, Unix. Slowly however steadily it caught traction, and by the Nineties Unix grew to become the go-to alternative for computer systems, with over 90% of the world’s high 500 quickest supercomputers utilizing it. To at the present time, Unix remains to be broadly used, most notably because the system underlying macOS.

There have been clearly different components past its simplicity that led to Unix’s success. However its light-weight design was, and nonetheless is, a extremely priceless asset of the system. That would solely come about as a result of the designers have been prepared to sacrifice performance. The info trade shouldn’t be afraid to to suppose the identical approach.

Again to Knowledge within the twenty first Century

Considering again at my very own experiences, the philosophy of most huge information engineering tasks I’ve labored on was much like that of Multics. For instance, there was a challenge the place we would have liked to automate standardising the uncooked information coming in from all our shoppers. The choice was made to do that within the information warehouse through dbt, since we might then have a full view of information lineage from the very uncooked recordsdata proper by way of to the standardised single desk model and past. The issue was that the primary stage of transformation was very handbook, it required loading every particular person uncooked consumer file into the warehouse, then dbt creates a mannequin for cleansing every consumer’s file. This led to 100s of dbt fashions needing to be generated, all utilizing primarily the identical logic. Dbt grew to become so bloated it took minutes for the info lineage chart to load within the dbt docs web site, and our GitHub Actions for CI (steady integration) took over an hour to finish for every pull request.

This might have been resolved pretty merely if management had allowed us to make the primary layer of transformations exterior of the info warehouse, utilizing AWS Lambda and Python. However no, that might have meant the info lineage produced by dbt wouldn’t be 100% full. That was it. That was the entire cause to not massively simplify the challenge. Much like the group who broke away from the Multics challenge, I left this challenge mid-build, it was just too irritating to work on one thing that so clearly might have been a lot less complicated. As I write this, I found they’re nonetheless engaged on the challenge.

So, What the Heck is Radical Simplicity?

Radical simplicity in information engineering isn’t a framework or data-stack toolkit, it’s merely a way of thinking. A philosophy that prioritises easy, simple options over complicated, all-encompassing programs.

Key ideas of this philosophy embody:

1. Minimalism: Specializing in core functionalities that ship essentially the most worth, somewhat than attempting to accommodate each doable situation or requirement.
2. Accepting trade-offs: Willingly sacrificing a point of completeness or perfection in favour of simplicity, pace, and ease of upkeep.
3. Pragmatism over idealism: Prioritising sensible, workable options that clear up actual enterprise issues effectively, somewhat than pursuing theoretically good however overly complicated programs.
4. Decreased cognitive load: Designing programs and processes which might be simpler to grasp, implement, and preserve, thus lowering the experience required throughout a number of instruments and applied sciences.
5. Price-effectiveness: Embracing less complicated options that always require much less computational sources and human capital, resulting in decrease general prices.
6. Agility and flexibility: Creating programs which might be simpler to switch and evolve as enterprise wants change, somewhat than inflexible, over-engineered options.
7. Deal with outcomes: Emphasising the tip outcomes and enterprise worth somewhat than getting caught up within the intricacies of the info processes themselves.

This mindset may be in direct contradiction to trendy information engineering options of including extra instruments, processes, and layers. Consequently, be anticipated to struggle your nook. Earlier than suggesting an alternate, less complicated, resolution, come ready with a deep understanding of the issue at hand. I’m reminded of the quote:

It takes a number of onerous work to make one thing easy, to actually perceive the underlying challenges and provide you with elegant options. […] It’s not simply minimalism or the absence of litter. It entails digging by way of the depth of complexity. To be really easy, you must go actually deep. […] It’s important to deeply perceive the essence of a product so as to have the ability to do away with the components that aren’t important.

— Steve Jobs

Facet word: Bear in mind that adopting radical simplicity doesn’t imply ignoring new instruments and superior applied sciences. In truth one among my favorite options for a knowledge warehouse in the mean time is utilizing a brand new open-source database known as duckDB. Test it out, it’s fairly cool.

Conclusion

The teachings from software program engineering historical past supply priceless insights for at present’s information panorama. By embracing radical simplicity, information groups can deal with lots of the ache factors plaguing trendy information options.

Don’t be afraid to champion radical simplicity in your information crew. Be the catalyst for change in case you see alternatives to streamline and simplify. The trail to simplicity isn’t straightforward, however the potential rewards may be substantial.