Over the decades I’ve seen the praise and critique of the “Unix way”. I don’t remember reading about the aspect I’ll describe below. This might be a banality, and I haven’t invested much thought into it either. I’m just curious to hear your opinions.
At the core of the Unix way is that tools should do just one thing. Users very often need results that can only be achieved after doing several things. With only single-purpose tools at hand, users have to integrate them. In other words, the tool authors delegate integration to the users.
Here’s the idea I want to discuss. In programming languages, integrating various modules to achieve a goal is already a complex task because we often have to handle numerous failure modes and edge cases. In shell scripts this is much worse, since shells are more limited than programming languages. Writing a tool that does only one thing looks like avoiding the most complicated task. Which leads to a conclusion that the Unix way only works well in a rather small fraction of use cases; the majority of cases are better served by different approaches.
What do you think?
A data pipeline commonly has three phases: extract, transform, and load (ETL). The Unix philosophy of “do one thing well” can apply here by making each program handle only one ETL phase; for example, a program that only extracts data. This strategy works well whether orchestration is done with a shell script or with a Python-based de facto tool such as Apache Airflow. In contrast to mixing ETL within a single program, Airflow and shell scripts using GNU Parallel can handle retries, concurrency, logging, etc. Data pipeline engineering is not a rare case. Still, other kinds of applications, such as GUI, TUI, and web apps, are rarely built on shell scripts; however, shell scripts may work behind the scenes for many apps. Moreover, “do one thing well” still applies, but the unit may be a function, module or service instead.
In contrast to mixing ETL within a single program, Airflow and shell scripts using GNU Parallel can handle retries, concurrency, logging, etc
Am I reading your message wrong? It appears to say that a single program cannot handle retries, concurrency, logging, etc.
Just so you know, the byproduct of over integrating technology systems is how monopolies are formed. By implementing tools to only do specific functions, it is fighting against monopolistic power. That is what the “Unix way” is describing.
In essence people are predictable. If people have a google home in their house, they will most likely be using google for email, searching, and messages, and its all tightly over integrated (IE: the home has access to the email, to the messages, and it integrates it into a single device). For me, I dropped google a few years ago. For other people, they may depend on google services more often, so it isn’t just painstakingly forwarding your new email address to applications, they’ll also loose the integration as well that comes with google.
Generally speaking, to combat technology monopoly’s you want to diversify your applications. You basically want to make a system so complicated no one single software has control over it aka a decentralized system. Octave is a decentralized system, in that packages are developed by members in the octave community. Octave integrates those packages in a server but octave doesn’t have control over those packages. Linux, delta chat, lemmy, mastodon, these are all uniquely decentralized systems with all of them inadvertently following the “Unix way.”
Before getting to overly integrated technology we first must take a step from the completely disjointed technology. I’m arguing for the first steps, not for going all the way to the other extreme.
I think you’re missing the difference between professional software development and a QnD script thrown together to accomplish a task.
The UNIX philosophy is also related to The Agile Manifesto and “worse is better”.
People are supposed to have the power to solve their problems. Unix provides primitives and tools to people, and they are expected to learn enough to use the system. It’s from a very different time. Well before the “don’t make me think” crowd showed up.
In programming languages, integrating various modules to achieve a goal is already a complex task because we often have to handle numerous failure modes and edge cases.
Now ignore all of the failure modes and edge cases except those that a script is going to directly run into, and that should simplify the requirements. This is “worse is better”.
Perl, Python, and Ruby are supposed to be approachable and easy to get started with. They were never supposed to be industrial programming languages like C, C++, Ada, Pascal, Java, etc., and what you’re noticing is the software industry not understanding there needs to be levels which address the different needs of people. Sometimes people need a simple tool, and not everyone needs to master the professional tool which takes two hours to set up and adjust.
The software industry destroying easy, simple to use software is another discussion.


