I especially enjoyed where you discovered that K8s runs on string interpolation. That feeling never goes away, FYI; every time you have to play with nindent, you lose a little piece of your sanity.
Gravityloss 2 days ago [-]
This is partially a tangent, but the whole managing of software infrastructure is a strange culture. There would of course be lots of better ways, but it's too bad the infrastructure side of the software industry lives partially in a parallel world and many of the lessons from elsewhere in the industry are not really widespread there (at least yet). Even banks seem intelligent and innovative compared to infrastructure management mainstream practices.
It's very error prone to describe a hierarchy of versioned software packages and configuration with shell scripts and yaml. Or use text files in git as a database. Or use the CI system as a bad substitute for a proper service (in order to do a simple thing, you must first create the world from scratch). And so on. Mainstream industry knows types, programming languages, relational databases, graphs and you name it, but those have not yet penetrated sufficiently into the infrastructure world. The understanding of the value of having types has been paid in sweat and blood when developing software and databases. Even a basic CRUD application with a basic SQL database will run rings around these Frankenstein monsters: in safety, speed, and extendability.
ranger207 1 days ago [-]
Yeah I think the difference ultimately is the API surface of infra versus app software. If you're developing software, you get to contain and abstract everything into your own vision of how things should be done with types etc. But infra is managing a whole bunch of other people's software, each with their own vision of how things should be done, which means that often things just fall back to the least common denominator, plain ASCII
solatic 1 days ago [-]
> basic CRUD application with a basic SQL database
The whole problem on the infrastructure side is that the APIs are slow and you don't have exclusive access. Building the CRUD app would be a million times harder if SQL calls took 100x longer to execute and other people could randomly reach into the database and change the schema and data at will - can your basic CRUD app gracefully handle somebody shoving newlines at the end of usernames? Odds are that the first time, no, not until you put a trim() call. And then you cry in pain at needing to put trim() calls everywhere. And then you gnash your teeth at people reporting performance problems because of all the defensive checks you put in because all trust has been thrown out. And then you find out that trim() was deprecated and will now be replaced by strim() that handles Unicode a little differently, and you need to port your code before End of Life to stay compliant.
A basic CRUD application with a basic SQL database doesn't solve the problem, because you don't seem to appreciate that there already is a database - however your vendor stores configuration - and it's your job to deal with it. Adding another database on top of it would make the problem worse.
Gravityloss 10 hours ago [-]
Let's have a toy example. You want to store some information about some software configuration and let users view it. Some fairly basic data.
If you are infrastructure software minded, you might store the data in files that are stored in git. For users to read the data, they can perhaps browse the repo or clone it. For some more complex questions you might have a document which tells the user how to fire a CI pipeline that runs some code over the data and the user has to read the pipeline logs to get the information. The CI pipeline will probably run in a container which will produce a large log and take a long time to start.
If you are software software minded, you store the values in a database, make a service with a REST or some other API and create a browser UI. User can browse the data in a table view and perform searches and perhaps there's some more complex business logic executed on the backend to get answers to some more complex questions.
In this case, both systems can easily do the job. But the solutions are very different.
outworlder 2 days ago [-]
> It's very error prone to describe a hierarchy of versioned software packages and configuration with shell scripts and yaml.
We have a multitude of configuration/markup languages all doing the exact same thing.
We have solved this problem in the 60s: S-Expressions. Should have just used that for everything. Configuration, code, websites, you name it. Infinitely extendable, easy to parse.
ashkankiani 1 days ago [-]
I personally have dealt a lot with S-expressions, and at this point, given the amount of tooling, I think JSON is just almost always the right choice. It parses fast with SIMDJSON or doing various other approaches to parse sparsely. It's a little under-specified in terms of stuff like number precision, but you can honestly write your own JSON parser which does >600MB/s with an LLM in one shot and do whatever custom validation you want these days. And you get unambiguous maps without having to glean from adjacency list structure like in sexp.
And if you really love sexps, then writing a bunch of arrays in JSON is also viable. If you asked me 10 years ago I might've complained about JSON, but now I think we messed up by having a bunch of other formats, tbh. I have my own TUI JSON editor that can render in tree view, miller tree view, flat view, and inferred table view, and it makes editing JSON very easy too.
Oh and duckdb can query against compressed json and jsonl at GB/s
gonzalohm 8 hours ago [-]
How would you create a configuration file with S-expressions? Wouldn't it require defining operators and a parser?
devin 1 days ago [-]
I'm a lisp nerd, but why do you say sexps are the answer here? I think the point of a lot of configuration formats is that they don't want their configuration to also be a full-fledged programming language.
majormajor 1 days ago [-]
> I think the point of a lot of configuration formats is that they don't want their configuration to also be a full-fledged programming language.
That seems to be everyone's initial goal, but then it inevitably gets leaky.
I think the dominant split here that forces everything into a weird mismash is that nobody really wants to have to understand arbitrary programming language configs for third-party stuff, they want to read config and know what's going on. BUT for their own stuff, they find reasons to want more power and expressibility...
cassianoleal 1 days ago [-]
> That seems to be everyone's initial goal, but then it inevitably gets leaky.
This is very much true. And until they do get leaky, you have to wrap them in all sorts of weird templates or other forms of abstraction.
This is one of the reasons I used to really like Chef's config language. It was a Ruby DSL, which meant that anything the DSL couldn't do natively, you could just use Ruby right there.
Of course, this is easily misused, but if you were careful it was as powerful as it was elegant.
jimbokun 1 days ago [-]
Sexps, on their own, are not a programming language.
devin 16 hours ago [-]
I’m aware.
gustavus 1 days ago [-]
> they don't want their configuration to also be a full-fledged programming language.
They say they don't want it but every configurmation format eventually ends up becoming turing complete at some point.
ianburrell 1 days ago [-]
There has been some config languages, like Cue, that are not Turing complete. It is useful of have them always complete and have deterministic evaluation.
Cue also has a lot of schema validation which is better than dynamic languages.
jimbokun 1 days ago [-]
Going in a completely different direction, Haskell could make a nice, powerful configuration language.
If it compiles, it’s almost certainly correct.
ashkankiani 1 days ago [-]
This idea exists (as in a functional language which is used for configuration), and it's called Dhall (https://dhall-lang.org/). But they made it a total language so that it has stronger guarantees of terminating and lots of other config specific stuff.
erulabs 1 days ago [-]
It no longer applies at all, and in retrospect might look silly, but a core principle of "infra as yaml" and "infra as piles of plain-text files in git" was that infra was supposed to be dumb.
A reliability engineer's nightmare, and a common one, is stepping into a company whose deployment pipeline is built in-house, with dynamic bespoke software. What happens when Jeff leaves to build his own startup and absolutely no one knows how that Node.js script shells out to SSH to call docker on a remote server? I can tell you what happens: everything works fine for awhile and then suddenly doesn't for awhile.
On the other hand, you join a company with K8s and plain-jane YAML and can instantly grok the entire system and fix prod on your first day while the S-team sweats behind you, watching every key stroke. The seconds I have to read some javascript or bash or perl are precious seconds the brand is suffering.
In software engineering, flexibility and abstraction is a benefit: the product must pivot and evolve. In reliability engineering we want dead simple, tried and true. You want to be able to step outside your SF office and say "Someone help me with EKS, my DevOps guy died last night in a car crash!" and have 4 people with wild hair take their weird off-brand not-air-pods out of their head and say "uhhh sure okay".
Does all of this change with AI? Yes. Did it make sense a year ago for business continuity reasons? Yes. Was it ever beautiful where the rubber hit the road? No.
Why aren't tires as complex as wiring harnesses?
otterley 15 hours ago [-]
> core principle of "infra as yaml" and "infra as piles of plain-text files in git" was that infra was supposed to be dumb.
The problem is that that's not true in practice. It's a myth, and always has been for all but the most trivial setups. Infrastructure is now highly dynamic and, like old cities, becomes more complex, calcified, and non-uniform with age.
Infrastructure configuration is an excellent example of the blurriness of code-vs-configuration. I'm now several decades into my career, and I'd rather have pure code generate my configurations than some Frankenstein's monster of "mostly config but lots of code in there when it gets too complicated" UX. Config file formats always evolve into this mess eventually, and a lot of my fellow practitioners don't seem to recognize this problem, leaving the cycle to continue.
EdwardDiego 1 days ago [-]
Golang string interpolation, to make it even more fun and hairy. I never understood why Helm didn't treat the YAML as a tree in memory so that you didn't have to worry about whitespace.
kryptn 1 days ago [-]
I definitely prefer using kustomize over helm, where the mentioned interpolation takes place.
cassianoleal 1 days ago [-]
I also prefer Kustomize over Helm. That said, their removal of vars was a boneheaded decision without a working replacement.
I set up a whole startup using Kustomize. It was great, except almost no-one used the inheritance feature except me - they all copy-pasted everything. Then the company grew and hired some dedicated devops people, who immediately started switching to Helm. I threw up my hands and went back to focusing on software development.
I’ve done some NGO-related work and in that context, they talk about “appropriate technology” - technology that’s suited to the local population, among other things. I’ve started to see tools like Helm and languages like Go as appropriate technology for many teams, but that doesn’t mean I’m going to switch to a pedal-powered water pump in my own house.
nokun7 1 days ago [-]
[flagged]
javcasas 1 days ago [-]
You can do JSON instead of YAML.
But no one does that because it is significantly uglier. So I guess we don't deserve sanity because we don't want our programs to be ugly?
nvme0n1p1 1 days ago [-]
The point is not how pretty the strings are. The point is there should be no strings at all. These are computer programs, so they should use normal objects. The python code `my_pod = nignx_pod()` will never throw a syntax error because the result has too many/too few spaces.
80fj0348i 1 days ago [-]
You can import the client library and work with types directly if you like. I've done that for years, it works great.
dilyevsky 1 days ago [-]
no one is stopping you from importing k8s types as protobufs and writing a program to talk to k8s api via protobuf encoding. all the types are already protos (except CRDs). it's just not the most user-friendly way so most people don't use it.
nvme0n1p1 1 days ago [-]
no, you don't have to resort to imperative protobuf slinging.
And plenty more exist today. But helm got popular first, so due to the collective action problem it's going to take the industry forever to switch off it.
dilyevsky 14 hours ago [-]
most ops people just like templates bc coding wasn't their strong suit (at least before AI) and helm delivered the least friction way of using templates.
No, crossplane types are CRDs so ironically you have to use the `unstructured` package for them instead of the proper typed structs in the core client libraries. (Unless you also generate typed libraries for your CRDs).
dilyevsky 14 hours ago [-]
crossplane is more about lifecycle management than just pure API
wodenokoto 1 days ago [-]
Of course it will. Just add a few spaces at the front of the line and it’ll throw an error.
solatic 1 days ago [-]
> The python code `my_pod = nignx_pod()` will never throw a syntax error because the result has too many/too few spaces
Yes, but it might throw errors because the client library you're using is too old and attempts to define fields that no longer exist in your newer infrastructure API. You do not find these errors at compile-time, but rather at runtime, and only after both serialization and API request execution finish and you get an error from the API, in the best case, or a silent failure misconfiguring your infrastructure, at worst.
The only time an infra engineer would trust the types is if the vendor themself supplied them, or there's a mainstream community effort to keep them up to date, or they're maintained in-house with procedures to try to keep them up-to-date with new releases. At the end of the day, the API takes a string, not some protobuf or CORBA or God knows what binary serialized monster dujour...
9dev 22 hours ago [-]
> Yes, but it might throw errors because the client library you're using is too old and attempts to define fields that no longer exist in your newer infrastructure API.
Well. Don’t use a client library that is too old then? Like you do in any other kind of software deployment, where you update your client libraries together with the upstream API you talk to?
solatic 21 hours ago [-]
> just use up-to-date dependencies
Yeah, sure, I'll file that away with the NPM "tens of thousands of tiny dependencies is OK" thinking. Constant dependency upgrades is a nightmare. There's a reason why shell + Helm is popular - they're stable.
9dev 21 hours ago [-]
There's a difference between NPM package updates and backwards-incompatible API changes, which shouldn't really occur often. Even in the NPM ecosystem, major version updates are not a regular event. So I'll take that over templated YAML in a heartbeat.
80fj0348i 1 days ago [-]
> The only time an infra engineer would trust the types is if the vendor themself supplied them, or there's a mainstream community effort to keep them up to date
Good news! The Kubernetes project has a documented compatibility strategy (https://kubernetes.io/docs/concepts/overview/kubernetes-api/...) and publishes new client library versions for every release. There's endpoints in the API to interrogate the API's versions and supported fields/features for every resource. Additionally, most API types will accept old versions and automatically convert them to the newer versions, wherever this is possible.
> At the end of the day, the API takes a string, not some protobuf or CORBA or God knows what binary serialized monster dujour...
This is not necessarily correct. Kubernetes API supports JSON, YAML, protobuf or CBOR over the wire, and all become in-memory structs on the server.
solatic 21 hours ago [-]
Kubernetes is a decent example of vendor-supplied and/or mainstream community effort, but that's beside the point that (a) Kubernetes is the exception, not the rule, and (b) Helm is popular for other reasons that belie non-string wire options, like: value overlays (how would you overlay binary-serialized CBOR again?), type safety for the stringly-typed JSON/YAML passed over the wire powered by validating admission webhooks, remote-executed-on-the-cluster hooks for pre-/post- installs/upgrades/uninstalls (big help for database migrations, for example, which cannot be connected to from the machine running Helm), etc.
80fj0348i 19 hours ago [-]
Everything you mentioned as a Helm feature is not in fact exclusive to Helm, and can be done either in code using the client libraries, or other deployment tools. For example I've written my own deployment tools that provide "value overlays" by simply merging the in-memory structs in Python or Go code - a manifest is just a data structure. I've also written code that can do ordered deployments with checks at multiple stages, which is both easier to do and more powerful when you write Python/Go code than trying to twist your dependencies to fit Helm's sync wave system.
12_throw_away 1 days ago [-]
Wait, but how does that help? Are text templates any better suited for creating JSON than they are for creating YAML? Trying to manipulate well-structured data as if it were syntax-free text seems downright deranged to me, regardless of the specifics of the format ...
burdock 1 days ago [-]
I've written at length about my disdain for text interpolation in helm.
I've done some experiments using Apple's Pkl and CNCF's KCL, which look to solve these problems with a simple "configuration language".
I don't think either have very widespread adoption, though I did hear a while back that Apple is using Pkl internally.
SOLAR_FIELDS 1 days ago [-]
I do thank the advent of LLM's for making the task of tracing through 3 layers of YAML rendering indirection to find an actual bug in layers of insanity. They are surprisingly good at it. A purist might look at the system and be taken aback in horror, but pragmatically the crazy templating things that used to be pretty major in terms of maintainability have become a lot less problematic.
I don't even mind text interpolation in helm that much. Yes, it's horrific, but it really doesn't become problematic until you are having an external system render the text being sent to helm that might be processed again after the fact with Kustomize. Your text might hit literally 3-4 different YAML rendering pipelines before it finally hits your cluster. Insanity
Hashicorp had a similar idea to this configuration language you are mentioning. They named it HCL. We do know how that turned out, but maybe that was more questionable design rather than problems endemic to the approach.
nvme0n1p1 1 days ago [-]
Even if you're all in on AI, it's still not a great situation. Do you want your agent to burn 10000 tokens fixing indentation bugs every time it touches a helm chart?
What takes effort for humans, takes effort for AI too.
And yeah, HCL was a horrible design. Anything else would have been better, Cue, jsonnet, dhall, pkl, etc. (I'm not sure which of those existed when HCL was created.)
80fj0348i 1 days ago [-]
> Do you want your agent to burn 10000 tokens fixing indentation bugs every time it touches a helm chart?
If your agent is doing this you need to get a much better agent. It should call `helm lint` to find those errors.
SOLAR_FIELDS 18 hours ago [-]
Helm lint will only find problems with whatever text is supplied to the Helm chart. It will not catch a large hunk of the class of errors being discussed here. There is plenty of scenarios that are valid Helm that are still invalid to be supplied to a cluster.
80fj0348i 18 hours ago [-]
I was responding specifically to finding indentation errors. Most of the class of errors you mentioned can only be validated by attempting to apply the manifest to a live cluster running the target API version anyway.
jen20 1 days ago [-]
HCL solves the problem of unknown values. It's certainly not a "horrible design" unless you're trying to compare it to something that does a different job.
For a single source of truth for YAML/JSON such as for Kubernetes manifests, any of the ones you list are fine though and a vast improvement on using Go templates for it.
(Yes I've used all of them - and indeed did the first public talk about Pkl two days after it was open-sourced, after several years of using it internally at Apple).
p_l 1 days ago [-]
It's why, except for upstream charts, I use exclusively jsonnet (through kubecfg) when I can.
String interpolation is a pox on the ecosystem, especially given how YAML was only originally a debug helper...
deathanatos 1 days ago [-]
> Why does it not matter if the state is unstable? If I'm operating a cluster that can't settle, I'd like to know immediately!
Because it's probably an impossible to answer question hinging on each person's pet definition of "stable".
In one example of an unstable cluster, pods are "regularly" crashing — but at a really low frequency, like, each pod crashes onces every few days. They're OOM'ing — a bug in the workload, not really the cluster's fault — but the control loop pattern still ensures the failure is corrected in the sense that the pod will either be restarted or evicted and re-created.
(But yes, you absolutely want monitoring to pick up on that. In the specific example here, I'd say it's "just monitor pod failures", and then you get not only OOMs, but all the other ways a workload might crash too.)
I've actually had more problems when the control loops get stuck. (Which they can, almost always because you've asked for the impossible. But sometimes it is hard for the various pieces to say more than "no solution". Why no solution is usually a human-level deduction.)
p_l 21 hours ago [-]
A good term for it is that every bigger system is going to be at best metastable
hckr1292 1 days ago [-]
> Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU"
Is it somewhat convoluted to setup? Absolutely! But it's possible to use relatively robust and well supported built in controllers with custom metrics.
12_throw_away 1 days ago [-]
> Is it somewhat convoluted to setup? Absolutely!
Kubernetes™
wpollock 1 days ago [-]
I sometimes wonder if Kubernetes was designed with selling support and training services as a goal.
fh973 1 days ago [-]
Built by Google and... Redhat!
jasonjayr 1 days ago [-]
I had read somewhere that Kubernetes is like a Tamagotchi for operations teams.
Thought it was just me when I was trying to set up HPAs.
SOLAR_FIELDS 1 days ago [-]
The node equivalent, also very useful for engineering cost effective cloud environments, is called KEDA.
Thev00d00 2 days ago [-]
Using literally anything other than helm will make your life 100x better with k8s.
bigstrat2003 1 days ago [-]
Helm is a great tool, but you absolutely must keep it simple. We use Helm at work, and our charts are 95% hard-coded YAML templates, with an occasional sprinkling of a Helm variable for values which differ between dev/test/prod. It works great and is a ton easier than Kustomize for that purpose. But I would never want to maintain the super complex Helm charts you see in public repos.
baby_souffle 1 days ago [-]
> We use Helm at work, and our charts are 95% hard-coded YAML templates, with an occasional sprinkling of a Helm variable for values which differ between dev/test/prod. It works great and is a ton easier than Kustomize for that purpose.
I have come to precisely the opposite conclusion. I can count all of the helm charts that didn't make me immediately want to cry with one hand... and I'd have at least 2 fingers still free.
If you have "base, and then we change three things per environment", Kustomize is about as good as it gets using simple patches.
at $dayJob, we have templates wrapping templates and not a ton of tests on anything. Lots of implicit defaults that are not documented anywhere other than in the master/root chart defaults. As far as I can tell, this was how policy was meant to be enforced rather than something like OPA.
bigstrat2003 1 days ago [-]
> If you have "base, and then we change three things per environment", Kustomize is about as good as it gets using simple patches.
Real talk, I could not for the life of me figure out how to use Kustomize. I spent a day or so poring over the documentation, and while I could recreate their hello world type examples, I couldn't figure out why they worked or how to make it work for our needs. Meanwhile, Helm immediately made sense and I had a chart banged out for the software I was trying to set up in under an hour.
I'm sure someone will tell me it's a skill issue, but Kustomize was simply way harder to use than Helm (or the documentation was worse, not sure which). So we wound up using Helm.
pirates 17 hours ago [-]
Way back in the day Kustomize didn’t support replacement or tempting for any arbitrary value, just some of them, and Helm didn’t have any of those restrictions. The instant we ran into a problem templating with Kustomize we dropped it and stuck with Helm. As you say maybe a skill issue but eh, what we have works.
Atotalnoob 1 days ago [-]
Then go change them?
ranger207 1 days ago [-]
I hate seeing public Helm charts that basically expose every single setting in the underlying manifests. What am I even using templates for then?
cwiggs 10 hours ago [-]
This was a great read. I've been a sysadmin/SRE/Devops/Platform/infra ( Why are there so many different titles) engineers for many years, for the past ~5 years I've been using Kubernetes and have helped companies migrate lots of workloads to k8s. I've worked with many people who dislike k8s, but almost all of them had almost no experience with k8s. A lot of the anti-k8s people would eventually build out something similar to k8s, which made me love [this blog post](https://www.macchaffee.com/blog/2024/you-have-built-a-kubern...) that helped explain that more.
Anyway, I love that you are/were skeptical of Kubernetes, but then looked into k8s more; we need more people like that in the world.
I think you did a great job explaining the internals of Kubernetes and then comparing it to "not" k8s. e.g. when you said the following:
> Compared to "classic" clusters, a Service provides a similar abstraction to the combination of:
** N nodes running Keepalived + NGINX (with a dynamic set of backends)
** CNAME records for the virtual (Keepalived) IP
It's a great point because a lot of anti-k8s people just say "Just put your service on a few VMs and put a LB in front of it". But if you don't just pay someone else to do it (ALB,NLB,etc) you gotta do something like Keepalived + NGINX like you mentioned, and then you are one step into creating your own k8s.
I also love how you still brought up issues with k8s. For as much love and experience I have of k8s, I feel like there is some odd behavior it has as defaults. e.g. the default is no cpu or mem requests (don't get me started on storage). I've worked many places where an engineers app that "shouldn't" use a ton of resources used up enough to cause the k8s worker node to have issues, which then caused other pods to fail.
mhh__ 24 hours ago [-]
My own version of this is that I was actually quite pleasantly surprised using k8s for the first time, expecting it to be much more complicated, but then pretty shocked at how complicated the ecosystem around it is.
sligbad 2 days ago [-]
Thanks for writing this up. The prose was really pleasant to read, a treat in this day.
ianjbutler 1 days ago [-]
K8s is a reconciliation engine for desired state, like many other adjacent tech in the infra space. Hating on yaml, json, missing types, or templating layers that drive the engine is kind of silly, and usually missing the point.. like hating dbs because you take issue with a sql layer or an orm layer. Disliking one or the other isn’t a deal breaker that invalidates the value proposition, so you may want to consider the other one before you retreat to flat text in csvs.
Kubernetes can be driven with tofu and hcl.. make it operator and even driven.. use any of many sdks, cdks to drive it, or add a completely different layer for faas or orchestration if that’s the relevant vocab for your problem space. It all runs on k8s.. that’s kinda the point.
> Given the Controller pattern, why isn't there support for "Cloud Native" architectures?
you're supposed to write your own.
> Why are the storage and networking implementations "out of tree" (CNI / CSI)?
> Given the above question, why is there explicit support for Cloud providers?
the in-tree stuff came first, and they have been extracting it in pieces for years now.
fh973 1 days ago [-]
In-tree is a maintenance and CI disaster. Openstack kept them in-tree and it has a huge cost both for the project (code reviews) and the vendors (required to keep a CI running, glacial progress due to lack of code reviewers).
ianeff 1 days ago [-]
If all you needed was a load-balanced nginx server, this would be a shallow, inaccurate reading of Kubernetes.
I have found that generating Kubernetes configurations using Jsonnet is quite pleasant. You don't have to use YAML. Jsonnet as far as I know, is the configuration language closest in semantics to Borg Config Language, which is what Google uses. I don't believe Googlers really thought people would use YAML directly. I think they assumed folks would use something nicer to generate YAML, which is the "assembly code" of Kubernetes configuration.
9dev 22 hours ago [-]
I’ll never understand why people don’t use JSON for generated Kubernetes definitions, which its API understands just fine. YAML is good when you need humans to read and modify it themselves, for config files and such. But why on earth would you use string manipulation tools to compose YAML templates? That takes a special kind of twisted cruelty.
red_candle_wax 1 days ago [-]
I hate it so much. I got moved from a software role to a devops/platform engineering role after some contract shifts and it's miserable.
At least some years ago, I could cope by viewing it as "resume fodder" but that doesn't even matter anymore.
simoncion 1 days ago [-]
If we had to live in a world with YAML, that world would have been so much better had every library that emits YAML defaulted to "flow" style instead of "block" style. Well, and had the standard fixed things so that the "|" multi-line string operator worked in "flow" style. That's a very convenient operator for embedding scripts into configuration.
But no, apparently '{' characters are scary or something.
jazdw 2 days ago [-]
> Basically, Horizontal Pod Autoscaler but with sensors which are not just "CPU"
I think you found the answer to that below - KEDA
tkz1312 1 days ago [-]
Using nix to generate k8s yaml is actually quite pleasant. You avoid all the string templating bullshit and can use nix functions and the nixos module system to build proper abstractions.
rgoulter 1 days ago [-]
I think this is worth elaborating on a bit.
I think any of the "JSON + functions" configuration languages mentioned in this thread (there are many) would be just as good.
Writing configurations in a modular way is really neat to be able to do. Rather than having to manage a single YAML file with the resource manifest, you can construct the resource by combining different parts of it.
e.g. rather than { image = "backend", tags = ["backend"] }, { image = "cache", tags = ["backend"] }, you could factor out the `tags`. -- Doing this from a "JSON + functions" language makes more sense than Helm's approach, from my experience.
The module system that NixOS uses builds an abstraction over this: each module declares like { options, config, output }. So the module author defines the option & output, and the module users declare the config, then the system knows how to render the configuration.
ewy1 17 hours ago [-]
my cope: yaml being a superset of json means you can usually do ` | toJson` in your templates instead of ` | toYaml | nindent x`
hugogu 1 days ago [-]
[dead]
forest_brothers 14 hours ago [-]
[flagged]
Rendered at 08:32:51 GMT+0000 (UTC) with Wasmer Edge.
It's very error prone to describe a hierarchy of versioned software packages and configuration with shell scripts and yaml. Or use text files in git as a database. Or use the CI system as a bad substitute for a proper service (in order to do a simple thing, you must first create the world from scratch). And so on. Mainstream industry knows types, programming languages, relational databases, graphs and you name it, but those have not yet penetrated sufficiently into the infrastructure world. The understanding of the value of having types has been paid in sweat and blood when developing software and databases. Even a basic CRUD application with a basic SQL database will run rings around these Frankenstein monsters: in safety, speed, and extendability.
The whole problem on the infrastructure side is that the APIs are slow and you don't have exclusive access. Building the CRUD app would be a million times harder if SQL calls took 100x longer to execute and other people could randomly reach into the database and change the schema and data at will - can your basic CRUD app gracefully handle somebody shoving newlines at the end of usernames? Odds are that the first time, no, not until you put a trim() call. And then you cry in pain at needing to put trim() calls everywhere. And then you gnash your teeth at people reporting performance problems because of all the defensive checks you put in because all trust has been thrown out. And then you find out that trim() was deprecated and will now be replaced by strim() that handles Unicode a little differently, and you need to port your code before End of Life to stay compliant.
A basic CRUD application with a basic SQL database doesn't solve the problem, because you don't seem to appreciate that there already is a database - however your vendor stores configuration - and it's your job to deal with it. Adding another database on top of it would make the problem worse.
If you are infrastructure software minded, you might store the data in files that are stored in git. For users to read the data, they can perhaps browse the repo or clone it. For some more complex questions you might have a document which tells the user how to fire a CI pipeline that runs some code over the data and the user has to read the pipeline logs to get the information. The CI pipeline will probably run in a container which will produce a large log and take a long time to start.
If you are software software minded, you store the values in a database, make a service with a REST or some other API and create a browser UI. User can browse the data in a table view and perform searches and perhaps there's some more complex business logic executed on the backend to get answers to some more complex questions.
In this case, both systems can easily do the job. But the solutions are very different.
We have a multitude of configuration/markup languages all doing the exact same thing.
We have solved this problem in the 60s: S-Expressions. Should have just used that for everything. Configuration, code, websites, you name it. Infinitely extendable, easy to parse.
And if you really love sexps, then writing a bunch of arrays in JSON is also viable. If you asked me 10 years ago I might've complained about JSON, but now I think we messed up by having a bunch of other formats, tbh. I have my own TUI JSON editor that can render in tree view, miller tree view, flat view, and inferred table view, and it makes editing JSON very easy too.
Oh and duckdb can query against compressed json and jsonl at GB/s
That seems to be everyone's initial goal, but then it inevitably gets leaky.
I think the dominant split here that forces everything into a weird mismash is that nobody really wants to have to understand arbitrary programming language configs for third-party stuff, they want to read config and know what's going on. BUT for their own stuff, they find reasons to want more power and expressibility...
This is very much true. And until they do get leaky, you have to wrap them in all sorts of weird templates or other forms of abstraction.
This is one of the reasons I used to really like Chef's config language. It was a Ruby DSL, which meant that anything the DSL couldn't do natively, you could just use Ruby right there.
Of course, this is easily misused, but if you were careful it was as powerful as it was elegant.
They say they don't want it but every configurmation format eventually ends up becoming turing complete at some point.
Cue also has a lot of schema validation which is better than dynamic languages.
If it compiles, it’s almost certainly correct.
A reliability engineer's nightmare, and a common one, is stepping into a company whose deployment pipeline is built in-house, with dynamic bespoke software. What happens when Jeff leaves to build his own startup and absolutely no one knows how that Node.js script shells out to SSH to call docker on a remote server? I can tell you what happens: everything works fine for awhile and then suddenly doesn't for awhile.
On the other hand, you join a company with K8s and plain-jane YAML and can instantly grok the entire system and fix prod on your first day while the S-team sweats behind you, watching every key stroke. The seconds I have to read some javascript or bash or perl are precious seconds the brand is suffering.
In software engineering, flexibility and abstraction is a benefit: the product must pivot and evolve. In reliability engineering we want dead simple, tried and true. You want to be able to step outside your SF office and say "Someone help me with EKS, my DevOps guy died last night in a car crash!" and have 4 people with wild hair take their weird off-brand not-air-pods out of their head and say "uhhh sure okay".
Does all of this change with AI? Yes. Did it make sense a year ago for business continuity reasons? Yes. Was it ever beautiful where the rubber hit the road? No.
Why aren't tires as complex as wiring harnesses?
The problem is that that's not true in practice. It's a myth, and always has been for all but the most trivial setups. Infrastructure is now highly dynamic and, like old cities, becomes more complex, calcified, and non-uniform with age.
Infrastructure configuration is an excellent example of the blurriness of code-vs-configuration. I'm now several decades into my career, and I'd rather have pure code generate my configurations than some Frankenstein's monster of "mostly config but lots of code in there when it gets too complicated" UX. Config file formats always evolve into this mess eventually, and a lot of my fellow practitioners don't seem to recognize this problem, leaving the cycle to continue.
https://github.com/kubernetes-sigs/kustomize/issues/4049
I’ve done some NGO-related work and in that context, they talk about “appropriate technology” - technology that’s suited to the local population, among other things. I’ve started to see tools like Helm and languages like Go as appropriate technology for many teams, but that doesn’t mean I’m going to switch to a pedal-powered water pump in my own house.
But no one does that because it is significantly uglier. So I guess we don't deserve sanity because we don't want our programs to be ugly?
https://tanka.dev/
https://www.pulumi.com/
And plenty more exist today. But helm got popular first, so due to the collective action problem it's going to take the industry forever to switch off it.
Yes, but it might throw errors because the client library you're using is too old and attempts to define fields that no longer exist in your newer infrastructure API. You do not find these errors at compile-time, but rather at runtime, and only after both serialization and API request execution finish and you get an error from the API, in the best case, or a silent failure misconfiguring your infrastructure, at worst.
The only time an infra engineer would trust the types is if the vendor themself supplied them, or there's a mainstream community effort to keep them up to date, or they're maintained in-house with procedures to try to keep them up-to-date with new releases. At the end of the day, the API takes a string, not some protobuf or CORBA or God knows what binary serialized monster dujour...
Well. Don’t use a client library that is too old then? Like you do in any other kind of software deployment, where you update your client libraries together with the upstream API you talk to?
Yeah, sure, I'll file that away with the NPM "tens of thousands of tiny dependencies is OK" thinking. Constant dependency upgrades is a nightmare. There's a reason why shell + Helm is popular - they're stable.
Good news! The Kubernetes project has a documented compatibility strategy (https://kubernetes.io/docs/concepts/overview/kubernetes-api/...) and publishes new client library versions for every release. There's endpoints in the API to interrogate the API's versions and supported fields/features for every resource. Additionally, most API types will accept old versions and automatically convert them to the newer versions, wherever this is possible.
> At the end of the day, the API takes a string, not some protobuf or CORBA or God knows what binary serialized monster dujour...
This is not necessarily correct. Kubernetes API supports JSON, YAML, protobuf or CBOR over the wire, and all become in-memory structs on the server.
I've done some experiments using Apple's Pkl and CNCF's KCL, which look to solve these problems with a simple "configuration language".
I don't think either have very widespread adoption, though I did hear a while back that Apple is using Pkl internally.
I don't even mind text interpolation in helm that much. Yes, it's horrific, but it really doesn't become problematic until you are having an external system render the text being sent to helm that might be processed again after the fact with Kustomize. Your text might hit literally 3-4 different YAML rendering pipelines before it finally hits your cluster. Insanity
Hashicorp had a similar idea to this configuration language you are mentioning. They named it HCL. We do know how that turned out, but maybe that was more questionable design rather than problems endemic to the approach.
What takes effort for humans, takes effort for AI too.
And yeah, HCL was a horrible design. Anything else would have been better, Cue, jsonnet, dhall, pkl, etc. (I'm not sure which of those existed when HCL was created.)
If your agent is doing this you need to get a much better agent. It should call `helm lint` to find those errors.
For a single source of truth for YAML/JSON such as for Kubernetes manifests, any of the ones you list are fine though and a vast improvement on using Go templates for it.
(Yes I've used all of them - and indeed did the first public talk about Pkl two days after it was open-sourced, after several years of using it internally at Apple).
String interpolation is a pox on the ecosystem, especially given how YAML was only originally a debug helper...
Because it's probably an impossible to answer question hinging on each person's pet definition of "stable".
In one example of an unstable cluster, pods are "regularly" crashing — but at a really low frequency, like, each pod crashes onces every few days. They're OOM'ing — a bug in the workload, not really the cluster's fault — but the control loop pattern still ensures the failure is corrected in the sense that the pod will either be restarted or evicted and re-created.
(But yes, you absolutely want monitoring to pick up on that. In the specific example here, I'd say it's "just monitor pod failures", and then you get not only OOMs, but all the other ways a workload might crash too.)
I've actually had more problems when the control loops get stuck. (Which they can, almost always because you've asked for the impossible. But sometimes it is hard for the various pieces to say more than "no solution". Why no solution is usually a human-level deduction.)
Since Kubernetes 1.23, there's support for using "custom metrics" to control HPAs: https://kubernetes.io/blog/2026/07/14/custom-metrics-exporte...
Is it somewhat convoluted to setup? Absolutely! But it's possible to use relatively robust and well supported built in controllers with custom metrics.
Kubernetes™
... and while searching for where I found that quote, it seems someone vibecoded that metaphor literally: https://github.com/Yscale-sh/Kubagachi
I have come to precisely the opposite conclusion. I can count all of the helm charts that didn't make me immediately want to cry with one hand... and I'd have at least 2 fingers still free.
If you have "base, and then we change three things per environment", Kustomize is about as good as it gets using simple patches.
at $dayJob, we have templates wrapping templates and not a ton of tests on anything. Lots of implicit defaults that are not documented anywhere other than in the master/root chart defaults. As far as I can tell, this was how policy was meant to be enforced rather than something like OPA.
Real talk, I could not for the life of me figure out how to use Kustomize. I spent a day or so poring over the documentation, and while I could recreate their hello world type examples, I couldn't figure out why they worked or how to make it work for our needs. Meanwhile, Helm immediately made sense and I had a chart banged out for the software I was trying to set up in under an hour.
I'm sure someone will tell me it's a skill issue, but Kustomize was simply way harder to use than Helm (or the documentation was worse, not sure which). So we wound up using Helm.
Anyway, I love that you are/were skeptical of Kubernetes, but then looked into k8s more; we need more people like that in the world.
I think you did a great job explaining the internals of Kubernetes and then comparing it to "not" k8s. e.g. when you said the following:
> Compared to "classic" clusters, a Service provides a similar abstraction to the combination of:
It's a great point because a lot of anti-k8s people just say "Just put your service on a few VMs and put a LB in front of it". But if you don't just pay someone else to do it (ALB,NLB,etc) you gotta do something like Keepalived + NGINX like you mentioned, and then you are one step into creating your own k8s.I also love how you still brought up issues with k8s. For as much love and experience I have of k8s, I feel like there is some odd behavior it has as defaults. e.g. the default is no cpu or mem requests (don't get me started on storage). I've worked many places where an engineers app that "shouldn't" use a ton of resources used up enough to cause the k8s worker node to have issues, which then caused other pods to fail.
Kubernetes can be driven with tofu and hcl.. make it operator and even driven.. use any of many sdks, cdks to drive it, or add a completely different layer for faas or orchestration if that’s the relevant vocab for your problem space. It all runs on k8s.. that’s kinda the point.
you're supposed to write your own.
> Why are the storage and networking implementations "out of tree" (CNI / CSI)?
> Given the above question, why is there explicit support for Cloud providers?
the in-tree stuff came first, and they have been extracting it in pieces for years now.
At least some years ago, I could cope by viewing it as "resume fodder" but that doesn't even matter anymore.
But no, apparently '{' characters are scary or something.
I think you found the answer to that below - KEDA
I think any of the "JSON + functions" configuration languages mentioned in this thread (there are many) would be just as good.
Writing configurations in a modular way is really neat to be able to do. Rather than having to manage a single YAML file with the resource manifest, you can construct the resource by combining different parts of it.
e.g. rather than { image = "backend", tags = ["backend"] }, { image = "cache", tags = ["backend"] }, you could factor out the `tags`. -- Doing this from a "JSON + functions" language makes more sense than Helm's approach, from my experience.
The module system that NixOS uses builds an abstraction over this: each module declares like { options, config, output }. So the module author defines the option & output, and the module users declare the config, then the system knows how to render the configuration.