Would you consider `foo` a blue function and `bar` a red function? That doesn't seem particularly helpful to me.
The virality of async await is that once you mark a function async, then you can only call it from another async function, which forces you to mark more functions async, which in turn means that if you want to use blocking I/O APIs then you just can't because it's incompatible with your execution model because by daring to express asynchronicity of operations, you were forcefully opted into stackless coroutines.
That's what Zig solves, and that's what is real function coloring. People have written reimplementations of the same libraries multiple times because of it.
Just as an example. Note also how, coincidentally, this duplication of effort resulted in asyncio-redis being semi-abandoned and looking for maintainers. And you have to have both libraries because the asyncio one can't do blocking, and vice versa the other one can't do async.
Would you write two instances of essentially the same library just because one is missing an argument that gives it access to an `Io` interface? No, because you would just pass that extra argument around and nothing else would have to change.
woodruffw 17 days ago [-]
> Would you consider `foo` a blue function and `bar` a red function? That doesn't seem particularly helpful to me.
In the sense of effect/capability typing, I think the answer is yes.
"Coloring" isn't magical, it's just a way to describe effects. Those effect can be described by keywords (`async` in JS, `throws` in Java, etc.) or special token parameters/types (what Zig does), but the consequences are the same: the effect propagates to the caller, and the caller becomes responsible for dealing with it.
nextaccountic 17 days ago [-]
Yeah, parameter passing can be seen as some kind of effect (like the Reader monad in Haskell). Passing parameters down the call stack is viral just like function coloring (if you need to do i/o and thus need to receive an io parameter, then your caller must receive an io parameter too, recursively; this is analogous to adding a keyword async in front of your function, and to its callers recursively). The solution is probably some sort of implicit parameter, again, like the Reader monad.
Note: async fn in Rust is also just a fn (in Rust's case, a fn that returns a future). It turns out that returning a future, or receiving as parameter a reference to the runtime, are equivalent in some sense
koiueo 17 days ago [-]
> Passing parameters down the call stack is viral
It's in a different league comparing to async-await abomination.
For one, you can store parameter on a struct thus working around the "virality" on a call site.
0x457 17 days ago [-]
Huh?
You have the following call stack (top to bottom): a -> b -> c -> d
Function "d" does IO, so a, b and c need to pass that IO down. It's the same as with async in terms of virality.
If I understood correctly, the main benefit here is that you don't have to dance around "can I call this function or is it going to block async runtime reactor?"
I would pick async/await style over this because (I think) extra function argument is higher visual load than "async fn".
`greet` has Io within its closure, so I’m pretty sure this is consistent with the GP’s point.
koiueo 17 days ago [-]
If you want to print a string from your function, you must supply it to the function. The fact that your function now receives `str: []u8` doesn't make it str-colored.
On a side note: the entire "coloring" metaphor is extremely unfortunate and confusing IMO. It's somewhere at the top of my list (right beside the shroedinger's cat) of things which could've been explained better without dumbing them down.
woodruffw 17 days ago [-]
That’s because str is not a meaningful effect, so we wouldn’t say that a function is “str-colored.”
Function coloring is indeed confusing, which is why I’ve been trying to ground this on effects instead. Effects are IMO easier to understand, and this is a straightforward example of effect typing.
0x457 15 days ago [-]
If this is a special str that you have to pass down all the way through your call stack, then yes, it's str-colored.
Rapzid 15 days ago [-]
It's also not true you can't call an async function from a sync function. In C# and JavaScript You can, you just can't "await" it. There are a lot of details involved, but it's just not true that you can't call async from sync.
n42 17 days ago [-]
I've been trying to beat this point in and failing. If a parameter type creates "colors", you can extrapolate that to an infinite set of colors in every single language and every single standard library, and the discussion on colors becomes meaningless.
Some people are so focused on categorical thinking that they are missing the forest for the trees.
The colors are a means of describing an observed outcome -- in Node's case, callback hell, in Rust's, 4 different standard libraries. Whatever it may be, the point is not that there are colors, it's the impact on there being colors.
> But there is a catch: with this new I/O approach it is impossible to write to a file without std.Io!
This sentence just makes me laugh, like it's some kind of "gotcha". It is the ENTIRE BASIS of the design!
dwattttt 17 days ago [-]
> you can extrapolate that to an infinite set of colors in every single language and every single standard library, and the discussion on colors becomes meaningless.
It's more that discussion about most of them becomes meaningless, because they're trivial. We only care when it's hard to swap between "colours", so e.g. making it easy to call an Io function from a non-Io function "removes" the colouring problem.
hamandcheese 17 days ago [-]
> so e.g. making it easy to call an Io function from a non-Io function "removes" the colouring problem.
Exactly. In golang (which is also a cooperatively multithreaded runtime if I understand correctly), calling a function that needs IO does not infect the callers type signature.
In async rust, and in "async param" zig, it does.
naasking 17 days ago [-]
Another poster up thread identified the exact problem: async/await contexts are not first-class values, they are second class citizens. If they were values then you could just stick the context in a struct/class and pass that around instead, and avoid having to refactor call chains every time something changes. It's their second class status that forces the "colouring" into the function signature itself at each point. This is also why ordinary first class values do not introduce colours, ie. you can hide new values/parameters inside other types that are already part of the function signature, thus halting the propagation/vitality of the change.
Of course, if these async contexts were first class citizens then you've basically just reinvented delimited continuations, and that introduces complications that compiler writers want to avoid, which is why async/await are second citizens.
SkiFire13 17 days ago [-]
> async/await contexts are not first-class values
Because async/await are not values, they are ways to run/structure your code. That's why they are so infectious. If you don't want the division the only solution is to make everything of one kind. Languages like C make everything sync/blocking, while languages like Go make everything async.
naasking 16 days ago [-]
> Because async/await are not values
They are computations that produce values. Computations can be reified as values. What do you think functions and threads are?
As I described above, "delimited continuations" are values that subsume async/await and many other kinds of effects. You can handle async/await like any other value if they were reified as delimited continuations, but this makes the compiler writer's life much more difficult.
SkiFire13 16 days ago [-]
> As I described above, "delimited continuations" are values that subsume async/await and many other kinds of effects.
Supporting delimited continuations force some specific ways of performing computations. They are akin to making everything async, which proves my point: you have to make everything of one kind in order to solve the problem.
naasking 16 days ago [-]
> They are akin to making everything async, which proves my point: you have to make everything of one kind in order to solve the problem.
You can use ordinary direct style compilation, but all references to stack values simply have to be relative offsets, then a simple implementation of shift/reset is just capturing context and copying stack fragments, which you can do using setjmp/longjmp in C (although there are better ways [1]).
This is not akin to making everything async, nor is everything "of one kind", whatever that means. A delimited continuation is very much its own kind of thing, distinct from other values, and doesn't have to influence the function call/return semantics unless you're targeting a less flexible runtime like the JVM.
> but all references to stack values simply have to be relative offsets
Then such references are no longer pointers, and in order to have a generic reference (that can point to both stack memory and heap memory) you have to store additional data (which one of them it points to) and conditionally use the correct one any time you access it. This is a very invasive change to the memory model.
> then a simple implementation of shift/reset is just capturing context and copying stack fragments, which you can do using setjmp/longjmp in C
That sounds like you're just reinventing green threads then, which basically forces everything to be async once you start noticing its issues.
> This is not akin to making everything async, nor is everything "of one kind", whatever that means.
Sure, if everything has to be able to "wait" for the result of a continuation then you're forcing async support on everything.
If instead you implement continuations as some kind of monads that users have to return from their functions then you got function coloring because you can't normally wait for the result of a continuation from a function that does not return a continuation.
> Sure, if everything has to be able to "wait" for the result of a continuation then you're forcing async support on everything.
If "forcing async support" doesn't mean that code has to change, or any other code in the call chain, then it's just meaningless pedantry.
dwattttt 17 days ago [-]
> the only solution is to make everything of one kind
That's not really the case. All you really need is a way to run async code from a sync function; "keep doing this async thing until it's done" is the primitive you need, and some languages/runtimes offer this.
SkiFire13 16 days ago [-]
Going by that reasoning then Rust solved the colored function problem too: when calling an `async` function from a sync one use `block_on`, while when calling a sync blocking function from an `async` one use `spawn_blocking`, but somehow people are not happy with this.
dwattttt 16 days ago [-]
Yes, it's quite curious. Having used both block_on and spawn_blocking, and not being worried about what "colour" my function is, I am also quite confused about the fuss.
On a practical note, since Rust doesn't standardise on an async runtime, it would be more accurate to say tokio solved the coloured function problem, for whatever that means. Or any and everyone else that made it easy to call one coloured function from another.
vips7L 17 days ago [-]
Go is preemptive. Async/await is cooperative because you’re explicitly cooperating and yield.
17 days ago [-]
alex_hirner 17 days ago [-]
> The virality of async await is that once you mark a function async, then you can only call it from another async function
That's not exactly true. Many languages return a curried version of it, which can be executed once you have a runtime.
rk06 17 days ago [-]
Well said. Honestly I don't understand this "function coloring argument" at all in zig's context.
Function is the smallest unit of logic. And function parameter is the fundamental way to control it.so, in theory, __A function parameter is the smallest possible design choice you have to control async vs sync__
There is no way to reduce it further. As you need to encode this somehow, otherwise it will be implicit and very hard to debug.
functions can compose. Function parameters can be composed. Making this a solid design choice.
Zig has somehow achieved this feat of decomposing async/sync to a single parameter. This deserves an ovation
Other langs should take note.
metaltyphoon 17 days ago [-]
> in theory, __A function parameter is the smallest possible design choice you have to control async vs sync__
So we are just going to forget Go exists?
rk06 17 days ago [-]
Disclaimer: I have not used golang, so i might be wrong.
as per my understanding, golang has it implicit. Like there is no way for a developer to tell if code will run under async context or not.
throwawaymaths 15 days ago [-]
no, in golang (and erlang) every function is async, and there are implicit yield points in "well-known" places. for erlang it's every function call boundary, for golang its the end of functions. in the past this was a problem for go because a for (ever) loop could block your cpu process. not a problem for erlang since all forever loops must be tco function calls (inducing a yield at each interation)
ivanjermakov 17 days ago [-]
> No, because you would just pass that extra argument around and nothing else would have to change.
> Semantically, passing std.Io to every function is no different from making every Node.js function async and returning a promise. Zig shifts function coloring from blocking/non-blocking choice to io/non-io.
> Complains are about how inconvenient it is to work with [blocking and non-blocking function handling] differences
"Extra argument" in this case is not much different from "extra async keyword", because you can only run I/O functions when having std.Io in scope.
Dylan16807 16 days ago [-]
They're similar, but if you're not writing Haskell it's super easy to make a variable always be in scope. You don't have to change all the intermediate functions to do it.
tcfhgj 17 days ago [-]
> The virality of async await is that once you mark a function async, then you can only call it from another async function
Rust calling async function in non-async function:
...
// Create the runtime
let rt = Runtime::new().unwrap();
// Get a handle from this runtime
let handle = rt.handle();
// Execute the future, blocking the current thread until completion
handle.block_on(async {
println!("hello");
});
Of course, spinning up a new runtime within the context of a boundary like that is probably wasteful (lots of new threads created if you’re not careful). But you could stash that runtime behind a OnceLock (you’d need to block_on the Handle I imagine rather than the Runtime directly, but doable).
And calling blocking from non-blocking:
let result = tokio::task::spawn_blocking(|| {
5
}).await;
This of course is basically essentially what Zig is doing, except instead of hidden global state it’s parameter passed. This is one area Zig does do better in - I wish Rust would default more to instance state instead of implicit global state.
Dylan16807 16 days ago [-]
So that's an escape in Rust and tokio.
But consider a language that doesn't have block_on, like javascript.
audunw 17 days ago [-]
I really don’t agree with the idea that this is functional colouring. Then we have to start talking about function colouring in a whole bunch of new contexts like with Zigs explicit passing of allocator. Or any other parameter that needs to be explicitly passed to use some kind of interface.
I think we should stick to talking about colouring when there is special calling conventions or syntax, which has the consequence of having to write separate libraries/modules for async code and non-async code.
That is the significant problem we have been seeing with many async implementation, and the one which Zig apparently fully solves.
dwattttt 17 days ago [-]
> Then we have to start talking about function colouring in a whole bunch of new contexts like with Zigs explicit passing of allocator.
That's pretty much where we are though. If you have a function that isn't passed an allocator, and now it needs to call a function that does take an allocator, we're in the same place.
Rust's 'async' keyword changes the type of the return value, but you can just write the different return value yourself; it's 'coloured' purely by what it means to be returning a 'Future'.
koolala 17 days ago [-]
So its impossible to get an allocator through a different means from inside a function? It must be passed in?
dwattttt 17 days ago [-]
No, it's entirely possible. And that's why we don't think about "has an allocator" as a colouring problem.
Likewise if JavaScript had an easy way to get a handle to its Runtime, and a function "block on promise" in its early days, we'd have never had all these "colouring" arguments.
Dylan16807 16 days ago [-]
If javascript worked differently we wouldn't have arguments about that significant feature of javascript.
How would you add "block on promise"? Javascript started off without the ability to block on anything, not even I/O because it didn't have I/O. Later when XMLHttpRequeest was added it was purely callback-based.
There are ways to make that API work but introducing the concept of blocking would be a big language change. It's not fixing a flaw, it's trading away some complications for a different pile of complications. There's still a conflict to resolve in language design when you want some functions that are synchronous and don't block and other functions that are asynchronous.
Edit: I guess javascript blocks on alert() but that doesn't seem like what you'd want "block on promise" to do...
dwattttt 16 days ago [-]
JavaScript is (notionally) "single threaded", you can block with just a busy loop; that's not a language change. It's bad design to block, which I expect is what you mean, but that's not a language problem.
If you did so in a Worker, you'd have your "don't block the UI thread" in there too.
Dylan16807 16 days ago [-]
If you busy loop waiting for something in javascript, your program dies. They're not just a bad idea, they're a non-option.
Blocking in the sense of blocking I/O or blocking on a system call is not possible in javascript. So again I ask what you have in mind for a "block on promise" call. The ideas that come to my mind all require significant redesigns of the execution model, or copping out by making every function async and treating it like an await.
In particular with workers in mind, I'll put it this way: If you don't allow "block on promise" in the main thread, you generally avoid these issues, but you also don't solve the problem, you still have async versus not async causing many headaches. If you do allow it on the main thread, how do you keep your page functioning? I can imagine resolutions but they all have severe compromises.
dwattttt 16 days ago [-]
Every time you call a JS function, it's blocking; if it internally busy-looped, you'll die just as much as if you do the loop yourself.
Being able to block on an async function call is just as much of a risk; if it returns quick, you live, if it returns slow, you die.
The solution to "main page unresponsive, terminate page?" is to always return promptly to the Runtime, so e.g. make everything async. But JavaScript is a language, and it doesn't just exist on web page main threads. As much as JS's proliferation everywhere else isn't my cup of tea, a blocking call in a JS cli tool is what you want.
EDIT: I guess to be explicit:
> So again I ask what you have in mind for a "block on promise" call
I expect it to have the same guarantees and risks of calling a non-async JS function.
Dylan16807 16 days ago [-]
> Every time you call a JS function, it's blocking; if it internally busy-looped, you'll die just as much as if you do the loop yourself.
But in a very temporary way. If it takes seconds that's a bug, while if blocking I/O takes seconds that's normal behavior.
And I've never seen a javascript busy loop in non-malicious code.
> I expect it to have the same guarantees and risks of calling a non-async JS function.
The main guarantee is that the function will be done and have all its temporary data cleaned up before your (non-worker) execution switches to other code. And in practice you can expect it to return within .1 seconds.
That guarantee is going to be an issue if you try to introduce some kind of "block on promise". And the timing expectation is meaningful too.
woodruffw 17 days ago [-]
This is effectively a special calling convention: Zig expects you to pass in a "token" object that communicates a kind of effect (I/O in this case). No token, no effect (modulo a soundness hole).
This is not a new pattern, and I think it's a pretty good one (and is arguably more ergonomic and general than syntax-level effects). But it's quintessential function coloring.
throwawaymaths 17 days ago [-]
> Zig expects you to pass
It does no such thing. you could pass a function a vtable and the vtable could have one implementation that calls an io stashed in the parent of the vtable, and a different vtable that doesnt and the function calling the vtable would be none the wiser. what is the color of the function that took the vtable?
this is not just academic; it would be for example the basis for mocked integration tests on a database or over the net api call.
woodruffw 17 days ago [-]
That's a calling convention, with indirection. You need some kind of capability token for this kind of asynchronicity scheme; it doesn't matter how you get it, but it needs to be there.
To be clear, there's nothing wrong with this; it's just another way to encode capabilities/effects.
> what is the color of the function that took the vtable?
It has the I/O effect.
throwawaymaths 17 days ago [-]
> It has the I/O effect
it does not, because it can take a vtable that does not call i/o
concretely:
const VTable = struct {
f: &fn (*VTable) void,
};
const A = struct {
io: IO,
v: VTable = .{ .f = &A.uses_io },
fn uses_io(this: *VTable) void {
const self: *A = @fieldParentPtr(.v, this);
self.io.some_io_fn(...);
}
};
const B = struct{v: VTable = .{.f = &void_fn}};
fn void_fn(_: *VTable) void {}
\\ WHAT IS THE COLOR OF THIS FUNCTION?
pub fn calls_vtable(v: VTable) {
v.f()
}
woodruffw 17 days ago [-]
It also has the I/O effect. You don’t need to call or make use of an effect to be “tainted” by it; it just needs to be in the closure (or whatever scope is relevant in the language).
Intuitively: if you mark a function as async, it doesn’t stop being async “colored” just because you don’t actually perform any async operations in it. This is the same thing.
throwawaymaths 16 days ago [-]
you might be talking about stackless coroutines, which are currently not part of zig, in which case, yes, the compiler might [0] have to instantiate different functions under the hood for the stackless-coro and the non-stackless-coro cases, with some mechanism to drop in an executor at the boundaries. until then its really hard to claim that the functions are colored.
[0] But even in that case there's not really coloring because if you provide a single io implementation there won't be different functions, even at the compiled level.
woodruffw 16 days ago [-]
No, I really just mean from a PLT perspective. The sync/async implementation used under the hood doesn't matter; what matters is that an `Io` in the closure is a token type for communicating effects. No instantiated or enclosed token; no effects.
People seem to be really defensive about this, like it's a bad thing. It isn't! It's arguably a significantly cleaner way to handle what people confusingly call "coloring." But that doesn't make it not "coloring," because coloring is about effects and vitality, not about keywords and syntax.
throwawaymaths 16 days ago [-]
1. the original author of the function coloring post wad not exactly a theorist so if you're applying some other idea of what coloring is, then you're muddying the waters.
2. here's what i have to say about your idea of what coloring is: That's all fine and good in theory, but in practice it makes no difference, at the user level, or at the compiler level.
woodruffw 16 days ago [-]
I don’t know anything about the original author, but I do know what function coloring is. It’s a way to describe effects. An effect is a function color.
> That's all fine and good in theory, but in practice it makes no difference, at the user level, or at the compiler level
Every example given so far shows Io’s virality, so I don’t know how you can assert how it doesn’t make a difference. The entire point of the design appears (reasonably) to be to introduce a token object that conveys an effect, rather than requiring a runtime to intermediate that effect.
Again, none of this is bad. Effect typing is cool. But it is, by definition, a way to color a function.
(Maybe the confusion here stems from the fact that languages like Go appear to have it “both ways” without coloring. Go is able to do that because it has an intrusive runtime that intermediates asynchronous events. Zig can’t do the same thing without making the same compromises as Go vis a vis FFI performance and ABI compatibility.)
throwawaymaths 16 days ago [-]
The vtable example is nonviral.
woodruffw 16 days ago [-]
I don’t see how that can be the case, given that Io is in the closure. Anything that wants to do I/O needs that token; that’s what virality is.
grayhatter 16 days ago [-]
I think I almost agree with what I think you mean, but I'm not sure, hopefully you'll entertain a few questions?
Say you have two functions one that does some kind of IO, (say a pair of bidirectional read, write, from stdin and stdout) and another that returns a block of memory of a constant size. Would you say that one is colored, and the other isn't?
I'll also try to make an attempt at the idea I'm trying to figure out too. I genuinely can't predict which side you'll answer for, but I'm assuming you'll say that it's not colored, because while it does do IO, but only directly though the global file descriptors which have no baring on the calling conventions. Which means the decisions and impact about which color this IO function doesn't ever apply to the callers and callees. This I think is the virality you mean where the callers and callees are required to know [something] about the semantics? I probably agree you could call this coloring and defend it... It's interesting that this might be closer to the "accepted" definitions of what any language means when it says this function is colored. But I don't think this understanding most people have about the concept of coloring. Most I see are arguing about the semantic effects, much more than the "accepted" definition. I have the distinct impression that when most say coloring, the most significant implication is when and where you resolve the specific color. Will it be sync or async code, when, where, and who gets to decide.
Or perhaps I'm wrong, and you would say that one function is colored, but the other isnt? Then I'd ask if you can tell it's colored, can you tell me which color?
In both cases I'd argue it's better to embrace the semantic widening of the concept of function coloring, because what I understand from how you describe it, I feel that saying this is coloring is strictly less enlightening about the reality of the code and system together, than calling this function alpha. With zig's new IO interface, I can't tell you what the RGB values are for the color, but I can confidently say, it's alpha channel is 0xff. You might not be able to tell me what color it is, but it without a doubt isn't a pure function, interacts with IO and does have a null byte in the alpha channel.
throwawaymaths 16 days ago [-]
im no virologist but i don't think your definition of viral is reasonable. this is like saying "anything that wants to use a database needs a socket, so a socket is viral"
16 days ago [-]
hamandcheese 17 days ago [-]
> which has the consequence of having to write separate libraries/modules for async code and non-async code.
That is not a consequence of function coloring or syntax, it is a consequence of having multiple ways of performing IO.
camgunz 17 days ago [-]
Look, either you move the program counter to a different place in memory (function call) or you push a task into an event loop. Even if you somehow elide all these differences, they're so different under the hood you'll always have to know in some circumstances. It's honestly wild we conflate them at all.
et1337 17 days ago [-]
I think this is a good kind of function coloring. It would avoid some scars I have from:
- seemingly harmless functions that unexpectedly end up writing four different files to disk.
- Packages that do I/O or start threads when you simply import them.
hiccuphippo 17 days ago [-]
I don't care about the color of the function. What matters is if you'll have to write two versions of the function, and this seems to solve that.
andyferris 17 days ago [-]
I am actually really excited about this.
The issues I've had with function colouring had to do with trying to compose code using (or expecting) blocking effects with those using async effects in NodeJS - if one library has a higher-order function that expects a non-async function and you have functionality which is provided to you as async, it can be very difficult to plumb them together! And if it's the other way around, it can be quite the performance killer (think how much faster better-sqlite3 is than alternatives). Zig's approach eliminates this problem, AFAICT.
If I had to choose between having to pass through an effect handler like `io` or write `async` everywere, the former seems like a better use of my time. It's explicit, but that can be good.
It also fits Zig well with the allocator. Code can expect an allocator or perhaps an allocator and `io`, or perhaps neither. It's analogous to Rust code that is core vs alloc/nostd vs std.
I am slightly amused that a "C-but-better" language is going to have an `io` passed through non-pure functions much like Haskell. It's that idea combined with Rust's pluggable async runtimes (and stackless concurrency) combined with Roc's "platforms" - but for systems programmers. Quite amazing.
ivanjermakov 17 days ago [-]
> much like Haskell
I have not mentioned this in the post, but it's surprising how monadic I/O operations are, including async/await approach
And it is certainly in FP land to pass I/O as parameter, because everything is a function
koiueo 17 days ago [-]
If we apply your logic, then every mainstream language has the coloring problem with not just two, but a multitude of colors.
// OMG we can't call this without passing the service
// This function is people-colored
public Person findPersonByName(PeopleService service, String name) {
// OMG we can't find without the service
service.find(name)
}
EDIT: formatting
vips7L 17 days ago [-]
This is off topic. But I really hate “services”. They always are just some random collection of functions and aren’t a real object at all.
koiueo 17 days ago [-]
I think I know what you are talking about. In a typical three-tier enterprise application there's no clear separation of concerns between "services" and DAO/Repo...
And this is due to how most persistence libraries are designed: they eagerly execute side-effects, this breaks composability due to poor transaction control, so you end up leaking a lot of your business logic into your supposedly persistence layer.
Take a look at Scala's doobie[1]. Any doobie operation returns a `ConnectionIO`, which is only a description of an operation (free monad). With a proper doobie usage the DAO layer is an algebra of possible persistence-related operations, and the service layer implements business logic by combining primitive ConnectionIOs and interpreting them with full control of transaction boundaries.
I’m sorry but I just don’t enjoy functional Scala.
koiueo 17 days ago [-]
Paraphrasing someone else: then keep enjoying the disfunctional one :-P
vips7L 17 days ago [-]
Haha you got a good chuckle out of me on that one.
dwattttt 17 days ago [-]
Function colouring created a lot of angst when it first came about, particularly because of the difficulties of calling a function of one colour from another. Whether that was possible, what it actually meant, wasn't really well defined.
As other comments have said, there's nothing special about "colouring"; sync/async functions are a case where those above problems are tough, but simpler versions of the problem are everywhere and we don't freak out about them e.g. call a fallible function from an infallible function.
It really all turns on how easy it is to ultimately make the call to the other "function" colour. In Zig's case, if its easy to get an Io in a function that didn't take an Io, it's a non-issue. Likewise for the "fallible function call from infallible function": if it fails, do something that doesn't result in the infallible function failing (do something else? Terminate? Anything will do).
jmull 17 days ago [-]
With Zig I/O you can call i/o functions from non-i/o functions and non-i/o functions from i/o functions.
In the analogy of “What color is your function”, you can call blue functions from red functions and red functions from blue functions.
The pernicious viral nature of function coloring doesn’t apply.
ivanjermakov 17 days ago [-]
> With Zig I/O you can call i/o functions from non-i/o functions and non-i/o functions from i/o functions
Yes, but you would either create a new std.Io just for that, or use a global/semi-global std.Io from somwhere else
jmull 17 days ago [-]
Yes… like every other argument one function provides to another that it doesn’t pass directly through from one of its own arguments.
The function color analogy captures a pernicious viral dynamic with async where it allows only one-way composition. That just doesn’t exist with zig i/o.
AYBABTME 17 days ago [-]
One way or another, I like how this is implemented. Explicitely passing dependencies like this, with a tight syntax, makes things easy to understand and write.
I haven't written any Zig but these demonstrations give me strong vibes of how I felt when I picked up Go more than 10y ago.
ezst 17 days ago [-]
I mean, isn't it how it's always been, up until JS/python "single threaded" runtimes became what most people had ever encountered?
pjmlp 17 days ago [-]
Single threaded is a bit old, it wasn't until recently that no one had multiple cores outside HPC.
I believe Zig's new I/O is neither colored nor not colored, instead the design is orthogonal to that. You won't be able to make code magically async by just swapping the implementation of Io, instead you'll need some kind of runtime that allows functions to wait without blocking OS threads. Depending on how this is implemented (if it's possible at all) you'll have colored functions or not.
Surac 17 days ago [-]
i never got warm with this async/await style of calling things. i much more prefer using threads/fibers/protothreads instead. Sure you have to be more expicit on creating and controlling these and also passing around data may get more complex, but at least they do not gloss over the underlying problems async and await tries to cover up
17 days ago [-]
Rendered at 02:44:22 GMT+0000 (UTC) with Wasmer Edge.
The virality of async await is that once you mark a function async, then you can only call it from another async function, which forces you to mark more functions async, which in turn means that if you want to use blocking I/O APIs then you just can't because it's incompatible with your execution model because by daring to express asynchronicity of operations, you were forcefully opted into stackless coroutines.
That's what Zig solves, and that's what is real function coloring. People have written reimplementations of the same libraries multiple times because of it.
https://github.com/redis/redis-py https://github.com/jonathanslenders/asyncio-redis
Just as an example. Note also how, coincidentally, this duplication of effort resulted in asyncio-redis being semi-abandoned and looking for maintainers. And you have to have both libraries because the asyncio one can't do blocking, and vice versa the other one can't do async.
Would you write two instances of essentially the same library just because one is missing an argument that gives it access to an `Io` interface? No, because you would just pass that extra argument around and nothing else would have to change.
In the sense of effect/capability typing, I think the answer is yes.
"Coloring" isn't magical, it's just a way to describe effects. Those effect can be described by keywords (`async` in JS, `throws` in Java, etc.) or special token parameters/types (what Zig does), but the consequences are the same: the effect propagates to the caller, and the caller becomes responsible for dealing with it.
Note: async fn in Rust is also just a fn (in Rust's case, a fn that returns a future). It turns out that returning a future, or receiving as parameter a reference to the runtime, are equivalent in some sense
It's in a different league comparing to async-await abomination. For one, you can store parameter on a struct thus working around the "virality" on a call site.
You have the following call stack (top to bottom): a -> b -> c -> d
Function "d" does IO, so a, b and c need to pass that IO down. It's the same as with async in terms of virality.
If I understood correctly, the main benefit here is that you don't have to dance around "can I call this function or is it going to block async runtime reactor?"
I would pick async/await style over this because (I think) extra function argument is higher visual load than "async fn".
On a side note: the entire "coloring" metaphor is extremely unfortunate and confusing IMO. It's somewhere at the top of my list (right beside the shroedinger's cat) of things which could've been explained better without dumbing them down.
Function coloring is indeed confusing, which is why I’ve been trying to ground this on effects instead. Effects are IMO easier to understand, and this is a straightforward example of effect typing.
Some people are so focused on categorical thinking that they are missing the forest for the trees.
The colors are a means of describing an observed outcome -- in Node's case, callback hell, in Rust's, 4 different standard libraries. Whatever it may be, the point is not that there are colors, it's the impact on there being colors.
> But there is a catch: with this new I/O approach it is impossible to write to a file without std.Io!
This sentence just makes me laugh, like it's some kind of "gotcha". It is the ENTIRE BASIS of the design!
It's more that discussion about most of them becomes meaningless, because they're trivial. We only care when it's hard to swap between "colours", so e.g. making it easy to call an Io function from a non-Io function "removes" the colouring problem.
Exactly. In golang (which is also a cooperatively multithreaded runtime if I understand correctly), calling a function that needs IO does not infect the callers type signature.
In async rust, and in "async param" zig, it does.
Of course, if these async contexts were first class citizens then you've basically just reinvented delimited continuations, and that introduces complications that compiler writers want to avoid, which is why async/await are second citizens.
Because async/await are not values, they are ways to run/structure your code. That's why they are so infectious. If you don't want the division the only solution is to make everything of one kind. Languages like C make everything sync/blocking, while languages like Go make everything async.
They are computations that produce values. Computations can be reified as values. What do you think functions and threads are?
As I described above, "delimited continuations" are values that subsume async/await and many other kinds of effects. You can handle async/await like any other value if they were reified as delimited continuations, but this makes the compiler writer's life much more difficult.
Supporting delimited continuations force some specific ways of performing computations. They are akin to making everything async, which proves my point: you have to make everything of one kind in order to solve the problem.
You can use ordinary direct style compilation, but all references to stack values simply have to be relative offsets, then a simple implementation of shift/reset is just capturing context and copying stack fragments, which you can do using setjmp/longjmp in C (although there are better ways [1]).
This is not akin to making everything async, nor is everything "of one kind", whatever that means. A delimited continuation is very much its own kind of thing, distinct from other values, and doesn't have to influence the function call/return semantics unless you're targeting a less flexible runtime like the JVM.
[1] https://github.com/koka-lang/libhandle
Then such references are no longer pointers, and in order to have a generic reference (that can point to both stack memory and heap memory) you have to store additional data (which one of them it points to) and conditionally use the correct one any time you access it. This is a very invasive change to the memory model.
> then a simple implementation of shift/reset is just capturing context and copying stack fragments, which you can do using setjmp/longjmp in C
That sounds like you're just reinventing green threads then, which basically forces everything to be async once you start noticing its issues.
> https://github.com/koka-lang/libhandle
This returns a 404 error for me.
> This is not akin to making everything async, nor is everything "of one kind", whatever that means.
Sure, if everything has to be able to "wait" for the result of a continuation then you're forcing async support on everything.
If instead you implement continuations as some kind of monads that users have to return from their functions then you got function coloring because you can't normally wait for the result of a continuation from a function that does not return a continuation.
Dropped the r somehow:
https://github.com/koka-lang/libhandler
> Sure, if everything has to be able to "wait" for the result of a continuation then you're forcing async support on everything.
If "forcing async support" doesn't mean that code has to change, or any other code in the call chain, then it's just meaningless pedantry.
That's not really the case. All you really need is a way to run async code from a sync function; "keep doing this async thing until it's done" is the primitive you need, and some languages/runtimes offer this.
On a practical note, since Rust doesn't standardise on an async runtime, it would be more accurate to say tokio solved the coloured function problem, for whatever that means. Or any and everyone else that made it easy to call one coloured function from another.
That's not exactly true. Many languages return a curried version of it, which can be executed once you have a runtime.
Function is the smallest unit of logic. And function parameter is the fundamental way to control it.so, in theory, __A function parameter is the smallest possible design choice you have to control async vs sync__
There is no way to reduce it further. As you need to encode this somehow, otherwise it will be implicit and very hard to debug.
functions can compose. Function parameters can be composed. Making this a solid design choice.
Zig has somehow achieved this feat of decomposing async/sync to a single parameter. This deserves an ovation Other langs should take note.
So we are just going to forget Go exists?
as per my understanding, golang has it implicit. Like there is no way for a developer to tell if code will run under async context or not.
> Semantically, passing std.Io to every function is no different from making every Node.js function async and returning a promise. Zig shifts function coloring from blocking/non-blocking choice to io/non-io.
> Complains are about how inconvenient it is to work with [blocking and non-blocking function handling] differences
"Extra argument" in this case is not much different from "extra async keyword", because you can only run I/O functions when having std.Io in scope.
Rust calling async function in non-async function:
https://docs.rs/tokio/latest/tokio/runtime/struct.Handle.htm...And calling blocking from non-blocking:
This of course is basically essentially what Zig is doing, except instead of hidden global state it’s parameter passed. This is one area Zig does do better in - I wish Rust would default more to instance state instead of implicit global state.But consider a language that doesn't have block_on, like javascript.
I think we should stick to talking about colouring when there is special calling conventions or syntax, which has the consequence of having to write separate libraries/modules for async code and non-async code.
That is the significant problem we have been seeing with many async implementation, and the one which Zig apparently fully solves.
That's pretty much where we are though. If you have a function that isn't passed an allocator, and now it needs to call a function that does take an allocator, we're in the same place.
Rust's 'async' keyword changes the type of the return value, but you can just write the different return value yourself; it's 'coloured' purely by what it means to be returning a 'Future'.
Likewise if JavaScript had an easy way to get a handle to its Runtime, and a function "block on promise" in its early days, we'd have never had all these "colouring" arguments.
How would you add "block on promise"? Javascript started off without the ability to block on anything, not even I/O because it didn't have I/O. Later when XMLHttpRequeest was added it was purely callback-based.
There are ways to make that API work but introducing the concept of blocking would be a big language change. It's not fixing a flaw, it's trading away some complications for a different pile of complications. There's still a conflict to resolve in language design when you want some functions that are synchronous and don't block and other functions that are asynchronous.
Edit: I guess javascript blocks on alert() but that doesn't seem like what you'd want "block on promise" to do...
If you did so in a Worker, you'd have your "don't block the UI thread" in there too.
Blocking in the sense of blocking I/O or blocking on a system call is not possible in javascript. So again I ask what you have in mind for a "block on promise" call. The ideas that come to my mind all require significant redesigns of the execution model, or copping out by making every function async and treating it like an await.
In particular with workers in mind, I'll put it this way: If you don't allow "block on promise" in the main thread, you generally avoid these issues, but you also don't solve the problem, you still have async versus not async causing many headaches. If you do allow it on the main thread, how do you keep your page functioning? I can imagine resolutions but they all have severe compromises.
Being able to block on an async function call is just as much of a risk; if it returns quick, you live, if it returns slow, you die.
The solution to "main page unresponsive, terminate page?" is to always return promptly to the Runtime, so e.g. make everything async. But JavaScript is a language, and it doesn't just exist on web page main threads. As much as JS's proliferation everywhere else isn't my cup of tea, a blocking call in a JS cli tool is what you want.
EDIT: I guess to be explicit:
> So again I ask what you have in mind for a "block on promise" call
I expect it to have the same guarantees and risks of calling a non-async JS function.
But in a very temporary way. If it takes seconds that's a bug, while if blocking I/O takes seconds that's normal behavior.
And I've never seen a javascript busy loop in non-malicious code.
> I expect it to have the same guarantees and risks of calling a non-async JS function.
The main guarantee is that the function will be done and have all its temporary data cleaned up before your (non-worker) execution switches to other code. And in practice you can expect it to return within .1 seconds.
That guarantee is going to be an issue if you try to introduce some kind of "block on promise". And the timing expectation is meaningful too.
This is not a new pattern, and I think it's a pretty good one (and is arguably more ergonomic and general than syntax-level effects). But it's quintessential function coloring.
It does no such thing. you could pass a function a vtable and the vtable could have one implementation that calls an io stashed in the parent of the vtable, and a different vtable that doesnt and the function calling the vtable would be none the wiser. what is the color of the function that took the vtable?
this is not just academic; it would be for example the basis for mocked integration tests on a database or over the net api call.
To be clear, there's nothing wrong with this; it's just another way to encode capabilities/effects.
> what is the color of the function that took the vtable?
It has the I/O effect.
it does not, because it can take a vtable that does not call i/o
concretely:
Intuitively: if you mark a function as async, it doesn’t stop being async “colored” just because you don’t actually perform any async operations in it. This is the same thing.
[0] But even in that case there's not really coloring because if you provide a single io implementation there won't be different functions, even at the compiled level.
People seem to be really defensive about this, like it's a bad thing. It isn't! It's arguably a significantly cleaner way to handle what people confusingly call "coloring." But that doesn't make it not "coloring," because coloring is about effects and vitality, not about keywords and syntax.
2. here's what i have to say about your idea of what coloring is: That's all fine and good in theory, but in practice it makes no difference, at the user level, or at the compiler level.
> That's all fine and good in theory, but in practice it makes no difference, at the user level, or at the compiler level
Every example given so far shows Io’s virality, so I don’t know how you can assert how it doesn’t make a difference. The entire point of the design appears (reasonably) to be to introduce a token object that conveys an effect, rather than requiring a runtime to intermediate that effect.
Again, none of this is bad. Effect typing is cool. But it is, by definition, a way to color a function.
(Maybe the confusion here stems from the fact that languages like Go appear to have it “both ways” without coloring. Go is able to do that because it has an intrusive runtime that intermediates asynchronous events. Zig can’t do the same thing without making the same compromises as Go vis a vis FFI performance and ABI compatibility.)
Say you have two functions one that does some kind of IO, (say a pair of bidirectional read, write, from stdin and stdout) and another that returns a block of memory of a constant size. Would you say that one is colored, and the other isn't?
I'll also try to make an attempt at the idea I'm trying to figure out too. I genuinely can't predict which side you'll answer for, but I'm assuming you'll say that it's not colored, because while it does do IO, but only directly though the global file descriptors which have no baring on the calling conventions. Which means the decisions and impact about which color this IO function doesn't ever apply to the callers and callees. This I think is the virality you mean where the callers and callees are required to know [something] about the semantics? I probably agree you could call this coloring and defend it... It's interesting that this might be closer to the "accepted" definitions of what any language means when it says this function is colored. But I don't think this understanding most people have about the concept of coloring. Most I see are arguing about the semantic effects, much more than the "accepted" definition. I have the distinct impression that when most say coloring, the most significant implication is when and where you resolve the specific color. Will it be sync or async code, when, where, and who gets to decide.
Or perhaps I'm wrong, and you would say that one function is colored, but the other isnt? Then I'd ask if you can tell it's colored, can you tell me which color?
In both cases I'd argue it's better to embrace the semantic widening of the concept of function coloring, because what I understand from how you describe it, I feel that saying this is coloring is strictly less enlightening about the reality of the code and system together, than calling this function alpha. With zig's new IO interface, I can't tell you what the RGB values are for the color, but I can confidently say, it's alpha channel is 0xff. You might not be able to tell me what color it is, but it without a doubt isn't a pure function, interacts with IO and does have a null byte in the alpha channel.
That is not a consequence of function coloring or syntax, it is a consequence of having multiple ways of performing IO.
- seemingly harmless functions that unexpectedly end up writing four different files to disk.
- Packages that do I/O or start threads when you simply import them.
The issues I've had with function colouring had to do with trying to compose code using (or expecting) blocking effects with those using async effects in NodeJS - if one library has a higher-order function that expects a non-async function and you have functionality which is provided to you as async, it can be very difficult to plumb them together! And if it's the other way around, it can be quite the performance killer (think how much faster better-sqlite3 is than alternatives). Zig's approach eliminates this problem, AFAICT.
If I had to choose between having to pass through an effect handler like `io` or write `async` everywere, the former seems like a better use of my time. It's explicit, but that can be good.
It also fits Zig well with the allocator. Code can expect an allocator or perhaps an allocator and `io`, or perhaps neither. It's analogous to Rust code that is core vs alloc/nostd vs std.
I am slightly amused that a "C-but-better" language is going to have an `io` passed through non-pure functions much like Haskell. It's that idea combined with Rust's pluggable async runtimes (and stackless concurrency) combined with Roc's "platforms" - but for systems programmers. Quite amazing.
I have not mentioned this in the post, but it's surprising how monadic I/O operations are, including async/await approach
And it is certainly in FP land to pass I/O as parameter, because everything is a function
Take a look at Scala's doobie[1]. Any doobie operation returns a `ConnectionIO`, which is only a description of an operation (free monad). With a proper doobie usage the DAO layer is an algebra of possible persistence-related operations, and the service layer implements business logic by combining primitive ConnectionIOs and interpreting them with full control of transaction boundaries.
[1] https://typelevel.org/doobie/index.html
As other comments have said, there's nothing special about "colouring"; sync/async functions are a case where those above problems are tough, but simpler versions of the problem are everywhere and we don't freak out about them e.g. call a fallible function from an infallible function.
It really all turns on how easy it is to ultimately make the call to the other "function" colour. In Zig's case, if its easy to get an Io in a function that didn't take an Io, it's a non-issue. Likewise for the "fallible function call from infallible function": if it fails, do something that doesn't result in the infallible function failing (do something else? Terminate? Anything will do).
In the analogy of “What color is your function”, you can call blue functions from red functions and red functions from blue functions.
The pernicious viral nature of function coloring doesn’t apply.
Yes, but you would either create a new std.Io just for that, or use a global/semi-global std.Io from somwhere else
The function color analogy captures a pernicious viral dynamic with async where it allows only one-way composition. That just doesn’t exist with zig i/o.
I haven't written any Zig but these demonstrations give me strong vibes of how I felt when I picked up Go more than 10y ago.