Zig's incremental builds are DEFINITELY a killer feature. In the short term, I could see why you'd make a switch to get it. But, in the medium term, can we really not expect to see this in Rust in the somewhat near future?
I want to go fast, but I don't want to go fast just to shoot my foot off.
If only somehow we could get Rust's safety with all of Zig's features and Go's runtime without GC...
This is cool and will likely enable some cool tooling.
I don't think a borrow checker is likely to be in that tooling. Borrow checking requires shaping the code, and all the dependencies, into easily analyzable (and at least in rust's version annotated) patterns. You can't borrow check arbitrary code not designed for it without false positives.
That's not sufficient - consider the following pseudocode
x = malloc();
if (opaque_cond()) free(x);
if (other_opaque_cond()) use(x);
Conditions can be opaque and non-analyzable due to rices theorem - in any turing complete language. This code is correct (or at least not memory unsound) if opaque_cond and other_opaque_cond are never both true. Otherwise it isn't.
And functionally compiler analyses of whether conditions hold have to be trivial because using some form of theorem prover to decide of code is correct or not leads to code that is brittle against compiler version changes, and slow compile times. Thus opaque_cond could be as simple as `len == 0` and `other_opaque_cond` could be `len > 0` and it's unlikely you'd want the compiler to realize those are mutually exclusive (at the stage where it accepts programs, obviously during optimization it is very likely to take advantage of this).
Rust solves this by simply rejecting the pattern. Very roughly forcing you to write if opaque_cond() { free(x) } else if other_opaque_cond() { use_x } (or something else where the program structure and not just the logic in the conditions guarantees correctness). Zig simply allows it and leaves it up to the programmer not to make a mistake.
And as onlyrealcuzzo suggests aliases are where this type of analysis (accepting enough programs to be useful but still imposing enough structure you can prove correctness) is really tricky.
It's possible to detect and ban the version of the problematic pattern that I made as short and simple as possible to illustrate the point, sure.
I'm general though, I don't believe it is practical to do so. Not without every library being designed with the checker in mind and annotated to more precisely describe their APIs. Which is why I'm not surprised to see the limitations.md that seems to exclude all the hard cases (aliasing, pointers used as first class values, cross function analysis): https://github.com/ityonemo/clr/blob/main/LIMITATIONS.md#mem...
Obviously if you rewrite the zig world to obey rust like rules and include rust like annotations you can implement a rust like borrow checker, but I don't think that it would still be meaningfully zig. It might be an interesting language worth exploring.
i have in mind a strategy to address all of them. this is a side project, a proof of concept, i have other things going on in my life. i dont chip away at it every week.
you make, without any evidence ("Obviously"), an assertion that "it would look like another language". so far if anything applying zig clr would push a user to write more idiomatically ziggy code, away from idiomatically c-ish code. i dont see why continuing with clr wouldn't go further along that trend. so consider what is "obvious" to you might just be flat out wrong.
> It might be an interesting language worth exploring.
worth how much? youre welcome to sponsor my exploration and put your money where your mouth is:
I wish them the best of luck, but I don't expect them to succeed, and unspecified plans don't constitute a demonstration of feasibility. Plenty of people have made plans to solve unsolvable problems in the past, me included. I strongly suspect that that is the case here (or that they're willing to iterate away from zig).
The evidence is the amount that rust had to iterate on the underlying language to make the borrow checker work well. Something it had the freedom to do since it was co-designing the language and linter.
Edit: Didn't realize this was your project (responded before you added the donation link) - I would have worded my response slightly differently but my opinion is unchanged. Seriously mean it with the best of luck getting this to work.
do you want a direct description of the plan for async/alias work? hash the globally accessed gid list and if any operation changes retrigger analysis of all functions in the same execution block with the new layout.
i have not had to change the language to accomodate uaf/df/leak analysis (which is not easy). i see no reason why i should have to to get async/alias to work.
unlike rust, as of zig 0.16 async conceptually is abstracted to a userland interface in the stdlib (versus a keyword), which means that its easier to detect, and easier to work into the existing clr system. that means it's either "not doable at all" (unlikely) or is unlikely to need any back and forths to be done with the language
i dont understand the downvotes here. the point of any safety checker is to flag and ban potentially unsafe code, and force the author to rewrite with existing language patterns that guarantee the desired safety parameters.
in this case, zig has a first class nullable syntax that the checker can use ti guarantee correctness for, so a checker can deterministically sidestep this turing completeness issue, by squeezing indeterminate code into the knowably safer language idiom.
1. rust didnt invent affine ownership. 2. It's possible to bolt on to other languages (see ada). zig in particular is easy (disclaimer: i think, i haven't implemented it yet)
> zig in particular is easy (disclaimer: i think, i haven't implemented it yet)
I guess it's "easy" compared to other languages - but if you think it's "easy", we have different definitions of "easy".
You could implement it, but it would look like efforts in Rust to get SPARK-like safety, and SPARK itself. It will essentially be a different language.
You will not be able to work seamlessly with any regular Zig code. That may or may not be a problem if you're willing to assume you can just use it all unsafely and it works enough that things are fine.
That's somewhat analogous to unsafe Rust. The difference with unsafe Rust is... That's a very small fraction of what you're using, not the vast majority of what you use.
When you use a Rust crate - you generally do not expect that it could have infinite race conditions. It may have some unsafe code, but that should be the exception, not the norm.
By all means, please build it. I'd consider using it [=
> You will not be able to work seamlessly with any regular Zig code. That may or may not be a problem if you're willing to assume you can just use it all unsafely and it works enough that things are fine.
that may be true, but it seems "not". to date, all of the patterns in zig-clr nudge you towards idiomatic zig and not away from it. i run tests on not-my-code (an unaltered version of an existing zig project -- you can see it's vendored in the "vendored/validate"), and it passes. still working through forestmq.
and I'm planning a mechanism to let you reach into a function and "oracle" its safety parameters, probably most useful when someone else has written code that you know is ok but you cant tell them "hey make this work to pass my linter"
Also remember that zig compiles as a single compilation unit so even if you draw in zig dependencies, unless they are hidden behind a .so, zig-clr will analyze the dependency code too.
By all means - reach out when it's ready and I'll give it a test.
I'm highly skeptical you can get it to work. If it was easy and optional and non-invasive and actually worked - the Zig team would almost certainly build it.
But, even if it just mostly works - that would still be very useful if it's non invasive.
2. The zig team is parsimonious about what they do and don't build. For example, they did not work on the language server, rather punting it to the community.
3. That you can do this with the zig compiler is a happy accident. The team was not designing towards this possibility, it's not really a part of the core zig ethos (and that's fine).
4. You certainly cannot do things exactly the way that rust does it without changing the language; because the rust conversation has sucked all of the air out of the discourse around safety, you really do have to make a paradigm shift away from "tagged ownership" to "data dependency tracking with ownership agnosticism" to do it with zig.
You either do it with Linear Types, Affine Types, or you have holes...
Or you've invented a novel new system and should be publishing white papers immediately.
You can do virtually everything Zig comptime does in Brute Force in C++23. If this was possible - completely, it would already be done for C++. You need to point out exactly what you can do in Zig that you can't in C that somehow magically makes this completely possible and bulletproof in Zig - otherwise, you have a "concept of a plan".
The reason Rust sucked the air out of the room is that people are much more interested in systems without holes than systems that mostly work, or partially solve small parts of the problem and/or rely on the developer getting it right.
its refinement types which are a "superset of affine types" iiuc (you can construct a refinement type system that can represent affine types, plus more).
there is nothing special about zig that makes it fundamentally possible in zig and impossible in C.
the difference is that compiler architecture, some language decisions, well-designed stdlib, not having to contend with past decisions of a committee that overweighted backwards compatibility, take it from "jesus christ this is too much of a pain in the ass to bother" to "oh theres actually a path forward for one person and an llm".
As a concrete example. In C there is no "anointed allocator", sure, most people use malloc/free, but a broadly useful analyzer needs to be able to contend with any of a billion different patterns, and "oops I forgot jemalloc's mallocx" is not really okay (plus you gotta deal with weird things like errno, etc, and maybe some program uses jemalloc's free in some places and stdlib free in other places, how do you track that). In zig, sure, you could build a function that allocates off the stdlib path, but it's not unreasonable to "ban" that and force people to use the stdlib interface, and most dependencies will be compliant. Hopefully you can see how this "jesus christ i don't want to deal with all that" vs "a clean path forward".
intermediate representation. attempting to analyze zig code directly would be too hard (especially with comptime). on the way to the compiler backend, the compiler builds a simplified representation that only has "actually existing functions" and is very straightforward, e.g.
function 10112:
0: argument 0
1: argument 1
2: argument 2
3: add 0, 1
4: store 2
5: call function 1342, (2, 4)
6: return 5
you can see how building a data dependency graph from this would be easy.
Instead of waiting for faster compiler in Rust, how about from the other direction, adding some kind of borrow checker to Zig? That sounds more within reach and practically achievable, possibly even in userland.
It's impossible to add a borrow checker to any existing language.
The reason Rust has a working borrow checker is because every part of the language from structs, enum, traits, generics and all the way to the syntax itself has been designed to support lifetimes and borrow checking.
It's is not something you can just tack on to an existing language without fundamentally changing it.
I wouldn't say it's impossible, rather un-ergonomic. TypeScript can add type information to ordinary JavaScript code via JSDoc comments; the result can both be executed as ordinary JavaScript as-is and type-checked with TypeScript. But it's a huge pain to try to write (and maintain) everything that way, it was supported as a hack to help migrate legacy codebases. You could probably take a similar "the lifetimes are embedded in comments" approach with other languages, and the result would be similarly un-ergonomic.
no. You don't need private fields. All you have to do is analyze the code, harness the compiler to generate a time-dependent data dependency graph, and map allocation/frees/uses, if you can 'color' branches where data are shared you can also track and check to see there isn't an aliasing violation too.
it is easy to patch the zig compiler to enable this this (export the code graph; about 50 LOC). The analysis is much much harder to get right.
it is possible to do in-code annotations in zig, if you're clever. you can get pretty far without them too.
as an example, you can check for double free without ownership tagging, by being agnostic about who should free, and flagging if two nondisjoint code paths attempt to free the same allocation.
It is only feasible to do this if the whole of the codebase idea designed to allow it, and it's still going to blow up in odd ways of you don't have a way to describe lifetimes in your interfaces. The magic of rust's design is that it turns this memory tracking into a local problem, such that you can design an interface and be sure that every use case is safe and verifiably so.
You may have missed the point here. You could add a comment to the struct field that marks the field as private, and build a TypeScript/JSDoc analogue that analyzes all accesses to the field and fails if it finds accesses from functions that aren't part of the struct that owns the field. You don't even need a comment on the field - you could copy Go's convention, add a comment to the struct definition marking it as "follows Go convention", and then fail any access from outside the struct to a field that starts with a lower-case character.
It doesn't prevent you from ignoring that tool and writing Zig code that imports the struct and accesses the field. It is, of course, not part of the Zig language itself. But if you adopted a tool like that, it would be your responsibility to run it across-the-board and pay attention to the results - same as how it is your responsibility to pay attention to the results if you added those JSDoc comments.
I have picked private fields as an example of feature that is needed because it is very simple. You're right that you can build an analyzer (with additional code annotations) to support that, but it's only one example.
Take another example: unsafe traits. They are fundamental to some safety encapsulations, most famously concurrency (`Send`/`Sync`). Here you cannot just build an analyzer to mark something unsafe, because Zig has no traits, its generics are duck-typed.
You can, of course, add traits. But at this point you're essentially creating your own language that compiles to Zig, with all problems this entails (e.g. bad ecosystem support). It's also hard to claim that Zig can be memory safe then.
> You can, of course, add traits. But at this point you're essentially creating your own language that compiles to Zig
I think herein lies the rub. What's the difference between a static analysis tool and an actual separate language that transpiles to the original? Hypothetically - again, very un-ergonomically - you could add traits to Zig code in comments, or in example-traits.typezig files that would be skipped by the Zig compiler (like how *.d.ts files are skipped). How much of a language is writing code in a particular syntax, versus how much of a language is writing code that will pass a tool "building" it, versus how much of a language is about the final compiled output that you get from the tool? All static analysis tools that support line-level exceptions are, essentially, programmed by comments, with their own language (typically highly simplified compared to a "full" programming language), that affect whether or not the "language" passes or not. What Typescript/JSDoc shows is that, actually, much more complicated tooling can be built with this programming-by-comments model than had been done before (to my knowledge), and thus even more powerful still tooling could be built with that model.
Of course there's a difference between static analysis and a language that transpiles. But perhaps it's more a question of degree than a simple binary classification.
empirically untrue. several projects exist that bolt on extra safety to unsafe languages or unsafe parts of language. SeL4 for C, MIRI for rust unsafe. i guess ada/spark for ada too, is the OG, spark being added to ada 4 years after its first release
Hardening is definitely possible, we've had sanitizers in C/C++ for a long time. It's not full memory safety though. Miri is the same.
SeL4C is formal verification, and while it can prove memory safety (and much more) it is much more difficult, to the point that you're basically programming in a different language.
Ada/SPARK is your best example, and also the example I know the least of, so I won't comment on.
SPARK omits some features of Ada, so it would only reinforce the sentiment that bolting on verifiability after-the-fact is difficult. Expressivity is generally the antithesis of static analysis, and it's very easy and tempting to make a language that is accidentally too expressive to support a given analysis without being required to make breaking changes to reduce expressivity.
A language is more expressive when it allows more programs and less expressive when it allows fewer programs. I don't know zig-clr, but if it rejects programs that Zig accepts (for example, by rejecting the aforementioned ambiguous pointers), then it is less expressive, not more (keeping in mind that being less expressive is not a pejorative).
no thats not the definition of more expressive. more expressive means the language can encode more programmer intent without making a dog's breakfast of the code.
Now we're just having a semantic argument over the word "expressiveness", which is not especially interesting; see https://chrispenner.ca/posts/expressiveness-spectrum . My argument above remains true regardless of the terms you choose to use.
> We focus on three mode axes: affinity, uniqueness and locality. Modes are fully backwards compatible with existing OCaml code and can be completely inferred. Our work makes manual memory management in OCaml safe and convenient and charts a path towards bringing the benefits of Rust to OCaml.
> It's impossible to add a borrow checker to any existing language.
Why do you say that. Have you tried and failed? It seems to be possible to add a borrow checker to zig, just as you can add MIRI to rust to get extra safety in unsafe blocks.
> how about from the other direction, adding some kind of borrow checker to Zig? That sounds more within reach and practically achievable, possibly even in userland.
It's doable, and as static analysis. see sibling comment.
the architecture doesn't make sense. MIRI doesn't perform static analysis on MIR. It is, as the name says, an interpreter. The borrow checker is entirely different from miri.
Rust's borrow checker requires lifetime annotations. Zig code doesn't contain any such annotations. How does your design handle this?
1. it is possible to do code annotations in zig even though i havent implemented it in this iteration of clr (the first poc demonstrated this). i want to see how far i can get without them.
2. let's take double free (easiest to explain).
you dont have to tag ownership, you can be agnostic about who should free, and merely report if two nondisjoint code paths attempt to free the same memory.
Are compile times that big of a deal? I haven't used Rust a ton, but the few times I have it seemed like the bulk of the compile time was a one off compiling the crates, and then compiling your own code was super fast.
I feel like I'd massively prefer to end up with a binary free of memory exploits than shaving some time off compile.
I'm not sure it would ever make sense to be. That makes the assumption tons of allocations get made that don't live long, which was(maybe is still?) more common in some languages. Go is more aggressive about not heap allocating, and has tools to help you avoid them.
You seem to be really hostile for no apparent reason. There are plenty of reasons to be generational, there are lots of workloads that the current implementation might fall flat if it wasn't.
> n practice, Go can typically outperform Rust in throughput (using more memory), despite having a mountain of disadvantages against it in theory
This is a huge claim that disagrees with both my real-world experience and everything I've seen from artificial comparisons.
Every high performance Go system I've worked on has quickly reached the point where we're optimizing memory management and doing things that would have been explicit in a non-GC language like Rust anyway.
The Go runtime is amazingly optimized, but it comes with overhead over doing the same work directly in a lower level language.
Go has few issues with performance (lack of in-line union types, interface overuse, inefficient idioms reg. collections, some missed optimizations) but its seems plausible for a idiomatic Go program to outperform an idiomatic rust program in some situations.
> It's literally the most sophisticated scheduling engine in the world.
That seems unlikely regardless of how good it is. This is a domain where state-of-the-art research is not in the public literature. Scheduling is an AI-complete problem.
I think this is interesting and warrants explanation. There are cases where a GC can be faster (sort of, Arenas get you most of the gains) but "the most sophisticated scheduling engine in the world" should be easy to at least partially support.
Erlang's scheduler is not sophisticated, which is what makes it AWESOME.
but yeah. i would be surprised if the JVM's scheduler is not more sophisticated than go's if for no other reason than it has way more knobs you can tune. you know they put that knob in there because someone (probably Google cough cough) asked for it
I want to go fast, but I don't want to go fast just to shoot my foot off.
If only somehow we could get Rust's safety with all of Zig's features and Go's runtime without GC...
That's what I'm working on building [=