proposal: ErrorOr type wrapping (T, error)

[jabolopes]

Hi everyone,

Following @robpike 's suggestion, I'd like to create a proposal for a new way to think about errors given that Go supports generics now.

This is similar in spirit the plan to think about APIs in the presence of generics but for error handling specifically.

Preliminary notes:

• This is my first proposal, so apologies if I used the wrong format or if it's too long.
• All names used in this proposal, such as, ErrorOr, Val(), Err(), etc, are meant for illustration purposes only and any new names are welcome!

Objective

Rethink Go's error handling mechanism in the presence of generics to overcome the following problems:

• (T, error) allows for 4 possible states but the majority of APIs only use 2 states and the other 2 states could be considered incorrect.
• Error-agnostic generic APIs: Generic parameters like T and generic functions like func()T cannot be instantiated with (T, error) or func()(T, error) respectively. This means API developers must artificially bake in errors in their APIs to allow callers to pass errors or pass error-returning functions.
• Errors / values in channels, maps, slices
• Naming problems with err.
• Inconsistent use of (T, error)

The following are out of scope for this proposal:

• Solve the context.Context problem for APIs.
• Treat (x, y, z, ...) argument like a tuple.
• Treat (T, error) return like a tuple (see alternatives considered).
• Currying.

Background

The 4 states of (T, error)

The current mechanism to return errors is (T, error) which being a product type means it allows for 4 possible states, i.e., any combination of a proper / improper value, and an error or nil. But the majority of APIs only care about either returning a value or an error, they don't care about the other 2 states, which could even be considered incorrect. Notable exceptions to this rule are the io.Reader.Read API (more on this later).

Because most APIs care only about 2 states, there is an opportunity to leverage the type system and generics to eliminate the 2 undesirable states, by introducing a new type ErrorOr that can only represent a value or an error, and doesn't allow for any other states. This would be used in new generics aware APIs.

Error-agnostic generic APIs

One such API is the singleflight.Do, which could be reimagined with generics as:

func (g *Group[K, V]) Do(key K, fn func()V) (v V, shared bool)

The Do API is error-agnostic, i.e., in principle it could accept in fn either an error-returning function or not because it doesn't do anything with the error. But in current generics, it's not possible to pass an error-returning function to func()V since V cannot be instantiated with multiple-return values or (T, error).

This applies not only to singleflight but to any API that accepts a generic argument T or a generic function func()T, does not inspect that argument, but simply returns it later.

To overcome this limitation, API developers must artificially bake in errors in their APIs, for example:

func (g *Group[K, V]) Do(key Key, fn func()(V, error)) (v V, err error, shared bool)

or:

func (g *Group[K, V]) Do(key Key, fn func()V) (v V, shared bool)
func (g *Group[K, V]) DoErr(key Key, fn func()(V, error)) (v V, err error, shared bool)

both of which are suboptimal because even though this API is error-agnostic, this property was lost and it's not longer reflected in its type or enforced by the typesystem.

With ErrorOr, the burden of deciding on whether the API needs error handling or not goes away, and instead API callers have a free choice on whether to call Do with an error-returning function (e.g., func()ErrorOr[T]) or not (e.g., func()T).

Also, with ErrorOr the fact that singleflight.Do is error-agnostic remains captured in the type and enforced by the typesystem. This is a property that is worth retaining.

Another (small) benefit of the ErrorOr in this case is that the singleflight.Group implementation would be simpler because it would need to store only 1 value per call (i.e., the ErrorOr[T]) instead of storing 2 values per call (i.e., the T and the error).

Errors / values in channels, maps, slices

When I use channels to implement pipelines or glue computations together that can either return a value or fail with an error, I found myself having to define a new struct type to encapsulate the value / error to use in the channel, e.g., chan IntOrError (pre-generics).

The ErrorOr would also be useful for channels, e.g, chan ErrorOr[int] so we don't need to redefine new struct types for this use case like ValueOrError (or a generic equivalent of that) because we can reuse ErrorOr instead.

We could also store ErrorOr[T] in collections such as maps, slices, etc. For example, spawn 10 goroutines and have each store their result in []ErrorOr[int]. Another example, implement a cache with positive and negative caching, e.g., map[K]ErrorOr[V].

Naming problem with err

When we use the pattern (T, error), we need to define the err many times:

x, err := myfunc1()
if err != nil { return ... }
y, err := myfunc2()
if err != nil { return ... }

In some cases, there are no new names on the left side of := so we need a few tricks there, either by writing var err error:

x := ...
y := ...
var err error
x, err = myfunc1()
...
y, err = myfunc2()

or by defining different names for the error:

x := ...
y := ...
x, err1 := myfunc1()
y, err2 := myfunc2()

With ErrorOr we don't need tricks:

x := myfunc1()
if x.Err() != nil { return ... }
y := myfunc2()
if y.Err() != nil { return ... }
return x.Val() + y.Val(), nil

Inconsistent use of (T, error)

There are APIs that are exception to 4 states of (T, error) rule, such as, the io.Reader.Read. But they are the minority.

The documentation for this interface method requires 4 paragraphs just to explain that this API can actually return a value and an error at the same time. If it were common practice in Go to return a value and an error at the same time, it would not be necessary 4 paragraphs of documentation to explain this notable exception.

Furthermore, the fact that it requires such as careful explanation is evidence in itself that this is a pitfall for API callers. And reading through the details it sounds very error prone and confusing.

A new type like ErrorOr could also be useful for a new API like Read(...)ErrorOr[int] because it would mean more consistency regarding error handling across APIs, we could also remove those 4 paragraphs of documentation, less pitfalls for developers, and also better type safety. API callers would need to either handle the value or the error, and there would be no ambiguity in that.

Design ideas

One possible definition of ErrorOr is the following:

package erroror

import ...

type ErrorOr[T any] struct {
	value T
	err   error
}

func New[T any](value T) ErrorOr[T]     { return ErrorOr[T]{value: value} }
func Error[T any](err error) ErrorOr[T] { return ErrorOr[T]{err: err} }
func (e ErrorOr[T]) Val() T             { return e.value }
func (e ErrorOr[T]) Err() error         { return e.err }

The New and Error constructors only allow for either a value or error, they don't allow for both an error and a proper value.

Related work

The ErrorOr has different names in other languages but in essence it's the same idea:

• C++ ABSL absl::StatusOr
• Haskell's Data.Either
• Ocaml Either
• F# Result

Known issues

new(ErrorOr[T]) can create a state that is neither error not a proper value. Perhaps we would need special tooling or compiler support to prevent or produce a warning if a developer wrote this code, since it would be desirable to always use either erroror.New or erroror.Error.

Future ideas

This is out of scope for this proposal but I think it's an interesting, possible future extension of the ErrorOr idea, that can help with Go exceptions without actually requiring exceptions.

Let's say we have a new operator (I will use <- but other syntactic choices are possible), then we could do the following:

func MyFunc(...) ErrorOr[T] ...

func OtherFunc(...) ErrorOr[T] {
  x := <- MyFunc(...)
  ...
}

func OtherFunc2(...) ErrorOr[T] {
  x := MyFunc(...)
  if x.Err() != nil {
    return x
  }
  ...
}

In this example, OtherFunc and OtherFunc2 are equivalent but in OtherFunc we use this new operator to avoid having to explicitly write the error handling code. The new operator <- does the automatic error handling for us by checking if the ErrorOr contains an error and if so, return that error to the caller. This may be an alternative to introducing exceptions without actually requiring full support for exceptions.

Alternatives considered

An alternative to ErrorOr would be to treat (T, error) like a tuple type so that a type like func()(T, error) could be used to instantiate generic functions like func()T. But this would be a much bigger change than ErrorOr with far reaching implications for the language. The ErrorOr is just a new type.

More resources

Initial discussion was #48287 (comment)

[Merovius]

Error-agnostic generic APIs

I think it's wroth pointing out, that this doesn't need an stdlib ErrorOr type to happen. You can write those APIs and let the user decide to just use struct{ T; error } directly, write their own ErrorOr type, or use one from a third party library. A package to do that only needs a handful of lines of code and would allow the community to vote with their feet.

Errors / values in channels, maps, slices

Again, there is no need for this to live in the stdlib. Note that even today, it's not uncommon to just define a custom type Data struct { A T1; B T2 }, if you need to send multiple values over a channel. And struct { Val T; Err error } doesn't seem particularly special here - or above, for that matter.

There are APIs that are exception to 4 states of (T, error) rule, such as, the io.Reader.Read.

I think an a little bit more instructive example is io.Writer.Write. Short writes are normal, need to be signalled somehow and Write (as opposed to Read) can't be safely retried to get the error on a second call.

You treat this "exception" as bad, say that ErrorOr would allow us to get rid of it. But we could have already decide just not to have it - we could've just specified that Read and Write must not return an error with a non-zero number of bytes read. The reasons we didn't have nothing to do with the absence of ErrorOr, really. It's that these APIs need to be able to return both.

But really, that's besides the point. Because I think if anything ErrorOr should be tried out via third party libraries first anyways, which will, either way, leave us with APIs doing both (T, error) and ErrorOr, so APIs which need it (and developers who want it) can just use the existing mechanism anyways.

This is out of scope for this proposal but I think it's an interesting, possible future extension of the ErrorOr idea, that can help with Go exceptions without actually requiring exceptions.

I think what you describe here is essentially the try proposal, which a) was retracted because people didn't like it and b) shows that this doesn't really need ErrorOr to work - it works just as well with the existing multiple return values.

[seankhliao]

This doesn't look like it enables any shorter, safer, or clearer form of error handling, and instead just adds an extra layer of indirection. If done in the standard library, this would be a significant churn of what is considered idiomatic for what is arguably little benefit, and it can't even handle the common usecase of multiple return values, resulting in even more inconsistency.

[jabolopes]

@Merovius

I think it's worth pointing out, that this doesn't need an stdlib

I agree. It doesn't need to be in the stdlib. The only reason I mentioned the stdlib is because there might be error-returning functions in the stdlib that would like to use ErrorOr. In this case, I doubt very much that the stdlib maintainers would introduce a dependency on a third party package. But running this as a trial from a third-party package or in the experiments (exp) package would also be a good approach.

It's that these APIs need to be able to return both.

I agree with you but I think a little more needs to be said. The io.Reader.Read documentation says:

"""(...) a Reader returning a non-zero number of bytes at the end of the input stream may return either err == EOF or err == nil."""

So, if we consider the following case:

• Read returns some bytes, nil
• Stream reaches EOF
• Read returns 0, io.EOF

This is equivalent to the ErrorOr[int] case. So the behaviour I proposed is already in Read, but it just happens to be one of the many cases that Read returns. So I was proposing narrowing it down to this one case.

But at the same time, if I think really long term, I imagine that at some point the ErrorOr could become the standard way of reporting errors in Go. This means that the majority of APIs would be using ErrorOr and only a minority of APIs (like io.Writer.Write) would be using (T, error).

At this point, the (T, error) would no longer be mistaken with ErrorOr[T] semantics because it would only be used by functions that actually need to return 2 values (like you said) since all the other functions that either return a value or an error would be using ErrorOr. At that point, I wouldn't want to change interfaces like io.Writer.Write at all, and perhaps even io.Reader.Read, because the ambiguity is gone.

try proposal (...) people didn't like it

Can you point me to the feedback for the try proposal so that I can evaluate whether the feedback applies to the ErrorOr proposal or not? Thanks!

@seankhliao

This doesn't look like it enables any shorter, safer, or clearer

The following is shorter (less chars), safer and clearer (it's either error or value, no ambiguity):

func (...) Error[int] {

than:

func (...) (int, error) {

This is shorter and clearer (less variables):

x := myfunc1()

than:

x, err := myfunc1()

This is shorter (less args / return values) and clearer (API type shows it's error-agnostic):

func (g *Group[K, V]) Do(key K, fn func()V) (v V, shared bool)

than:

func (g *Group[K, V]) Do(key Key, fn func()(V, error)) (v V, err error, shared bool)

This is shorter and clearer:

chan ErrorOr[int]

than:

type IntOrError {
  v int
  err error
}
chan IntOrError

idiomatic

The question of idiomacy is very debatable because the current error idiom was established in a pre-generics era. Now, with generics we want to rethink what the idiom should look like. That's the whole point of issues like this one and also #48287.

If we don't allow the idiom to change ever, then we will never be able to use generics to their full potential. The whole point of a feature like generics is so that we can improve the language and its libraries across and reach a new level of idiom.

it can't even handle the common usecase of multiple return values

I think it can, e.g., ErrorOr[struct{int; float32}]. And that's just one possibility. There are many others.

Also, if Go ever supports tuples, we can also combine ErrorOr with tuples, e.g., ErrorOr[(int, int)], (assuming (x, y) is the tuple syntax).

[seankhliao]

The error handling proposals have been focused on checking the error part, not the declarations. Saving a few characters on the function declaration is missing the point.

x := myfunc1()

is arguably less clear, as without context (such as knowing the type), it doesn't look like there is an error path in here.

I'm not saying that idioms can't change, but they have to bring benefits proportional to the cost they incur. Right now the benefits of this proposal aren't very clear.

Types are written once, but the values they produce are used many times.

x := foo()
if x.Err() != nil {
    return x.Err()
}

vs

x, err := foo()
if err != nil {
    return err
}

Here, it's longer and harder to read, with more symbols in the way.

ErrorOr[struct{int; float32}]

Feels like a step backwards, your type declaration is now much longer (and you haven't named the fields yet), and if you move the struct to be a named type, you now have to deal with the an inflation of types that exist solely to return multiple values when the language can already do that. You're also forcing return values to be grouped together when they may have no reason to be.

[ianlancetaylor]

A somewhat minor point, but I think of func F() (int, error) as a function that returns an int (and also an error). If I write func F() ErrorOr[int] then it looks like I have a function that returns an error (and also an int). For a function that returns a value, the fact that it also returns an error is not an important fact about that function. It would be nicer if it didn't take precedence when writing the function signature.

[Merovius]

@jabolopes

Can you point me to the feedback for the try proposal so that I can evaluate whether the feedback applies to the ErrorOr proposal or not? Thanks!

I did. There's probably more, on golang-nuts or the subreddit or the gopher slack, but TBQH I'm not really better at finding that than you and the github issue should be plenty.

IMO it is pretty much out of the question that the same arguments apply. The only real difference between the two is that one spells it try(F()), while the other spells it <-F(). Apart from that, they seem to be exactly equivalent. Which was the real point: Doing anything like that doesn't require ErrorOr.

[Merovius]

I agree. It doesn't need to be in the stdlib. The only reason I mentioned the stdlib is because there might be error-returning functions in the stdlib that would like to use ErrorOr. In this case, I doubt very much that the stdlib maintainers would introduce a dependency on a third party package. But running this as a trial from a third-party package or in the experiments (exp) package would also be a good approach.

FWIW the true benefit of providing it as a third party package is that it makes this discussion completely obsolete. You could just write that package, publish it and publicize it and no one could object to that. If the community likes it, people will start using it. And instead of arguing about its value in the abstract, you could file a proposal in year or two pointing at all the projects using it, to make the case of moving it into the stdlib.

I highly doubt that an addition like this would make it into the stdlib without that path. Even if we would, say, add the type to errors, 1. old stdlib function can't use it, because of backwards compatibility and 2. new functions wouldn't use it, before it has clearly crystallized as the accepted new idiom. Because consistency matters and while we can change conventions and live with the resulting inconsistency for a while, we wouldn't risk getting locked into a new convention which then gets deprecated again. That's why code which doesn't have to be in the stdlib, shouldn't start out there.

I think the history of github.com/pkg/errors can serve as a good model here. It existed as an "unblessed" third party package for years, slowly gaining adoption by the community. At some point, it became clear that the community wants something like it enough to warrant trying to put something in the stdlib. So a proposal was filed, resulting in golang.org/x/exp/errors, which eventually was moved into errors. There are a couple extra steps here and github.com/pkg/errors is not the only package which can claim heritage over this change. But the basic model is that this was done as a third party package and all the draft designs and discussions and proposals to move it into the stdlib happened after it was clear that the community wants something like it.

It just makes things better for everyone involved - you don't have to argue that it's a good idea and just do it, the Go projects can benefit from the experience gained by the community over the time as a third-party package and when we finally do add it, we can discuss bigger changes. Maybe even to the language proper. Maybe allowing the existing stdlib functions adopt it, without breaking compatibility.

[jabolopes]

@seankhliao

without context (...) it doesn't look like there is an error path in here.

The problem you are describing already exists in Go today and it relies on naming conventions (or context) for clarity. For example, the following examples are made clear not because we know the types of the variables but because we follow a naming convention. With ErrorOr, it's no different.

x := f()         // A value (i.e., not an error)
err := f()       // An error
x, err := f()    // A value and error
x, y := f()      // 2 values
x, ok := f()     // A value and a bool
x, y, ok := f()  // 2 values and a bool

Types are written once, but the values they produce are used many times.

In the general case, we should assume that both the caller and callee use ErrorOr[T], so the example is like this:

x := foo()
if x.Err() != nil {
    return x
}

and you haven't named the fields yet

A lot of APIs don't name the return values either when using (T, error). It's a matter of preference, not a strict requirement. With ErrorOr, it should be no different.

grouped together when they may have no reason to be.

The reason is so that we can instantiate generic types such as T and func()T with errors ErrorOr[T] and func()ErrorOr[T], respectively. This is not possible with the current error handling mechanism since (T, error) cannot be used to instantiate T and func()(T, error) cannot be used to instantiate func()T. This explained in detail in the section titled "Error-agnostic generic APIs".

@ianlancetaylor

It would be nicer if it didn't take precedence when writing the function signature.

I agree. If you have some ideas I'd love to hear more. Here are few ideas (in no particular order):

1. Change the name ErrorOr to something that emphasizes the final value rather than the error. Naming is hard, but a few options could be Result[int] or Checked[int] or Safe[int] or Handled[int], or any short form like R[int], C[int], S[int], etc. Suggestions are very much welcome!

2. Another way to go about this would be to use someting like Either[int, error], which is more similar to the (T, error) approach. The disadvantage of this approach is that Either does not name its methods as Val() and Err() but something more general like Left() and Right() so I think it wouldn't read as well on the calling side.

3. Another idea would be to have syntactic sugar for ErrorOr, for example, writing int? would mean ErrorOr[int]. For example, func MyFunc() int? would be syntactic sugar for func MyFunc() ErrorOr[int] (assuming we're still using the ErrorOr name, because that can also be changed). Instead of ? we could also pick another syntax, e.g., int+, int., int!, etc, and even invert the order, e.g., ?int, +int, .int, etc, which is probably even more consistent with the slice syntax, for which [] precedes the underlying type.

Does this go in the direction you're thinking? Feedback would be great!

@Merovius

I agree with your approach. I will create a third party package with the ErrorOr so that community members can use it if they wish. I will create this package over the weekend and paste the link to the package back to this discussion.

A few more thoughts below:

The only real difference between the two

I don't want to lose sight of the bigger picture here. This proposal is to overcome the problems with the titled sections "The 4 states of (T, error)", the "Error-agnostic generic APIs", the "Errors / values in channels, maps, slices", the "Naming problem with err", the "Inconsistent use of (T, error)".

The section about "Future ideas" is out scope and is only meant as an idea. So I don't want to reduce the whole proposal to just the "Future ideas" section.

I will take a closer look at the try proposal over the weekend to better evaluate the feedback.

discussion completely obsolete

The point of this discussion is to rethink error handling in the presence of generics. I think this got lost when my proposal's title was unilaterally changed to something with a different meaning. I have changed the title back and I'd prefer (if possible) that the title's meaning remain unchanged, although the actual wording can be changed.

This proposal is not about defining the ErrorOr type, although getting feedback about the ErrorOr type is also useful.

[seankhliao]

On the title, proposals should have a clear scope and design that can be evaluated and decided upon. If you want a general discussion of error handling possibilities, it's out of scope and should be done in one of the forums

[jabolopes]

@seankhliao

I followed @robpike 's suggestion to create a proposal and chose the title following his comment "You are asking for a new way to think about errors.". So I don't see what is wrong with that.

clear scope and design

This has a clear scope described in the "Objectives" section and a clear design described in the "Design ideas" section.

I would like to have a title along the lines of "thinking error handling in the presence of generics" without you unilaterally changing it without my consent. Please let me know how we can get there. I'm open to suggestions on wording / spelling if that's the blocking issue.