const-eval: full support for pointer fragments

[RalfJung]

This fixes rust-lang/const-eval#72 and makes swap_nonoverlapping fully work in const-eval by enhancing per-byte provenance tracking with tracking of which of the bytes of the pointer this one is. Later, if we see all the same bytes in the exact same order, we can treat it like a whole pointer again without ever risking a leak of the data bytes (that encode the offset into the allocation). This lifts the limitation that was discussed quite a bit in #137280.

For a concrete piece of code that used to fail and now works properly consider this example doing a byte-for-byte memcpy in const without using intrinsics:

use std::{mem::{self, MaybeUninit}, ptr};

type Byte = MaybeUninit<u8>;

const unsafe fn memcpy(dst: *mut Byte, src: *const Byte, n: usize) {
    let mut i = 0;
    while i < n {
        *dst.add(i) = *src.add(i);
        i += 1;
    }
}

const _MEMCPY: () = unsafe {
    let ptr = &42;
    let mut ptr2 = ptr::null::<i32>();
    // Copy from ptr to ptr2.
    memcpy(&mut ptr2 as *mut _ as *mut _, &ptr as *const _ as *const _, mem::size_of::<&i32>());
    assert!(*ptr2 == 42);
};

What makes this code tricky is that pointers are "opaque blobs" in const-eval, we cannot just let people look at the individual bytes since we don't know what those bytes look like -- that depends on the absolute address the pointed-to object will be placed at. The code above "breaks apart" a pointer into individual bytes, and then puts them back together in the same order elsewhere. This PR implements the logic to properly track how those individual bytes relate to the original pointer, and to recognize when they are in the right order again.

We still reject constants where the final value contains a not-fully-put-together pointer: I have no idea how one could construct an LLVM global where one byte is defined as "the 3rd byte of a pointer to that other global over there" -- and even if LLVM supports this somehow, we can leave implementing that to a future PR. It seems unlikely to me anyone would even want this, but who knows.^^

This also changes the behavior of Miri, by tracking the order of bytes with provenance and only considering a pointer to have valid provenance if all bytes are in the original order again. This is related to rust-lang/unsafe-code-guidelines#558. It means one cannot implement XOR linked lists with strict provenance any more, which is however only of theoretical interest. Practically I am curious if anyone will show up with any code that Miri now complains about - that would be interesting data. Cc @rust-lang/opsem

[rustbot]

r? @davidtwco

rustbot has assigned @davidtwco.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[rustbot]

Some changes occurred to the CTFE machinery

cc @RalfJung, @oli-obk, @lcnr

Some changes occurred to the CTFE / Miri interpreter

cc @rust-lang/miri

Some changes occurred to the CTFE / Miri interpreter

cc @rust-lang/miri, @RalfJung, @oli-obk, @lcnr

The Miri subtree was changed

cc @rust-lang/miri

[RalfJung]

This will need t-lang FCP since it is an insta-stable extension of what const can do, but I first want to make CI pass.

r? @oli-obk

[RalfJung]

Let's see if there are any perf issues here.
@rust-timer queue
@bors2 try

[rust-bors]

⌛ Trying commit e97b4fe with merge 09988fb…

To cancel the try build, run the command @bors2 try cancel.

[rust-bors]

☀️ Try build successful (CI)
Build commit: 09988fb (09988fb389684ce1e69a7ca9c7238fbe228aa65e, parent: bf5e6cc7a7a7eb03e3ed9b875d76530eddd47d5f)

[rust-timer]

Finished benchmarking commit (09988fb): comparison URL.

Overall result: ❌ regressions - please read the text below

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.3%	[0.2%, 0.3%]	9
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results (primary -3.7%, secondary -3.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-3.7%	[-3.7%, -3.7%]	1
Improvements ✅ (secondary)	-3.4%	[-4.0%, -2.5%]	4
All ❌✅ (primary)	-3.7%	[-3.7%, -3.7%]	1

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 464.659s -> 464.054s (-0.13%)
Artifact size: 374.78 MiB -> 374.88 MiB (0.02%)

[RalfJung]

Good catch! The latest change should fix that.

[nikomatsakis]

@RalfJung

When writing an opaque pointer with provenance p into memory, it gets stored as 8 bytes that carry provenance Some((p, 0)), ..., Some((p, 7)).

I think I understand but maybe I don't. What is the value of these bytes -- is it just "opaque pointer"? i.e., should I think of that as enum Value { OpaquePointer, ScalarValue(u128) } or something?

[tmandry]

All implementations have to do this to make the new pass-tests that this PR added actually pass.

But, as this PR shows, it doesn't really have a performance impact. It does have a complexity impact, that's true. I don't think we should hold back on improvements that remove weird semantic papercuts (like errors talking about "overwriting parts of a pointer is not possible") for the sake of simplifying alternative Rust implementations.

But is the complexity required by an implementation if it doesn't attempt to catch all UB?

[nikomatsakis]

Let me restate some of what @tmandry pointed out in the meeting---

• I believe what we are saying is that an impl has to model this behavior in CTFE with sufficient resoluton to make examples like the one in the OP pass
• They don't necessarily have to catch the same set of bugs (UB) that rust would do

Put another way, we don't guarantee that we'll continue to fail in all the same cases, just that we won't regress the cases that work.

This implies that we are definitely requiring a complex impl but not necessarily the same level of complexity.

(In case, "an impl" is most likely a future version of rustc)

[RalfJung]

But is the complexity required by an implementation if it doesn't attempt to catch all UB?

Most of it, yes. Tracking provenance on the per-byte level and correctly carrying it along on copy/copy_nonoverlapping are something we are permanently committing to here, or else the "pass" tests added here will not work any more.

If we consider it UB to mix the bytes of different pointers and use the result as a single pointer, then the parts of this that track the exact fragment index, and the check whether all the fragments are consistent on a pointer load, are optional. I didn't consider this to be UB, I considered it to be just "unsupported", so I think we should check for this. But we do document that ptr-to-int transmutes are UB in const so it'd be consistent with that.

I think I understand but maybe I don't. What is the value of these bytes -- is it just "opaque pointer"? i.e., should I think of that as enum Value { OpaquePointer, ScalarValue(u128) } or something?

Hm, I don't know how you use the word "value" here because in my nomenclature, "value of a byte" is not a valid expression.

The data stored in a byte, with this PR, can be described as

enum Byte {
  Uninit,
  Init {
    val: u8,
    provenance: Option<(AllocId, PointerFragIdx)>,
  }
}

A value is a high-level concept, and can be described like

enum Value {
  Int(Int),
  Pointer {
    provenance: AllocId,
    offset: usize,
  },
  ...
}

When a value is stored in memory, we "serialize" it into a bunch of bytes. When we load a value from memory, we de-serialize. If the value is Pointer { provenance: a, offset: o }, then its serialized form is

[Init { val: o.to_le_bytes()[0], provenance: Some((a, 0)),
...
Init { val: o.to_le_bytes()[7], provenance: Some((a, 7))]

This assumes we are on a 64bit little-endian target.

(The actual representation in the compiler is different for efficiency reasons, this is the conceptual view.)

[nikomatsakis]

@RalfJung I think my question then is:

• what does to_le_bytes return, during const-eval, when the value you are invoking it on is a Pointer { ... }?

[scottmcm]

@rfcbot resolve spec-complexity-implication

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[RalfJung]

@RalfJung I think my question then is:

* what does `to_le_bytes` return, during const-eval, when the value you are invoking it on is a `Pointer { ... }`?

If you try to actually look at those bytes, it aborts const-eval with an error ("reading pointer as integer") -- this is a ptr-to-int transmute which will always abort execution.

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

[RalfJung]

@bors r=oli-obk
(here)

[bors]

📌 Commit ba5b6b9 has been approved by oli-obk

It is now in the queue for this repository.

[Zalathar]

Queue scheduling: If the current rollup fails, fall back to a rollup=never PR.

@bors p=1

[bors]

⌛ Testing commit ba5b6b9 with merge 99ba556...