Rust Deep Dive - Pinning 📌

rust-deep-dive-pinning.md

Rust Deep Dive - Pinning 📌

If a Pin drops in a room, and nobody around understands it, does it make an unsound?

Josh Triplett

Introduction

Pre-reqs

• Auto traits: Traits that rust automatically implements for primitives and anything with all constituents (e.g. fields of a struct) implementing that trait.
• Existence of std::mem::swap implies that ability to gain mutable ref means ability to move

TL;DR:

• In some cases, a type can contain a reference to itself. In such cases, we need to ensure that it can't be moved in memory, to keep the self-reference valid & safe. Rust's type system facilitates this by providing Pin & Unpin
• Pin is a wrapper for a pointer (so anything implementing Deref), which can be used in two ways:
  • To wrap pointers to such self-referential types, thus preventing movement.
  • To make our API only accept types that are already wrapped, ensuring safety of the operation you're going to perform on them.
• Unwrapping the Pin, then, either requires Unpin or unsafe code.
  • Rust provides Unpin as an auto trait: most types implement it, so most types when wrapped by pin can be unwrapped for free.
  • To indicate unsafety, and make Pin restrictive, the type must implement !Unpin — not Unpin.
• Now coming to Futures, they can often fall under the unsafe category, because they have to take references as they need to store state so that it's available during the next poll. This can also mean taking references to the data, leading to a self referential type.
  • Specifically, the poll method requires a pinned reference to Self, because poll is going to progress the future, which mean storing state, which means moving, which is unsafe to do if the future is not pinned!
• For consuming API requiring Pinned types, rust provides us with certain ways to go from impl Deref to Pin<impl Deref>:
  • Add indirection, e.g.: using Box::pin, which gives us Pin<Box<T>>: This is Unpin if T is Unpin, which is true for pointers. This means the reference is to a reference, so moving only leads to moving the pointer, the object itself isn't moved, ensuring the address pointed to by the self-reference remains valid
  • Manually ensure that the object doesn't move, e.g.: using pin-projection: The wrapped type, constructed using unsafe code (e.g. using Pin::new_unchecked), shadows the original type, making it unavailable to be used. In this case we need to manually uphold Pin's contract that the object doesn't move.
  • pin! macro can be used for "local" pinning (could be either stack or heap depending on the specifics, unlike Box::pin which is always on heap)

Relevant API Source

• Relevant Pin API:

impl<'a, T> Pin<&'a mut T>
where
    T: ?Sized,
{
    // Can only get to underlying &mut for Unpin types
    pub fn get_mut(self) -> &'a mut T
    where
        T: Unpin,

    // Otherwise unsafe code is required
    pub unsafe fn get_unchecked_mut(self) -> &'a mut T
    //  ^^^^^^
}

// Can only get to underlying &mut for Unpin types
impl<Ptr> DerefMut for Pin<Ptr>
where
    Ptr: DerefMut,
    <Ptr as Deref>::Target: Unpin,

// Box<T> is always Unpin, regardless of T being Unpin or not
impl Unpin for Box<T>
where
    T: ?Sized,

impl<T> Box<T> {
    pub fn pin(x: T) -> Pin<Box<T>>
}

Auxiliary Examples

Self-referential Future requiring Pin

(Example from The Why, What, and How of Pinning in Rust by Jon Gjengset- YouTube)

• Consider asynchronously reading into a local buffer:

  async fun foo(s: &mut dyn AsyncRead) {
      let mut buf = [0; 1024];
      tokio::io::read(s, &mut buf[..]).await;
  }

• This involves 2 futures, which means two state machines, which could look something like this:

struct IoRead<'a> {
    buf: &'a mut [u8],
    s: &mut dyn AsyncRead,
}

struct FooFuture {
    buf: [u8; 1024],
    s: &mut dyn AsyncRead,
    progress: enum {
        Step0,
        Step1(IoRead<'buf>)
    }
}

• If we implement Future for FooFuture, it might look something like this:

impl Future for FooFuture {
    type Output = ...;
    // Assume we don't have Pin yet
    fn poll(self: &mut Self) -> Poll<Self::Output> {
        match self.progress {
            Step0 => {
                self.progress = Step1(IoRead::new(s, &mut self.buf[..]));
            }
            Step1(ref mut ioReadFuture) => {
                match(ioReadFuture.poll()) {
                    Poll::Ready(_) => {/* ... Step 2 ... */},
                    Poll::Pending => return Poll::Pending,
                }
            }
        }
    }
}

• Then this becomes problematic:

fun main() {
    let f = foo(); // Step0
    let g = f;
    let x = g;
    let f = x;     // No issues yet, as still Step0, so no references yet!
    f.poll();      // Step1, f.progress.step1.ioread.buf points to f.buf
    let z = f;     // f moved, but f..ioread.buf still points to f.buf ???
    z.poll();      // 💥 Crash with invalid pointer
}

• However, if we now introduce Pin into the picture, let z = f becomes safe because we're moving the pin, not f itself. And we can't get from the pin to f without using unsafe code

fun main() {
    let f = foo();
    let f = Pin::new(&mut f);
    f.poll()
    let z = f; // Fine, as only pin is moved, f stays in same position
    z.poll();
    mem::replace(&mut *f, foo());
    //                ^^
    // Can't do -> DerefMut is only implemented for T: Unpin
}

A Future not requiring Pin

struct Ready<T> {
    value: Option<T>,
}

impl<T: Unpin> Future for Ready<T> {
    type Output = T;

    fn poll(self: Pin<&mut Self>) -> Poll<Self::Output> {
        Poll::Ready(self.value.take().unwrap())
        //          ^^^^^^^^^^
        // Allowed as DerefMut is implemented for Pin<P<T>> for T: Unpin
    }
}

fn main() {
    let r = Pin::new(Ready {value: Some(42u8)});
    r.poll();
    std::mem::replace(&mut *r, Ready { value: Some(43u8) });
    //                     ^^
    // Allowed as DerefMut is implemented for Pin<P<T>> for T: Unpin
    r.poll();
}

Non Future example of Pinning: Generators

// Assumed syntax for generators when they come in rust
gen fn numbers() -> Option<usize> {
    let mut buf = [0; 1024];
    for i in 0..buf.len() {
        yield Some(buf[i]); // Self-reference
    }
    None
}

Advanced

Drop Guarantee

• The purpose of Pinning is not just to prevent movement, but more generally to be able to rely on a pinned value remaining valid at a specific place in memory.
• This implies that the memory location that stores the value must not get invalidated or otherwise repurposed during the lifespan of the pinned value until its drop returns or panics
• For common cases such as dropping on function return and freeing heap storage, rust runs drop for us when using standard safe code. However, unsafe code must make sure to call ptr::drop_in_place before deallocating and re-using the storage.
• Additionally, when manually implementing Drop for a type that could have been pinned, we must treat Drop as implicitly taking Pin<&mut Self> instead of the normal &mut Self. This could be done safely as follows:

impl Drop for Type {
    fn drop(&mut Self) {
        // okay as self is never used again
        inner_drop(unsafe{ Pin::new_unchecked(self) })
        fn inner_drop(this: Pin<&mut Self>) {
            // Actual drop implementation
        }
    }
}

Structural Pinning

If you move a type, the fields also move. While defining a structure that can be pinned, we need to decide if pinning also propagates to the fields. So when projecting from Pin<&mut Struct>, will we get Pin<&mut Field> or &mut Field. Both are fine as long as we stick to only one of the two per field. Pinning that propagates to field in the former case is called Structural: it follows the structure of the type.

// If MyFuture moves, f also moves. So structural pinning.
struct MyFuture<F> {
    f: F
}

impl<F: Future> Future for MyFutureBox<F> {
    type Output = F::Output;
    fn poll(self: Pin<&mut Self>, cx: Context<'_>) -> Poll<Self::Output> {
        // [Take 1]: Poll away!
        // self.f.poll(cx) ❌-> No method poll for type F

        // [Take 2]: Pin and poll!
        // Pin::new(&mut self.f).poll(cx) ❌-> Unpin not implemented for F

        // [Take 3]: Unpin, pin and poll!
        // Make F: Future + Unpin --> compiles, but when we actually
        //                            create a MyFuture:
        // pub MyFuture {
        //     f: async {}
        // }.await ❌-> Unpin not implemented for GenFuture (the generated
        //              future for the async block)

        // [Take 4]: Unsafe tiz :/
        let f = unsafe { self.map_unchecked_mut(|this| &mut this.f) };
        // Safety of this relies on the requirement that the closure
        // does NOT move things that aren't Unpin.
        // This is true here because:
        // - `this.f` doesn't move as long as `this` doesn't move, by
        //   structural pinning
        // - `this` won't move, because we have a pinned reference to
        //   it in the argument.
        // However, if we `impl Unpin for MyFuture`, it can be moved
        // despite the pinned reference, so we need to ensure not to
        // do that
        f.poll(cx)
    }
}

A Pin of Box

struct MyFutureBox<F> {
    // Wrapping F in Box makes MyFutureBox<F> Unpin, as moving it only
    // moves the box, not F. However, F itself remains !Unpin
    f: Box<F>,
}

impl<F: Future> Future for MyFutureBox<F> {
    type Output = F::Output;
    fn poll(self: Pin<&mut Self>, cx: Context<'_>) -> Poll<Self::Output> {
        // Pin::new(&mut *self.f).poll(cx);
        //      ^^^      ^
        // Not valid as F is still !Unpin
        let f = unsafe { self.map_unchecked_mut(|this| &mut *this.f) };
        f.poll(cx)
    }
}

// Now if someone accidentally writes this, it's completely fine, as we
// can safely access f from self
impl <F> MyFutureBox<F> {
    fn get_f(&mut self) -> &mut F {
        &mut *self.f
    }
}

async fun baz() {}

// However, it makes this possible to do
fn main() {
    let x = MyFuture { f: Box::new(baz()) };
    pin_mut!(x); // Equivalent to let mut x = Pin::new(&mut x);
    std::mem::replace(x.get_f(), baz()); // 💥
}
// Which is just breaking the assumption made by the usafe block inside poll

// So we use Pin<Box<T>> instead of Box<T>
struct MyFuturePinBox<F> {
    f: Pin<Box<F>>,
}

impl <F> MyFuturePinBox<F> {
    fn get_f(&mut self) -> &mut F {
        &mut *self.f
        //   ^
        // This now becomes invalid as we can't DerefMut out of Pin
        // without using unsafe
    }
}

// And now we can also remove all unsafe code
impl<F: Future> Future for MyFuturePinBox<F> {
    type Output = F::Output;
    fn poll(self: Pin<&mut Self>, cx: Context<'_>) -> Poll<Self::Output> {
        // Look ma, no unsafe!
        self.f.as_mut().poll(cx)
    }
}

// But how?
// - The `Box` guarantees that moving `MyFuturePinBox` does not move `F`
// - The `Pin` guarantees that there is no way to move `F` anymore (unless
//   it's `Unpin`)
// The only potential drawback is heap allocation

• So to summarize, Pin<Box<T>> works because Box puts T on the heap somewhere, and Pin ensures you can never get an immutable reference to it.
• Another thing to note is that Box does not lead to structural pinning: moving Box<T> does NOT move T.

Project Pin

pin-project (proc-macro), pin-project-lite (non proc-macro) etc. help do useful things without writing unsafe code, such as:

• Adding project method(s) that returns an intermediate struct containing pinned references to fields that we want to be structurally pinned, and direct references to other fields.
• Adding an Unpin implementation that depends only on the fields that we want to be structurally pinned to be Unpin, regardless of whether other fields are Unpin or !Unpin
• Helping uphold Drop guarantee by disallowing impl Drop and providing PinnedDrop that basically enforces the recommended pattern

use std::pin::Pin;
use pin_project_lite::pin_project;

pin_project! {
    struct MyFuture<F,T> {
        #[pin]
        f: F
        // Not marked with pin
        t: T
    }
}

impl<F: Future, T> Future for MyFuture<F, T> {
    type Output = F::Output;
    fn poll(self: Pin<&mut Self>, cx: Context<'_>) -> Poll<Self::Output> {
        let this = self.project();
        let f: Pin<&mut F> = this.f; // Pinned reference to #[pin] field
        let _: &mut T = this.t; // Normal reference to non-#[pin] field
        f.poll(cx)
    }
}

The generated code looks something like (simplified):

use std::pin::Pin;
use core::marker::{PhantomData, Unpin};

struct MyFuture<F, T> {
    f: F,
    t: T,
}

// Intermediate struct returned by project method
struct Projection<'_p, F, T>
where
    MyFuture<F, T>: '_p,
{
    f: Pin<&'_p mut F>,
    t: &'_p mut T,
}

// project method
impl<F, T> MyFuture<F, T> {
    fn project<'_p>(
        self: Pin<&'_p mut Self>,
    ) -> Projection<'_p, F, T> {
        unsafe {
            let Self { f, t } = self.get_unchecked_mut();
            Projection {
                f: Pin::new_unchecked(f),
                t: t,
            }
        }
    }
}

// Intermediate struct for use in bounds for Unpin impl
struct __Origin<'_p, F, T> {
    __dummy_lifetime: PhantomData<&'_p ()>,
    // depends on F: Unpin | !Unpin
    f: F,
    // Doesn't depend on T, wrapped in AlwaysUnpin: Unpin
    t: ::pin_project_lite::__private::AlwaysUnpin<T>,
}

// Unpin impl
impl<'_p, F, T> Unpin for MyFuture<F, T> where
    __Origin<'_p, F, T>: Unpin
{
}

// Upholding drop guarantee: impl Drop becomes a compiler error
trait MustNotImplDrop {}
// This impl always applies regardless of MyFuture: Drop | !Drop
impl<F, T> MustNotImplDrop for MyFuture<F, T> {}
// This impl *also* applies iff MyFuture: Drop, leading to ambiguity
// during compilation, thus preventing compilation
impl<T: Drop> MustNotImplDrop for T {}

More Advanced

A reading of std::pin

First we put the data in a box, which will be its final resting place

Notable snippets

• … it has been permanently (until the end of its lifespan) attached to its location in memory …
• Pinning a value is an incredibly useful building block for unsafe code to be able to reason about whether a raw pointer to the pinned value is still valid.
• You can move out of a Box<T>, or you can use mem::replace to move a T out of a &mut T
• We say that a value has been pinned when it has been put into a state where it is guaranteed to remain located at the same place in memory from the time it is pinned until its drop is called.
• Types that have an address-sensitive state usually follow a lifecycle that looks something like so:
  1. A value is created which can be freely moved around.
     • e.g. calling an async function which returns a state machine implementing Future
  2. An operation causes the value to depend on its own address not changing
     • e.g. calling poll for the first time on the produced Future
  3. Further pieces of the safe interface of the type use internal unsafe operations which assume that the address of the value is stable
     • e.g. subsequent calls to poll
  4. Before the value is invalidated (e.g. deallocated), it is dropped, giving it a chance to notify anything with pointers to itself that those pointers will be invalidated
     • e.g. dropping the Future
• Why not just Box with restricted API?
  • Although there were other reasons as well, this issue of expensive composition is the key thing that drove Rust towards adopting a different model. It is particularly a problem when one considers, for example, the implications of composing together the Futures which will eventually make up an asynchronous task (including address-sensitive async fn state machines). It is plausible that there could be many layers of Futures composed together, including multiple layers of async fns handling different parts of a task. It was deemed unacceptable to force indirection and allocation for each layer of composition in this case.
• We call such a Pin-wrapped pointer a pinning pointer, (or pinning reference, or pinning Box, etc.). Notice that the thing wrapped by Pin is not the value which we want to pin itself, but rather a pointer to that value! A Pin<Ptr> does not pin the Ptr; instead, it pins the pointer’s pointee value.
• In order to accomplish the goal of pinning the pointee value, Pin<Ptr> restricts access to the wrapped Ptr type in safe code. For example, a Pin<&mut T> makes it impossible to obtain the wrapped &mut T safely because through that &mut T it would be possible to move the underlying value out of the pointer with mem::replace, etc.
• AddrTracker Example:

  #[derive(Default)]
  struct AddrTracker {
      prev_addr: Option<usize>,
      // --- New ---
      // remove auto-implemented `Unpin` bound to mark this type as having
      // some address-sensitive state. This is essential for our expected
      // pinning guarantees to work, and is discussed more below.
      _pin: PhantomPinned,
  }
  
  impl AddrTracker {
      // If we haven't checked the addr of self yet, store the current
      // address. If we have, confirm that the current address is the same
      // as it was last time, or else panic.
      // -- Old --
      // fn check_for_move(&mut self) {
      //     let current_addr = self as *mut Self as usize;
      //     match self.0 {
      //         None => self.0 = Some(current_addr),
      //         Some(prev_addr) => assert_eq!(prev_addr, current_addr),
      //     }
      // }
      // -- New --
      fn check_for_move(self: Pin<&mut Self>) {
          let current_addr = &*self as *const Self as usize;
          match self.prev_addr {
              None => {
                  // SAFETY: we do not move out of self
                  let self_data_mut = unsafe { self.get_unchecked_mut() };
                  self_data_mut.prev_addr = Some(current_addr);
              },
              Some(prev_addr) => assert_eq!(prev_addr, current_addr),
          }
      }
  }
  
  // --- Old ---
  // // Create a tracker and store the initial address
  // let mut tracker = AddrTracker::default();
  // tracker.check_for_move();
  // 
  // // Here we shadow the variable. This carries a semantic move, and may
  // // therefore also with a mechanical memory *move*
  // let mut tracker = tracker;
  // 
  // // May panic!
  // // tracker.check_for_move();
  
  // --- New ---
  // 1. Create the value, not yet in an address-sensitive state
  let tracker = AddrTracker::default();
  
  // 2. Pin the value by putting it behind a pinning pointer, thus putting
  // it into an address-sensitive state
  let mut ptr_to_pinned_tracker: Pin<&mut AddrTracker> = pin!(tracker);
  ptr_to_pinned_tracker.as_mut().check_for_move();
  
  // Trying to access `tracker` or pass `ptr_to_pinned_tracker` to anything that requires
  // mutable access to a non-pinned version of it will no longer compile
  
  // 3. We can now assume that the tracker value will never be moved, thus
  // this will never panic!
  ptr_to_pinned_tracker.as_mut().check_for_move();

• Note that the interaction between a Pin<Ptr> and Unpin is through the type of the pointee value, <Ptr as Deref>::Target. Whether the Ptr type itself implements Unpin does not affect the behavior of a Pin<Ptr>. For example, whether or not Box is Unpin has no effect on the behavior of Pin<Box<T>>, because T is the type of the pointee value, not Box. So, whether T implements Unpin is the thing that will affect the behavior of the Pin<Box<T>>.
• Exposing a field of a pinned value through a pinning pointer is called “projecting” a pin, and the more general case of deciding in which cases a pin should be able to be projected or not is called “structural pinning.”
• Self reference type example: Unmovable

  use std::pin::Pin;
  use std::marker::PhantomPinned;
  use std::ptr::NonNull;
  
  /// This is a self-referential struct because `self.slice` points into `self.data`.
  struct Unmovable {
      /// Backing buffer.
      data: [u8; 64],
      /// Points at `self.data` which we know is itself non-null. Raw pointer because we can't do
      /// this with a normal reference.
      slice: NonNull<[u8]>,
      /// Suppress `Unpin` so that this cannot be moved out of a `Pin` once constructed.
      _pin: PhantomPinned,
  }
  
  impl Unmovable {
      /// Creates a new `Unmovable`.
      ///
      /// To ensure the data doesn't move we place it on the heap behind a pinning Box.
      /// Note that the data is pinned, but the `Pin<Box<Self>>` which is pinning it can
      /// itself still be moved. This is important because it means we can return the pinning
      /// pointer from the function, which is itself a kind of move!
      fn new() -> Pin<Box<Self>> {
          let res = Unmovable {
              data: [0; 64],
              // We only create the pointer once the data is in place
              // otherwise it will have already moved before we even started.
              slice: NonNull::from(&[]),
              _pin: PhantomPinned,
          };
          // First we put the data in a box, which will be its final resting place
          let mut boxed = Box::new(res);
  
          // Then we make the slice field point to the proper part of that boxed data.
          // From now on we need to make sure we don't move the boxed data.
          boxed.slice = NonNull::from(&boxed.data);
  
          // To do that, we pin the data in place by pointing to it with a pinning
          // (`Pin`-wrapped) pointer.
          //
          // `Box::into_pin` makes existing `Box` pin the data in-place without moving it,
          // so we can safely do this now *after* inserting the slice pointer above, but we have
          // to take care that we haven't performed any other semantic moves of `res` in between.
          let pin = Box::into_pin(boxed);
  
          // Now we can return the pinned (through a pinning Box) data
          pin
      }
  }
  
  let unmovable: Pin<Box<Unmovable>> = Unmovable::new();
  
  // The inner pointee `Unmovable` struct will now never be allowed to move.
  // Meanwhile, we are free to move the pointer around.
  let mut still_unmoved = unmovable;
  assert_eq!(still_unmoved.slice, NonNull::from(&still_unmoved.data));
  
  // We cannot mutably dereference a `Pin<Ptr>` unless the pointee is `Unpin` or we use unsafe.
  // Since our type doesn't implement `Unpin`, this will fail to compile.
  // let mut new_unmoved = Unmovable::new();
  // std::mem::swap(&mut *still_unmoved, &mut *new_unmoved);

• The memory location that stores the value must not get invalidated or otherwise repurposed during the lifespan of the pinned value until its drop returns or panics. This point is subtle but required for intrusive data structures to be implemented soundly. E.g. For a doubly linked list, we might want to update references to an element in the preceding/succeeding elements in its drop method.
• Drop guarantee example:

  // Pin something inside a `ManuallyDrop`. This is fine on its own.
  let mut pin: Pin<Box<ManuallyDrop<Type>>> = Box::pin(ManuallyDrop::new(Type));
  // [Insight] Why is this fine?
  // - Ptr in Pin<Ptr> here is Box, not ManuallyDrop, so Ptr::Target is
  //   ManuallyDrop, not T.
  // - ManyallyDrop doesn't have any Drop implementation, so it upholds
  //   Pin's Drop guarantee by default, as not having a Drop implementation
  //   means no extra handling was required for that type's destruction,
  //   like it would for example be required for an intrusive data
  //   structure such as a linked list
  
  // However, creating a pinning mutable reference to the type *inside*
  // the `ManuallyDrop` is not!
  let inner: Pin<&mut Type> = unsafe {
      Pin::map_unchecked_mut(pin.as_mut(), |x| &mut **x)
  };
  // [Insight] Why is this not fine?
  // - ManuallyDrop wrapping T implies that T's Drop implementation won't
  //   be even if it has one, until it's manually invoked, which requires
  //   an unsafe operation
  // - However, giving out a Pin to T requires the drop guarantee -- as T
  //   might a type that requires its Drop implementation to be invoked to
  //   uphold soundness
  // - These two are in contradiction, so this code can never be made safe.

• Unsafe yet sound manual implementation of assignment for Unmovable:

  impl Unmovable {
      // Copies the contents of `src` into `self`, fixing up the
      // self-pointer the process.
      fn assign(self: Pin<&mut Self>, src: Pin<&mut Self>) {
          unsafe {
              let unpinned_self = Pin::into_inner_unchecked(self);
              let unpinned_src = Pin::into_inner_unchecked(src);
              *unpinned_self = Self {
                  data: unpinned_src.data,
                  slice: NonNull::from(&mut []),
                  _pin: PhantomPinned,
              };
  
              // Need to calculate the offset and length of the slice
              // within the data
              let data_ptr = unpinned_src.data.as_ptr() as *const u8;
              let slice_ptr = unpinned_src.slice.as_ptr() as *const u8;
              let offset = slice_ptr.offset_from(data_ptr) as usize;
              let len = (*unpinned_src.slice.as_ptr()).len();
  
              unpinned_self.slice = NonNull::from(&mut unpinned_self.data[offset..offset+len]);
          }
      }
  }

• Note that it is possible to assign generically through a Pin<Ptr> by way of Pin::set(). This does not violate any guarantees, since it will run drop on the pointee value before assigning the new value.
• For both Box<T> and Pin<Box<T>>, whether the content is pinned is entirely independent of whether the pointer is pinned, meaning pinning is not structural.

Takeaways

• The responsibility of upholding the pinning contract rests with the user of unsafe code interacting with the Pin API, including:
  • While pinning (e.g.: when creating a pinning pointer):
    • Ensuring that the type being pinned has implementations of Deref & DerefMut that don't go against the pinning contract
    • Ensuring that the pointer itself doesn't move out of or otherwise invalidate the pointee during Drop, Deref or DerefMut implementations
  • While creating a type that relies upon pinning for soundness:
    • Ensuring that it's not marked Unpin by adding PhantomPinned field
    • Ensuring Drop is treated s implicitly taking self: Pin<&mut Self>
  • While creating wrapper type to convert an external Unpin type to !Unpin:
    • Ensuring that Pin<&mut Wrapped> is not accessible, as Wrapped is Unpin and so trivially movable in safe code, defeating the purpose
• Adding Pin-ability can be a multi-step process:
  • It might not be enough to just use Pin<Ptr> in the API, the type might also need to be made !Unpin
• On Drop guarantee: while a lot of responsibility is assigned to the implementer of a type that requires pinning for soundness, the Drop guarantee is something that provides them with another tool allowing safe implementations in the form of the Drop implementation which can be used ensure and maintain soundness at the time of the mandatory "unpinning" that happens at the end of the object's lifetime.
• Further understanding the drop guarantee with the ManuallyDrop example:
  • Why is {rust} let mut pin: Pin<Box<ManuallyDrop<Type>>> = Box::pin(ManuallyDrop::new(Type)); fine?
    • Ptr in Pin<Ptr> here is Box, not ManuallyDrop, so Ptr::Target is ManuallyDrop, not T.
    • ManuallyDrop doesn't have any Drop implementation, so it upholds Pin's Drop guarantee by default, as not having a Drop implementation means no extra handling was required for that type's destruction, like it would for example be required for an intrusive data structure such as a linked list
  • Why is {rust} let inner: Pin<&mut Type> = unsafe { Pin::map_unchecked_mut(pin.as_mut(), |x| &mut **x) } not fine?
    • ManuallyDrop wrapping T implies that T's Drop implementation won't be invoked even if it has one, until it's manually invoked, which requires an unsafe operation
    • However, giving out a Pin to T requires the Drop guarantee -- as T might be a type that requires its Drop implementation to be invoked to uphold soundness
    • These two are in contradiction, so this code can never be made safe.
• On Projections:
  • Projection is nothing but a fancier word for, say, getter/setter in Java
    • {rust} impl Struct { fn field(&mut self) → &mut Field { &mut self.field } }
    • {java} class Clazz { Field getField() { return this.field; } }
  • Pinned projection is thus the decision whether {rust} field(self: Pin<&mut Self>) should return {rust} &mut Field or {rust} Pin<&mut Field>, and it turns out that it's upto the author of {rust} Struct to decide, with the sole requirement of either always projecting the pin, or never.
    • Pinning that propagates is also called structural, because it follows the structure of the type.
    • This choice can depend on whether the field itself is address-sensitive, or participates in the parent struct's address sensitivity, in which case it will need to be structurally pinned
• We may manually {rust} impl Unpin for Struct {} even when the type of a field in not {rust} Unpin, if the way the field's type interacts with pinning is no longer relevant in the context of its use in {rust} Struct, that is likely to be the case when pinning is NOT structural for the field
• Opting for structural pinning comes with some additional requirements:
  • Structural Unpin: Struct should be Unpin iff all of its structurally pinned fields are too
  • Pinned Destruction: Destructor must be written as if the arg was {rust} self: Pin<&mut Self>
  • Structural Notice of Destruction: Uphold the Drop guarantee for structurally pinned fields
  • Ensure no API is offered that can be used to move out of the structurally pinned fields
    • E.g.: Don't have an operation with type {rust} fn(Pin<&mut Struct<T>>) -> Option<T>
• Pointer types such as Box, and even Pin<Box<T>> & Pin<&mut T>, do NOT have structural pinning and are Unpin themselves
• Safe ways of constructing Pins:
  • pin! can be used for local pinning (often referred to as "stack" pinning, but this is not the case in async code as state is stored by the future across poll invocations)
  • Box::pin can be used for pinning to the heap

Exercises

Question

Imagine Rust did not require that futures were pinned in order to be polled. Which of the following async functions could potentially cause undefined behavior if not pinned?

async fn example(x: i32) -> i32 {
    let y = &x;
    sleep(Duration::from_secs(1)).await;
    *y
}

async fn example(x: i32) -> i32 {
    let x = 0;
    sleep(Duration::from_secs(1)).await;
    x
}

async fn example(x: i32) -> i32 {
    sleep(Duration::from_secs(1)).await;
    *x
}

async fn example(x: Vec<i32>) -> i32 {
    sleep(Duration::from_secs(1)).await;
    x[0]
}

Answer

Option 1:

async fn example(x: i32) -> i32 {
   let y = &x;
   sleep(Duration::from_secs(1)).await;
   *y
}

Context: The core problem addressed by pinning in self-reference, or a future which contains a pointer to itself. This happens an async block contains a local variable that refers to another local variable in the future. Here, that would be y = &x.

(Source: The quiz in Ch 17.5 of CS Brown's version of The Rust Book)

---

Exercise: Custom Wrapper Future (TimedWrapper)

Implement TimedWrapper: A custom Future type that wraps any given future and returns tuple of response and time taken on calling await.

Usage Example:

// Some async function, e.g. polling a URL with [https://docs.rs/reqwest]
// Remember, Rust functions do nothing until you .await them, so this isn't
// actually making a HTTP request yet.
let async_fn = reqwest::get("http://adamchalmers.com");

// Wrap the async function in my hypothetical wrapper.
let timed_async_fn = TimedWrapper::new(async_fn);

// Call the async function, which will send a HTTP request and time it.
let (resp, time) = timed_async_fn.await;
println!("Got a HTTP {} in {}ms", resp.unwrap().status(), time.as_millis())

(Source: Pin, Unpin, and why Rust needs them)

References | Resources

• Pin and suffering
• Pin, Unpin, and why Rust needs them
• Pinning - Asynchronous Programming in Rust
• Pin in std::pin - Rust
• Unpin in std::marker - Rust
• Box in std::boxed - Rust
• pin in std::pin - Rust
• std::pin - Rust
• pin-project-lite - lib.rs - source
• The Why, What, and How of Pinning in Rust - YouTube
• ManuallyDrop in std::mem - Rust

Contribs

• Fix typo by aksh1618 · Pull Request #87 · taiki-e/pin-project-lite · GitHub
• Fix typos in pin.rs by aksh1618 · Pull Request #133466 · rust-lang/rust · GitHub