umoci 0.5.1 -- "🖤 Yuki (2021-2025)"

This is a fairly minor update to umoci, containing a few bugfixes for
some potential issues, as well as finally removing the requirement for
oci-image-tool validation. We still do not support the latest image-spec
release, but decoupling for oci-image-tool is a very important first
step.

Fixed

• For images with an empty index.json, umoci will no longer incorrectly set
  the manifests entry to null (which was technically a violation of the
  specification, though such images cannot be pushed or interacted with outside
  of umoci).
• Based on some recent developments in the image-spec, umoci
  will now produce an error if it encounters descriptors with a negative size
  (this was a potential DoS vector previously) as well as a theoretical attack
  where an attacker would endlessly write to a blob (this would not be
  generally exploitable for images with descriptors).

Changed

• We now use go:embed to fill the version information of umoci --version,
  allowing for users to get a reasonable binary with go install. However, we
  still recommend using our official binaries, using distribution binaries, or
  building from source with make.

• Rather than using oci-image-tool validate for validating images in our
  tests, we now make use of some hand-written smoke tests as well as the
  jq-based validators maintained in docker-library/meta-scripts.

  This is intended to act as a stop-gap until umoci validate is implemented
  (and after that, we may choose to keep the jq-based validators as a
  double-check that our own validators are working correctly).

Thanks to the following contributors who made this release possible:

• Adam Korcz Adam@adalogics.com
• Akhil Mohan akhilerm@gmail.com
• Aleksa Sarai cyphar@cyphar.com

---

This release is dedicated to our cat Yuki who sadly passed away on
Friday. Most of the code I've written in the past four years was written
with him purring away on my chest, and he was the most loving cat I've
ever met. Rest in peace, little buddy. I hope you enjoyed your time with
us, and I'll always keep you in my heart. 🖤

Signed-off-by: Aleksa Sarai cyphar@cyphar.com

umoci 0.5.0 -- "A wizard is never late, Frodo Baggins. Nor is he early; he arrives precisely when he means to."

This is a long-awaited release of umoci containing some Go API breaking
changes, some new features, and many other minor changes and
improvements.

Note that the Go API is still considered to be unstable, so downstream
users should generally be aware that future updates may contain more
breaking changes until we release umoci v1.0.0. However, the umoci CLI
is considered to be stable (as it has been widely used for nearly a
decade now) and we will endeavour to not make breaking changes.

This version of umoci requires Go 1.23 to build.

Security

• A security flaw was found in the OCI image-spec, where it is possible to
  cause a blob with one media-type to be interpreted as a different media-type.
  As umoci is not a registry nor does it handle signatures, this vulnerability
  had no real impact on umoci but for safety we implemented the now-recommended
  media-type embedding and verification. CVE-2021-41190

Breaking

• The method of configuring the on-disk format and MapOptions in
  RepackOptions and UnpackOptions has been changed. The on-disk format is
  now represented with the OnDiskFormat interface, with DirRootfs and
  OverlayfsRootfs as possible options to use. MapOptions is now configured
  inside the OnDiskFormat setting, which will require callers to adjust their
  usage of the main umoci APIs. In particular, examples like

  unpackOptions := &layer.UnpackOptions{
      MapOptions: mapOptions,
      WhiteoutMode: layer.StandardOCIWhiteout, // or layer.OverlayFSWhiteout
  }
  err := layer.UnpackManifest(ctx, engineExt, bundle, manifest, unpackOptions)

  will have to now be written as

  unpackOptions := &layer.UnpackOptions{
      OnDiskFormat: layer.DirRootfs{ // or layer.OverlayfsRootfs
          MapOptions: mapOptions,
      },
  }
  err := layer.UnpackManifest(ctx, engineExt, bundle, manifest, unpackOptions)

  and similarly

  repackOptions := &layer.RepackOptions{
      MapOptions: mapOptions,
      TranslateOverlayWhiteouts: false, // or true
  }
  layerRdr, err := layer.GenerateLayer(path, deltas, repackOptions)

  will have to now be written as

  repackOptions := &layer.RepackOptions{
      OnDiskFormat: layer.DirRootfs{ // or layer.OverlayfsRootfs
          MapOptions: mapOptions,
      },
  }
  layerRdr, err := layer.GenerateLayer(path, deltas, repackOptions)

  Note that this means you can easily re-use the OnDiskFormat configuration
  between both UnpackOptions and RepackOptions, removing the previous need
  to translate between WhiteoutMode and TranslateOverlayWhiteouts.

  For users of the API that need to extract the MapOptions from
  UnpackOptions and RepackOptions, there is a new helper MapOptions which
  will help extract it without doing interface type switching. For
  OnDiskFormat there is also a Map method that gives you the inner
  MapOptions regardless of type.

• layer.NewTarExtractor now takes *UnpackOptions rather than
  UnpackOptions to match the signatures of the other layer.* APIs. Passing
  nil is equivalent to passing &UnpackOptions{}.

• In umoci 0.4.7, we added support for overlayfs unpacking using the
  still-unstable Go API. However, the implementation is still missing some key
  features and so we will now return errors from APIs that are still missing
  key features:

  • layer.UnpackManifest and layer.UnpackRootfs will now return an error
    if UnpackOptions.OnDiskFormat is set to anything other than DirRootfs
    (the default, equivalent to WhiteoutMode being set to
    OCIStandardWhiteout in umoci 0.4.7).

    This is because bundle-based unpacking currently tries to unpack all
    layers into the same rootfs and generate an mtree manifest -- this
    doesn't make sense for overlayfs-style unpacking and will produce garbage
    bundles as a result. As such, we expect that nobody actually made use of
    this feature (otherwise we would've seen bug reports complaining about it
    being completely broken in the past 4 years). [#574][]
    tracks re-enabling this feature (and exposing to umoci CLI users, if
    possible).

    Note that layer.UnpackLayer still supports OverlayfsRootfs
    (OverlayFSWhiteout in umoci 0.4.7).

  • Already-extracted bundles with OverlayfsRootfs (OverlayFSWhiteout in
    umoci 0.4.7) will now return an error when umoci operates on
    them -- we included the whiteout mode in our umoci.json but as the
    feature is broken, umoci will now refuse to operate on such bundles. Such
    bundles could only have been created using the now-error-inducing
    UnpackRootfs and UnpackManifest APIs mentioned above, and as mentioned
    above we expect there to have been no real users of this feature.

    Note that this only affects extracted bundles (a-la umoci unpack).
    Images created from such bundles are unaffected (even though their
    contents probably should be audited, since the implementation of this
    feature was quite broken in this usecase).

  Users should expect more breaking changes in the overlayfs-related Go APIs in
  a future umoci 0.6 release, as there is still a lot of work left to do.

Added

• umoci unpack now supports handling layers compressed with zstd. This is
  something that was added in image-spec v1.2 (which we do not yet support
  fully) but at least this will allow users to operate on zstd-compressed
  images, which are slowly becoming more common.
• umoci repack and umoci insert now support creating zstd-compressed
  layers. The default behaviour (called auto) is to try to match the last
  layer's compression algorithm, with a fallback to gzip if none of the layer
  algorithms were supported.
  • Users can specify their preferred compression algorithm using the new
    --compress flag. You can also disable compression entirely using
    --compress=none but --compress=auto will never automatically choose
    none compression.
• GenerateLayer and GenerateInsertLayer with OverlayfsRootfs
  (called TranslateOverlayWhiteouts in umoci 0.4.7) now support
  converting trusted.overlay.opaque=y and trusted.overlay.whiteout
  whiteouts into OCI whiteouts when generating OCI layers.
• OverlayfsRootfs now supports compatibility with the userxattr mount
  option for overlayfs (where user.overlay.* xattrs are used rather than
  the default trusted.overlay.*). This is a pretty key compatibility feature
  for users that use unprivileged overlayfs mounts and will hopefully remove
  the need for most downstream forks hacking in this functionality (such as
  stacker). For Go API users, to enable this just set UserXattr: true in
  OverlayfsRootfs. Note that (as with upstream overlayfs), only one xattr
  namespace is ever used (so if OverlayfsRootfs.UserXattr == true then
  trusted.overlay.* xattrs will be treated like any other non-overlayfs
  xattr).

Changes

• In this release, the primary development branch was renamed to main.
• The runtime-spec version of the config.json version we generate is no
  longer hard-coded to 1.0.0. We now use the version of the spec we have
  imported (with any -dev suffix stripped, as such a prefix causes havoc with
  verification tools -- ideally we would only ever use released versions of the
  spec but that's not always possible). #452
• Add the cgroup namespace to the default configuration generated by umoci unpack to make sure that our configuration plays nicely with runc when on
  cgroupv2 systems.
• umoci has been migrated away from github.com/pkg/errors to Go stdlib error
  wrapping.
• The gzip compression block size has been updated to be more friendly with
  Docker and other tools that might round-trip the layer blob data (causing the
  hash to change if the block size is different). #509

Fixed

• In 0.4.7, a performance regression was introduced as part of the
  VerifiedReadCloser hardening work (to read all trailing bytes) which would
  cause walk operations on images to hash every blob in the image (even blobs
  which we couldn't parse and thus couldn't recurse into). To resolve this, we
  no longer recurse into unparseable blobs. #373 #375 #394
• Handle EINTR on io.Copy operations. Newer Go versions have added more
  opportunistic pre-emption which can cause EINTR errors in io paths that
  didn't occur before. #437
• Quite a few changes were made to CI to try to avoid issues with fragility.
  #452
• umoci will now return an explicit error if you pass invalid uid or gid values
  to --uid-map and --gid-map rather than silently truncating the value.
• For Go users of umoci, GenerateLayer (but not GenerateInsertLayer) with
  OverlayfsRootfs (called TranslateOverlayWhiteouts in umoci
  0.4.7) had several severe bugs that made the feature unusable:
  • All OCI whiteouts added to the archive would incorrectly have the full host
    name of the path rather than the correctly rooted path, making the whiteout
    practically useless.
  • Any non-whiteout files would not be included in the layer, making the layer
    data incomplete and thus resulting in silent data loss.
    Given how severe these bugs were and the lack of bug reports of this issue in
    the past 4 years, it seems this feature has not really been used by anyone (I
    hope...).
• For Go users of umoci, UnpackLayer now correctly handles several aspects of
  OverlayfsRootfs (OverlayFSWhiteout in umoci 0.4.7) extraction
  that weren't handled correctly:
  • Unlike regular extractions, overlayfs-style extractions require us to
    create the parent directory ...

umoci 0.4.7

NOTE This release has a minor bug -- umoci --version will tell you that the version is "unknown". This was fixed in #369.

A security flaw was found in umoci, and has been fixed in this release.
If umoci was used to unpack a malicious image (using either
umoci unpack or umoci raw unpack) that contained a symlink entry for /.,
umoci would apply subsequent layers to the target of the symlink
(resolved on the host filesystem). This means that if you ran umoci as
root, a malicious image could overwrite any file on the system (assuming
you didn't have any other access control restrictions). Thanks to Robin
Peraglie from Cure53 for discovering this bug. CVE-2021-29136

Other changes in this release:

• umoci now compiles on FreeBSD and appears to work, with the notable
  limitation that it currently refuses to extract non-Linux images on any
  platform (this will be fixed in a future release -- see #364). #357
• Initial fuzzer implementations for oss-fuzz. #365

• umoci will now read all trailing data from image layers, to combat the
  existence of some image generators that appear to append NUL bytes to the end
  of the gzip stream (which would previously cause checksum failures because we
  didn't read nor checksum the trailing junk bytes). However, umoci will still
  not read past the descriptor length. #360
• umoci now ignores all overlayfs xattrs during unpack and repack operations,
  to avoid causing issues when packing a raw overlayfs directory. #354
• Changes to the (still-internal) APIs to allow for users to use umoci more
  effectively as a library.
  • The garbage collection API now supports custom GC policies. #338
  • The mutate API now returns information about what layers were added by the
    operation. #344
  • The mutate API now supports custom compression, and has in-tree support for
    zstd. #348 #350
  • Support overlayfs-style whiteouts during unpack and repack. #342

Thanks to all of the people who made this release possible:

• Adam Korcz adam@adalogics.com
• Aleksa Sarai cyphar@cyphar.com
• Mateusz Kwiatkowski mateusz@serveraptor.com
• Nisha K nishak@vmware.com
• Ramkumar Chinchani rchincha@cisco.com
• Shengjing Zhu zhsj@debian.org
• Tycho Andersen tycho@tycho.pizza

LGTMs: @cyphar
Signed-off-by: Aleksa Sarai cyphar@cyphar.com

umoci 0.4.6

umoci has been adopted by the Open Container Initative as a reference
implementation of the OCI Image Specification. This will have little impact
on the roadmap or scope of umoci, but it does further solidify umoci as a
useful piece of "boring container infrastructure" that can be used to build
larger systems.

NOTICE: As part of the adoption procedure, the import path and module
name of umoci has changed from github.com/openSUSE/umoci to
github.com/opencontainers/umoci. This means that users of our (still
unstable) Go API will have to change their import paths in order to update
to newer versions of umoci.

The old GitHub project will contain a snapshot of v0.4.5 with a few
minor changes to the readme that explain the situation. Go projects which
import import the archived project will receive build warnings that
explain the need to update their import paths.

• umoci now builds on MacOS, and we currently run the unit tests on MacOS to
  hopefully catch core regressions (in the future we will get the
  integration tests running to catch more possible regressions).
  #318

• Suppress repeated xattr warnings on destination filesystems that do not
  support xattrs. #311
• Work around a long-standing issue in our command-line parsing library (see
  urfave/cli#1152) by disabling argument re-ordering for umoci config,
  which often takes --prefixed flag arguments. #328

Thanks to all of the people who made this release possible:

• Aleksa Sarai asarai@suse.de
• David Trudgian dave@trudgian.net
• Ramkumar Chinchani rchincha@cisco.com
• Tycho Andersen tycho@tycho.ws

LGTMs: @cyphar @tych0
Signed-off-by: Aleksa Sarai asarai@suse.de

umoci 0.4.5

• Expose umoci subcommands as part of the API, so they can be used by other Go
  projects. openSUSE/umoci#289
• Add extensible hooking to the core libraries in umoci, to allow for
  third-party media-types to be treated just like first-party ones (the key
  difference is the introspection and parsing logic). openSUSE/umoci#299
  openSUSE/umoci#307

• Use type: bind for generated config.json bind-mounts. While this doesn't
  make too much sense (see opencontainers/runc#2035), it does mean that
  rootless containers work properly with newer runc releases (which appear to
  have regressed when handling file-based bind-mounts with a "bad" type).
  openSUSE/umoci#294 openSUSE/umoci#295
• Don't insert a new layer if there is no diff. openSUSE/umoci#293
• Only output a warning if forbidden extended attributes are present inside the
  tar archive -- otherwise we fail on certain (completely broken) Docker
  images. openSUSE/umoci#304

Thanks to all of the people who made this release possible:

• Aleksa Sarai asarai@suse.de
• David Trudgian dave@trudgian.net
• Tycho Andersen tycho@tycho.ws

Signed-off-by: Aleksa Sarai asarai@suse.de

umoci 0.4.4

• Full-stack verification of blob hashes and descriptor sizes is now done on
  all operations, improving our hardening against bad blobs (we already did
  some verification of layer DiffIDs but this is far more thorough).
  openSUSE/umoci#278 openSUSE/umoci#280 openSUSE/umoci#282

Thanks to all of the people that made this release possible:

• Aleksa Sarai asarai@suse.de
• Cameron Nemo camerontnorman@gmail.com
• Tobias Klauser tklauser@distanz.ch

Signed-off-by: Aleksa Sarai asarai@suse.de

umoci 0.4.3

• All umoci commands that had --history.* options can now decide to omit a
  history entry with --no-history. Note that while this is supported for
  commands that create layers (umoci repack, umoci insert, and umoci raw add-layer) it is not recommended to use it for those commands since it can
  cause other tools to become confused when inspecting the image history. The
  primary usecase is to allow umoci config --no-history to leave no traces in
  the history. See OSInside/kiwi#871. openSUSE/umoci#270
• umoci insert now has a --tag option that allows you to non-destructively
  insert files into an image. The semantics match umoci config --tag.
  openSUSE/umoci#273

Thanks to all of the people that made this release possible:

• Aleksa Sarai asarai@suse.de
• Jintao Zhang zhangjintao9020@gmail.com
• Tycho Andersen tycho@tycho.ws

Signed-off-by: Aleksa Sarai asarai@suse.de

umoci 0.4.2

• umoci now has an exposed Go API. At the moment it's unclear whether it will
  be changed significantly, but at the least now users can use
  umoci-as-a-library in a fairly sane way. openSUSE/umoci#245
• Added umoci unpack --keep-dirlinks (in the same vein as rsync's flag with
  the same name) which allows layers that contain entries which have a symlink
  as a path component. openSUSE/umoci#246
• umoci insert now supports whiteouts in two significant ways. You can use
  --whiteout to "insert" a deletion of a given path, while you can use
  --opaque to replace a directory by adding an opaque whiteout (the default
  behaviour causes the old and new directories to be merged).
  openSUSE/umoci#257

• Docker has changed how they handle whiteouts for non-existent files. The
  specification is loose on this (and in umoci we've always been liberal with
  whiteout generation -- to avoid cases where someone was confused we didn't
  have a whiteout for every entry). But now that they have deviated from the
  spec, in the interest of playing nice, we can just follow their new
  restriction (even though it is not supported by the spec). This also makes
  our layers slightly smaller. openSUSE/umoci#254
• umoci unpack now no longer erases system.nfs4_acl and also has some more
  sophisticated handling of forbidden xattrs. openSUSE/umoci#252
  openSUSE/umoci#248
• umoci unpack now appears to work correctly on SELinux-enabled systems
  (previously we had various issues where umoci wouldn't like it when it was
  trying to ensure the filesystem was reproducibly generated and SELinux xattrs
  would act strangely). To fix this, now umoci unpack will only cause errors
  if it has been asked to change a forbidden xattr to a value different than
  it's current on-disk value. openSUSE/umoci#235 openSUSE/umoci#259

Thanks to all of the people that made this release possible:

• Aleksa Sarai asarai@suse.de
• Tycho Andersen tycho@tycho.ws

Signed-off-by: Aleksa Sarai asarai@suse.de

umoci 0.4.1

• The number of possible tags that are now valid with umoci subcommands has
  increased significantly due to an expansion in the specification of the
  format of the ref.name annotation. To quote the specification, the
  following is the EBNF of valid refname values. openSUSE/umoci#234

  refname   ::= component ("/" component)*
  component ::= alphanum (separator alphanum)*
  alphanum  ::= [A-Za-z0-9]+
  separator ::= [-._:@+] | "--"
  

• A new umoci insert subcommand which adds a given file to a path inside the
  container. openSUSE/umoci#237
• A new umoci raw unpack subcommand in order to allow users to unpack images
  without needing a configuration or any of the manifest generation.
  openSUSE/umoci#239
• umoci how has a logo. Thanks to Max Bailey for contributing
  this to the project. openSUSE/umoci#165 openSUSE/umoci#249

• umoci unpack now handles out-of-order regular whiteouts correctly (though
  this ordering is not recommended by the spec -- nor is it required). This is
  an extension of openSUSE/umoci#229 that was missed during review.
  openSUSE/umoci#232
• umoci unpack and umoci repack now make use of a far more optimised gzip
  compression library. In some benchmarks this has resulted in umoci repack
  speedups of up to 3x (though of course, you should do your own benchmarks).
  umoci unpack unfortunately doesn't have as significant of a performance
  improvement, due to the nature of gzip decompression (in future we may
  switch to zlib wrappers). openSUSE/umoci#225 openSUSE/umoci#233

Thanks to all of the contributors that made this release possible:

• Abhilash Raj maxking@users.noreply.github.com
• Akihiro Suda suda.akihiro@lab.ntt.co.jp
• Aleksa Sarai asarai@suse.de
• Felix Abecassis fabecassis@nvidia.com
• Max Bailey maxbailey@flywheel.io
• Tycho Andersen tycho@tycho.ws

Signed-off-by: Aleksa Sarai asarai@suse.de

umoci 0.4.0

• umoci repack now supports --refresh-bundle which will update the
  OCI bundle's metadata (mtree and umoci-specific manifests) after packing the
  image tag. This means that the bundle can be used as a base layer for
  future diffs without needing to unpack the image again. openSUSE/umoci#196
• Added a website, and reworked the documentation to be better structured. You
  can visit the website at umo.ci. openSUSE/umoci#188
• Added support for the user.rootlesscontainers specification, which allows
  for persistent on-disk emulation of chown(2) inside rootless containers.
  This implementation is interoperable with @AkihiroSuda's PRoot
  fork (though we do not test its interoperability at the
  moment) as both tools use the same protobuf
  specification. openSUSE/umoci#227
• umoci unpack now has support for opaque whiteouts (whiteouts which remove
  all children of a directory in the lower layer), though umoci repack does
  not currently have support for generating them. While this is technically a
  spec requirement, through testing we've never encountered an actual user of
  these whiteouts. openSUSE/umoci#224 openSUSE/umoci#229
• umoci unpack will now use some rootless tricks inside user namespaces for
  operations that are known to fail (such as mknod(2)) while other operations
  will be carried out as normal (such as lchown(2)). It should be noted that
  the /proc/self/uid_map checking we do can be tricked into not detecting
  user namespaces, but you would need to be trying to break it on purpose.
  openSUSE/umoci#171 openSUSE/umoci#230

• Fix a bug in our "parent directory restore" code, which is responsible for
  ensuring that the mtime and other similar properties of a directory are not
  modified by extraction inside said directory. The bug would manifest as
  xattrs not being restored properly in certain edge-cases (which we
  incidentally hit in a test-case). openSUSE/umoci#161 openSUSE/umoci#162
• umoci unpack will now "clean up" the bundle generated if an error occurs
  during unpacking. Previously this didn't happen, which made cleaning up the
  responsibility of the caller (which was quite difficult if you were
  unprivileged). This is a breaking change, but is in the error path so it's
  not critical. openSUSE/umoci#174 openSUSE/umoci#187
• umoci gc now will no longer remove unknown files and directories that
  aren't flock(2)ed, thus ensuring that any possible OCI image-spec
  extensions or other users of an image being operated on will no longer
  break. openSUSE/umoci#198
• umoci unpack --rootless will now correctly handle regular file unpacking
  when overwriting a file that umoci doesn't have write access to. In
  addition, the semantics of pre-existing hardlinks to a clobbered file are
  clarified (the hard-links will not refer to the new layer's inode).
  openSUSE/umoci#222 openSUSE/umoci#223

---

Thanks to all of the contributors that made this release possible:

• Aleksa Sarai asarai@suse.de
• Jonathan Boulle jonathanboulle@gmail.com
• Serge Hallyn serge@hallyn.com
• Tobias Klauser tklauser@distanz.ch
• Tycho Andersen tycho@tycho.ws
• W. Trevor King wking@tremily.us

Signed-off-by: Aleksa Sarai asarai@suse.de