[cipd] Fix directory -> symlink (and vice versa) upgrades.

CIPD package upgrade works in two stages:

1. Put all new files, removing/adjusting everything that stands in the way. For
   example, when directory 'a' is replaced by a file 'a' in a new package
   version, all 'a/*' files are removed and 'a' is replaced with a file. Same
   happens if 'a' is replaced with a symlink.
2. Remove all files that are no longer present in the new version of the
   package (per its manifest file).

Imagine a directory 'a' (with a file 'a/1') is updated to a symlink that points
to 'b' (that has 'b/1' inside). During the first stage, 'a' becomes a symlink
to 'b'. During the second stage, we remove 'a/1' (since it's no longer in the
package), and accidentally remove 'b/1' instead. Oops.

This is now prevented by granting 'a/*' "immutability" from removes. This
mechanism already existed to deal with file<->directory upgrades, it is
extended to understand symlink situation.

Now imagine we update a symlink 'a' that points to 'b' (which has 'b/1' inside)
back to be a directory (with 'a/1' inside). During the first stage, when
deploying 'a/1', we unknowingly follow the symlink and write to 'b/1' instead.
And during the second stage, 'b/1' is removed completely. So we end up with
a symlink to an empty directory. Oops.

This is now also prevented by ensuring all intermediary paths inside the site
root are real directories, not symlinks. We rely here on .cipd/manifest.json
containing only real directories in paths. Symlinks can only appear as leafs
there.

Besides unit tests, tested by running:
  cipd pkg-deploy old_xcode.zip -root r
  cipd deployment-check -root r
  cipd pkg-deploy new_xcode.zip -root r
  cipd deployment-check -root r
  cipd pkg-deploy old_xcode.zip -root r
  cipd deployment-check -root r

R=iannucci@chromium.org
BUG=915278

Change-Id: If429045ad5389cae51858396f8cb7587c18c78da
Reviewed-on: https://chromium-review.googlesource.com/c/1378599
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
4 files changed
tree: 33c2100c3a0e153319298fc775d7bb8aa2c3b0b0
  1. appengine/
  2. auth/
  3. buildbucket/
  4. cipd/
  5. client/
  6. common/
  7. config/
  8. dm/
  9. examples/
  10. gce/
  11. grpc/
  12. hardcoded/
  13. infra/
  14. logdog/
  15. luci_notify/
  16. lucicfg/
  17. lucictx/
  18. machine-db/
  19. milo/
  20. mmutex/
  21. mp/
  22. scheduler/
  23. scripts/
  24. server/
  25. starlark/
  26. tokenserver/
  27. tools/
  28. tumble/
  29. vpython/
  30. web/
  31. .travis.yml
  32. AUTHORS
  33. codereview.settings
  34. CONTRIBUTING.md
  35. CONTRIBUTORS
  36. LICENSE
  37. OWNERS
  38. pre-commit-go.yml
  39. PRESUBMIT.py
  40. README.md
README.md

luci-go: LUCI services and tools in Go

GoDoc

Installing

LUCI Go code is meant to be worked on from an Chromium infra.git checkout, which enforces packages versions and Go toolchain version. First get fetch via depot_tools.git then run:

fetch infra
cd infra/go
eval `./env.py`
cd src/go.chromium.org/luci

Contributing

Contributing uses the same flow as Chromium contributions.