commit | 125d102e429465dc3468b4b235f9e3a5df4b252c | [log] [tgz] |
---|---|---|
author | Robert Iannucci <iannucci@chromium.org> | Mon May 13 21:12:02 2019 |
committer | Commit Bot <commit-bot@chromium.org> | Mon May 13 21:12:02 2019 |
tree | 34d81d7e59866fdf9ce6e45e2d314a18b9521d54 | |
parent | 9b3ed1d5e912b8e9195060c3401da9b5370f2fce [diff] |
Reland: Refactor recipe engine step execution. This CL is unfortunately massive, but because the logic was previously tangled across the whole codebase it was difficult to make this refactor smaller. 1. The StepRunner API is vastly simplified. StepRunner API has been improved to have a small collection of well-defined methods, replacing the previous API which required implementations to return active 'OpenStep' implementations and implement context handlers. None of these new methods are allowed to raise an exception, meaning that the engine can operate purely in terms of their stated API. If an implementation is naughty and raises an exception anyway, it will be logged to the UI (as a RECIPE CRASH step) and then the engine will abort. 2. The StreamEngine is now solely owned by the Engine. The StreamEngine is now owned entirely by the RecipeEngine. The StepRunner is purely concerned with running steps, not with manipulating or reporting anything to the UI. 3. All recipe running logic is now implemented in the Engine. The engine now handles all calculation of step exceptions. Previously the raising of exceptions was handled by various parties in various ways around the engine and StepRunner, leading to bugs like crbug.com/931473. 4. Unexpected errors are now reported and handled much better. All steps now have a '$debug' log which records the engine's internal thoughts about how it's processing the step. Now if anything goes wrong while attempting to run the step, it will be recorded here. This includes placeholders raising exceptions, bugs in the engine, failure to locate the executable to run, failure to CD into the target directory of the step, etc. 5. Recipe engine logs are no longer mixed into stdout of steps. Previously information about the step would be dumped into the step's stdout log (e.g. the command we're running, the working directory, environment tweaks, the final return code of the step, etc.). Now all of this data is reported to an 'execution details' log on every step. 6. Timeouts are now handled orthogonally to step status. Previously timeouts would raise a StepTimeout and (I think? It's hard to tell) would turn the step red regardless of if it was an infra step or not. Now timeouts are recorded alongside the failure, and `infra_step` correctly controls the default color and exception. 7. Parent steps are now explicitly handled vs. non-parent steps Previously 'parent steps' were handled by the engine as-if they were normal steps without a command. However their semantics are very different since they 'remain open' even while other steps run (i.e. the steps inside of them). The engine now explicitly handles parent steps separately from leaf steps. This also means that parent nesting steps now have much clearer defined behavior in terms of what color they'll turn under different circumstances. 8. Test runner is much improved. Simulation tests now exercise a much greater portion of the recipe engine logic. Test expectations now include (normalized) stack traces when the recipe has exception(s), and (non-dropped) test expectations are always written to disk, both of which make debugging failing recipe simulations much easier. 9. Placeholder data on StepData will always be populated with None as a default, even if one of the placeholders crashes. This can help avoid secondary errors where one placeholder crashes and the second placeholder's collection step isn't able to run. Previously placeholder data was populated in-order for placeholders and if a placeholder didn't run there was nothing added to StepData for that value (leading to AttributeError's in unrelated areas of the recipe). Reland: Originally this CL failed to resolved commands (i.e. cmd0) which were given as a relative path (e.g. './foo'). R=tandrii@chromium.org Bug: 931473, 910369 Change-Id: I554cf4a2067d26c419c79464f5877c0f1ca8534e Recipe-Manual-Change: build Recipe-Manual-Change: release_scripts Recipe-Nontrivial-Roll: build_limited_scripts_slave Recipe-Nontrivial-Roll: chromiumos Recipe-Nontrivial-Roll: depot_tools Recipe-Nontrivial-Roll: infra Recipe-Nontrivial-Roll: skia Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/recipes-py/+/1610703 Commit-Queue: Robbie Iannucci <iannucci@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Recipes are a domain-specific language (embedded in python) for specifying sequences of subprocess calls in a cross-platform and testable way.
README.md
This file!
doc/
Documentation for the recipe engine (including this file!). See the design doc for more detailed design information about the recipe engine.
infra/
Chrome infra config files.
recipes.py
The main entry point to the recipe engine. It has many subcommands and flags; run recipes.py -h
to see them. Include this in your repository to start using recipes.
recipes/
Recipes in the recipe engine. These are either example recipes, or recipes which are used to test the engine (see run_test.py to see these run)
recipe_modules/
Built in recipe modules. These are very useful when writing recipes; take a look in there, and look at each of their examples
subfolders to get an idea how to use them in a recipe.
recipe_engine/
The core functionality of the recipe engine. Noteworthy files include:
main.py
-- The main entrypoint for the recipe engine.recipes_cfg.proto
-- The protobuf file which defines the format of a recipes.cfg
file.third_party/
-- third_party code which is vendored into the recipe engine.recipe_api.py
-- The api exposed to a recipe module.unittests
-- Unittests for the engine.There are also several files which correspond to a subcommand of recipes.py; run
, and autoroll
are some examples.
unittests/
Somewhat poorly named, these are higher level integration tests.
user.email
and user.name
are configured in git config
.Run the following to setup the code review tool and create your first review:
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git $HOME/src/depot_tools export PATH="$PATH:$HOME/src/depot_tools" git checkout -b work origin/master # hack hack git commit -a -m "This is awesome" # This will ask for your Google Account credentials. git cl upload -s -r joe@example.com # Wait for approval over email. # Click "Submit to CQ" button or ask reviewer to do it for you. # Wait for the change to be tested and landed automatically.
Use git cl help
and git cl help <cmd>
for more details.