Running tests
In order to run a test you need to install the Lila CLI. If you haven't already, see the Install section.
To run your tests, you need to use lila run.
Local run or cloud run?
Lila runs tests locally.
Existing AI testing tools usually run over publicly available URLs which suck in the sense that it requires a deployment. Most of the times you are testing PRs, or private environments. Lila was design so that it runs the tests on a local browser, meaning localhost, staging.internal or even prod.com is reachable.
That's why playwright is required by the CLI.
When you run lila run, a browser runs in the background and it communicates with an cloud orchestrator (the Lila app) that guides the test execution.
This enables:
- Running Lila in localdev.
- Add Lila tests as PR checks.
- Monitoring production deploymments.
- Running Lila as a pre-deployment requirement before prod deploy.
- etc.
Running options
A single file
# Running a single test case
lila run some-test.yaml
Running an entire directory
# Running all yaml test files in the directory, recurisvely.
lila run tests/admming-tests
Running by tag
Each test case can define tags you can use later to run
# Running all yaml test files in the directory that have the 'smoke' and 'admin' tag
lila run tests --tags smoke,admin
Running by excluding tag
Each test case can define tags you can use later to exclude
# Running all yaml test files in the directory that do not have tag 'login-required'
lila run tests --exclude-tags login-required
If a test has a tag found in --tags and a tag found in --exclude-tags, it will not run.
For more information on the different running options, check the CLI documantation (here)[]