End-to-end tests with Great Expectations and Pytest (e2e-etl-python)
End-to-end tests for ETLs quickstarter project
Purpose of this quickstarter
This is a python based quicktarter intended to develop end-to-end tests for data pipelines. In order to do that it uses two testing technologies: 1. Great Expectations, meant for data transformation testing data within relational tables. e.g.: You could test the schema of a database, the number of rows, that a specific column has no null values, etc 2. Pytest together with Boto it allows for testing etl triggers, notification system, content of S3 buckets, etc
What files / architecture is generated?
├── Jenkinsfile - This file contains Jenkins stages. ├── README.md ├── environments │ ├── dev.json - This file describes parameters for the development AWS environment. │ ├── test.json - This file describes parameters for the test AWS environment. │ └── prod.json - This file describes parameters for the production AWS environment. ├── tests - This folder contains the root for test-kitchen │ ├── acceptance/great_expectations - This folder contains the Great Expecations tests to test │ └── acceptance/pytest - This folder contains the pytest tests to test
Usage - how do you start after you provisioned this quickstarter
Check the README.md file at root level for further instructions after the quickstarter has been provisioned.
Builder agent used
This quickstarter uses terraform Jenkins agent.