Pipelines: How to Setup
- Updated On 06 Jan 2021
- 3 Minutes To Read
What is a Pipeline?
Pipelines let you automate multi-stage workflows such as your build, test, or Event Data Pipelines. Pipelines are versioned by default, so that you can easily measure process changes & roll back as needed.
Pipelines are Cloud Native and Serverless, allowing you to run them from Slack or trigger them from manually, from Github, GitLab, or BitBucket. External triggering coming soon.
Get Started with Pipelines
1. Init a Pipeline Op
To start using the Pipeline feature, you will use the Ops init flow on the CLI. Run the following command in your terminal
- You will be prompted to choose between different types of Ops.
pipelineand press enter.
- Choose a name, description and a version for your Pipeline.
You can work on your Pipeline by editing the
ops.yml file in the newly created project directory.
Refer below to the "Example Use Case" section for a walk through of a common uses case using Pipelines.
2. Build the Pipeline Op
Before you can run your Pipeline you will need to build it.
- In your terminal run
ops build pipeline_name
(replace "pipeline_name" by your actual pipeline name)
For more information on Building click the link.
3. Test the Pipeline Op
After the Pipeline has been successfully built you may want to test it before deciding to publish it. You can test the Pipeline on your local docker engine via the Ops CLI command
ops run pipeline_name
4. Publish the Pipeline Op
Once built, publish your Pipeline to run from the CLI, Slack and/or the CTO.ai Dashboard.
- In your terminal run
ops publish pipeline_name
For more information on Publishing click the link.
Congratulations! Your Pipeline is now built, published and ready for you and your teammates to run it!
5. Start or Run a Pipeline Op
Once the Pipeline has been published it can be started or run from the CLI, Slack and/or the Ops Dashboard. Below walks you through how to run from each.
Using the CLI to start the Pipeline local via Docker
- Go to your terminal
- run the following command
ops start pipeline_name
- you will see all your logs displayed in your local terminal
- Associate your Ops user & team with a Slack Workspace and Channel (if you haven't already)
- Run directly in the associated Channel with the command
/ops run pipeline_name
- Through the CTO.ai Dashboard
- Go to cto.ai
- Sign in and go to the Dashboard
- Pick the team you have published your Pipeline to
- Access the Pipelines tab which will list all of your current Pipelines
- Select the Pipeline of your choice and click the
Using the CTO.ai Dashboard
The image below shows a list of Pipelines in the Dashboard.
You can see:
- The list of Pipelines created
- Who ran them
- When it was Run
- Status (Success/Complete, Running, Failed)
You can click into each Pipeline and see a detailed view- Steps, Logs, and other Pipeline specific information
Example Use Case
Let's walk through an example of a common CICD pipeline with 3 steps- Clone, Test, and Deploy.
Below is the sample code for this example.
version: "1" pipelines: - name: example-pipeline:0.1.2 description: Example Pipeline jobs: - name: clone description: clone the application repo sdk: "2" packages: - git steps: - git clone https://github.com/shiehn/documentation-example.git /tmp/state/documentation-example - echo "successfully cloned application code" - name: test description: test the code sdk: "2" packages: - golang-go steps: - cd /tmp/state/documentation-example - go test . - echo "tests completed succesfully" - name: deploy description: build and deploy the artifact sdk: "2" packages: - golang-go steps: - cd /tmp/state/documentation-example - go build -o $HOME/final-build - echo "the final-build has been created and is ready to deployed!" # NOTE: it would be very common to deploy this 'final-build' to a server
There are 3 steps:
Step 1: Clone: Clone a public Golang repo into /tmp/state.
/tmp/state is a special directory which is persisted across all 'steps'- if you put something in /tmp/state in step one it can be accessed in step two
Step 2: Test: Simply run the Golang tests. When the tests succeed it will return a zero exit code and the Pipeline will continue.
In the event the tests fail it will return a non-zero exit code and the entire Pipeline will fail
Step 3: Deploy: This is placeholder code. This is where you would place code to deploy your artifact, website, etc to a server. For demo reasons this is commented out, but put your own logic here.
🚀 What's Next
Contact Support — we're here to help!