Data pipeline

How it is used

Data engineers build and maintain pipelines; analysts notice when one breaks or runs late. You build, schedule, monitor and fix a pipeline, and people say a pipeline 'failed', 'ran' or 'is down'. In context often just 'pipeline'.

Example sentences

1. The dashboard is empty because the data pipeline failed overnight.

2. We're building a new data pipeline to bring in app events every hour.

Common collocations

• build a data pipeline
• pipeline failure
• maintain a pipeline
• pipeline runs daily

Related terms

• ETLShort for extract, transform, load: the process of taking data out of source systems, cleaning and reshaping it, and loading it into a warehouse.
• data warehouseA central database built for reporting and analysis that collects historical data from many systems and is designed for large read queries rather than small updates.
• batch processingProcessing data in large groups at scheduled times, for example once every night, rather than handling each record the moment it arrives.
• data lineageA record of where data comes from, which steps and systems it passes through and how it changes along the way, right up to the reports that use it.

This word in other fields

The same English word can have a different meaning in another field:

• data pipeline · Data Science & Analytics

More terms: Machine Learning and Data Processing

• artificial intelligence
• machine learning
• model
• dataset
• training data
• label
• train
• supervised learning
• neural network
• deep learning
• loss function
• overfitting