What is a Data Pipeline?

The classic pattern is ETL - extract from a source, transform it into the target shape, then load it into a destination - though modern cloud-native pipelines often run ELT instead, loading raw data first and transforming it afterward inside the destination system, which takes advantage of cheap storage and powerful compute in warehouses. Either way, a pipeline is really a chain of dependent jobs, and its reliability depends on how well failures in any one stage are detected and handled rather than silently propagated. Good pipelines are idempotent - running the same batch twice produces the same result rather than duplicating data - and observable, meaning each stage emits enough logging or metrics that a failure can be localized quickly instead of requiring someone to re-run the whole thing blind. AI agents increasingly sit inside pipelines not just as data movers but as decision points - classifying, enriching, or validating records mid-stream in ways a fixed transformation script couldn't.