Data Engineering for AI: Pipelines, Vector Stores and Data Quality

A premium, advanced course on the data foundation that every serious AI system depends on, updated for 2026. This is not a retrieval-and-generation (RAG) course — it is the data engineering underneath: how raw data becomes trustworthy fuel for LLMs, agents and vector search. You will master batch versus streaming paradigms, ETL/ELT and orchestration with Apache Airflow, Dagster and dbt, lakehouse architecture with Parquet and Apache Iceberg, ingestion from diverse sources with change data capture and schema evolution, data quality and validation with Great Expectations, data contracts and governance, chunking and preprocessing for LLM pipelines, embeddings pipelines at scale, and vector databases in depth (pgvector, Pinecone, Qdrant, Weaviate) including HNSW and IVF index internals, hybrid search, sharding and metadata filtering. It closes with metadata and lineage, PII detection and anonymization, GDPR-compliant pipelines, cost and performance engineering, pipeline monitoring, and a comprehensive final assessment. Real Python and SQL throughout.