Expert, AI Engineer

## Ôn thi & Chuẩn bị cho vị trí Expert, AI Engineer ### 📚 Kiến thức nền tảng cần ôn lại #### **1. Spark Deep Dive (Priority: CRITICAL)** ```python # Topics cần master: # 1. Spark Architecture # - Driver, Executors, Cluster Manager # - Task, Stage, Job relationship # - Shuffle behavior và optimization # 2. DataFrame API & Optimization # - Catalyst Optimizer # - Tungsten Engine (off-heap memory) # - Adaptive Query Execution (AQE) # 3. Performance Optimization # - Partitioning strategies # - Broadcast joins vs shuffle joins # - Caching (persist vs cache) # - Data skew handling (salt, etc.) # Code example - Optimization techniques: df.filter(col("date") > "2024-01-01") # Predicate pushdown df.repartition(200) # Optimal partitioning df.join(broadcast(small_df), "id") # Broadcast small table df.cache() # Cache for reuse ``` #### **2. Lakehouse Architecture (Priority: HIGH)** | Concept | Databricks | Open Source | |---------|------------|-------------| | Storage | S3/ADLS | S3/ADLS/GCS | | Format | Delta Lake | Delta/Iceberg/Hudi | | Catalog | Unity Catalog | Hive Metastore + | | Processing | Spark/DBSQL | Spark/Flink/Trino | | Governance | Unity Catalog | Apache Atlas, OpenMetadata | **Key Delta Lake features:** - ACID transactions - Time travel (versioning) - Z-Ordering for data skipping - Merge, upsert, delete #### **3. MLOps & LLM Engineering (Priority: HIGH)** ``` ML Pipeline Architecture: ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Data Prep │───▶│ Training │───▶│ Evaluation │ └─────────────┘ └─────────────┘ └─────────────┘ │ ▼ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Monitoring │◀───│ Deployment │◀───│ Validation │ └─────────────┘ └─────────────┘ └─────────────┘ LLM Pipeline: ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Embedding │───▶│ Vector DB │───▶│ Retrieval │ └─────────────┘ └─────────────┘ └─────────────┘ │ ▼ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Response │◀───│ Prompt Eng. │◀───│ Context │ └─────────────┘ └─────────────┘ └─────────────┘ ``` **RAG Implementation:** ```python # Conceptual RAG flow documents = load_documents(source) embeddings = embedding_model.encode(documents) vector_db.insert(embeddings, documents) # At inference: query_embedding = embedding_model.encode(user_query) retrieved_docs = vector_db.search(query_embedding, k=5) context = format_context(retrieved_docs) response = llm.generate(prompt.format(context, query)) ``` #### **4. Cloud & DevOps (Priority: MEDIUM-HIGH)** | Area | AWS | Azure | GCP | |------|-----|-------|-----| | Data Lake | S3 | ADLS Gen2 | GCS | | Compute | EMR, Glue | Databricks, Synapse | Dataproc, BigQuery | | Orchestration | MWAA | Data Factory | Cloud Composer | | Streaming | Kinesis | Event Hubs | Pub/Sub | | IaC | Terraform | Terraform/Bicep | Terraform | --- ### 📖 Tài liệu tham khảo khuyên dùng #### **Books:** | Book | Link | Priority | |------|------|----------| | Learning Spark | https://www.oreilly.com/library/view/learning-spark-2nd/9781492050032/ | ⭐⭐⭐ | | Designing Data-Intensive Applications | Amazon | ⭐⭐⭐ | | The Databricks Lakehouse Platform | databricks.com | ⭐⭐⭐ | #### **Online Courses:** | Course | Platform | For | |--------|----------|-----| | Data Engineering with Databricks | Databricks Academy (Free) | Lakehouse | | Spark Performance Optimization | Databricks | Spark tuning | | MLOps Specialization | Coursera/AWS | ML pipelines | | Generative AI with LLMs | DeepLearning.AI | LLM basics | #### **Techcombank-specific resources:** - Techcombank Engineering Blog (techcombank.com) - Techcombank tech talks trên YouTube - Techcombank Engineering trên Medium/LinkedIn --- ### 🗓️ Lộ trình chuẩn bị 2 tuần #### **Week 1: Foundation & Deep Dive** ``` Day 1-2: Spark Architecture ├── Watch: Spark internals videos (Databricks blog) ├── Read: Learning Spark chapters 3-6 └── Practice: Spark join strategies, partitioning Day 3-4: Lakehouse & Databricks ├── Read: Delta Lake documentation ├── Watch: Unity Catalog architecture └── Practice: Delta Lake time travel, Z-ordering Day 5-6: MLOps & ML Pipelines ├── Read: MLOps principles ├── Watch: End-to-end ML pipeline design └── Practice: Basic MLflow tracking Day 7: Review & gaps identification ``` #### **Week 2: AI/LLM & System Design** ``` Day 8-9: LLM & RAG ├── Read: RAG architecture patterns ├── Watch: Vector embeddings, similarity search ├── Practice: Simple RAG implementation └── Explore: ChromaDB, Pinecone, Weaviate Day 10-11: System Design Practice ├── Practice: Design data platform questions ├── Practice: Design ML pipeline questions ├── Review: Common design patterns └── Prepare: Your own architecture stories Day 12-13: Interview Prep ├── Review: STAR stories ├── Practice: Behavioral questions ├── Prepare: Questions for interviewer └── Mock: System design with friend Day 14: Final Review ├── Review: All technical notes ├── Check: Logistics └── Rest: Be fresh for interview ``` --- ### 🔧 Hands-on Labs khuyên làm ```python # Lab 1: Spark Optimization Exercise # - Create skewed dataset # - Measure baseline performance # - Apply: salt, broadcast, partition optimization # - Compare before/after # Lab 2: Delta Lake Time Travel # - Write to Delta table # - Query historical versions # - Perform upsert/merge # - Use Z-ordering for query optimization # Lab 3: Simple RAG Pipeline # - Load documents # - Create embeddings (OpenAI or local) # - Store in vector DB # - Implement retrieval + generation # Lab 4: IaC with Terraform # - Define S3 bucket, IAM roles # - Deploy Databricks workspace # - Version control infrastructure ```