_LLM

LLM use-cases

Large Language Models can fulfill a wide variety of tasks.

Illustration from https://aclanthology.org/2022.emnlp-main.340.pdf

And they can be used for very different semantic areas with the same business, so the performance may vary significantly.

Screenshot from 2024-04-06 20-44-51.png

illustration from https://arxiv.org/pdf/2401.15042.pdf

Observability

Building a RAG app from scratch with Langchain (by svpino)

Advanced Langchain Chains
Extract & transcribe Audio from Youtube Video (using youtube and openai-whisper libs)
https://www.youtube.com/watch?v=BrsocJb-fAo
https://github.com/svpino/youtube-rag/blob/main/rag.ipynb

Evaluating an LLM-powered RAG application automatically (by svpino)

Generate a test-set from the documents
Automatically evaluate the RAG system using Giskard
Integrate the evaluation with PyTest in order include it in CI/CD steps
https://www.youtube.com/watch?v=ZPX3W77h_1E
https://github.com/svpino/llm/blob/main/evaluation/notebook.ipynb

Builing a RAG app from scratch with LlamaIndex (by Arize)

Very simple RAG with LlamaIndex
Questions generation using Phoenix (but LlamaIndex can do it as well) for evaluation
Compute Retrieval scores: ndcg_score@K (with sklearn.metrics), precision@K and Hit-rate
Compute Response score: Q&A correctness, Hallucinations
https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/evals/evaluate_rag.ipynb
https://www.youtube.com/watch?v=LrMguHcbpO8

LLM brenchmarks review and origin

Describe 6 of the most famous LLM benchmarks (Glue & SuperGLUE, Adversarial NLI, Big Bench, MMLU, HELM, LM Evaluation Harness)
Explain why we need to use these benchmarks instead of the old NLP metrics and benchmarks
Part1: https://medium.com/@myschang/evaluation-of-large-language-model-llm-introduction-9343424ad253
Part2: https://medium.com/@myschang/benchmark-of-llms-part-1-glue-superglue-adversarial-nli-big-bench-8d1aed6bae12
Part3: https://medium.com/aimonks/benchmark-of-llms-part-2-mmlu-helm-eleuthera-ai-lm-eval-e6fc54053e3d

Phoenix demo

This video actually gives a great demo on how to identify problematic queries on a RAG app.
https://www.youtube.com/watch?v=hbQYDpJayFw
http://bit.ly/llama-index-phoenix-tutorial (associated notebook)
https://phoenix-demo.arize.com/projects (live demo of Phoenix)

LLM / RAG / Evaluation overview

Great overview of the LLM + RAG + Evaluation steps
https://arize.com/blog-course/large-language-model-monitoring-observability/

Data Science clearly explained

Lots of videos explaining many ML concepts
https://www.youtube.com/@statquest/videos

_LLM

LLM use-cases

Fine-tuning

Summarization

Knowledge Graphs

Preference Alignment algorithms

RAG - Retrieval Augmented Generation

Observability

Observability - Traces & Spans

Observability - Evaluations & Golden Dataset

Observability - Search & Retrieval

Observability - Prompt Engineering

Building a RAG app from scratch with Langchain (by svpino)

Evaluating an LLM-powered RAG application automatically (by svpino)

Builing a RAG app from scratch with LlamaIndex (by Arize)

LLM brenchmarks review and origin

Phoenix demo

LLM / RAG / Evaluation overview

Data Science clearly explained

LLM use-cases

LLM related techniques

Observability

Observability - Evaluations & Golden Dataset

Building a RAG app from scratch with Langchain (by svpino)

Evaluating an LLM-powered RAG application automatically (by svpino)

Builing a RAG app from scratch with LlamaIndex (by Arize)

LLM brenchmarks review and origin

Phoenix demo

LLM / RAG / Evaluation overview

Data Science clearly explained