---
tags: 
parents: 
date/time: 2024-11-02T00:58:00
---


https://www.youtube.com/watch?v=9vM4p9NN0Ts stanford course




http://www.incompleteideas.net/


# Overview


- ## Architecture

- ## Training algorithm/loss

- ## Data

- ## Evaluation

- ## Systems


# Notes

pre-training (upto GP3) and pos-training(>gpt3)

**Overview**

- Language Models:  are essentially probability distributions over sequence of tokens or words $P(x_1, ... x_L )$. 
for example:
- $P(the,mouse,ate,the,cheese)=0.2$
- $P(the,the,mouse,ate,the,cheese)=0.0001$ syntactic knowledge
- $P(the,cheese,ate,the,cheese)=0.0001$ semantic knowledge
- LLM are generative since given the distribution one can sample from it
- LLMs are generally Autoregressive (AR): $$ p(x_1,x_2,...,x_L)=p(x_1)p(x_2|x_1)p(x_3|x_1,x_2),...p(x_L|x_1:x_{L-1})= \prod_i p(x_i|x_1:x_{i-1})$$

**Task**: predict next word: