Dense Encoding
November 20, 2024 |
permanent
tags :
- similar
- One Hot Encoding
Definition #
Dense encoding is a type of encoding where each category is represented by a single number. This is in contrast to one hot encoding where each category is represented by a vector of numbers.
- few dimensions, 378, 768, 1024, 2048 etc
- less zeros in every dimension
- richer representation
- good generation of data
- at the cost of hallucination
- LLM models learns these dimensions by backpropagation, to get them right to capture the meaning or good generation of data.
