FreeSolo-L-743M

Paper · GitHub · Models · Usage

RoPE-free long-range attention, trained at 4k with zero-shot length extrapolation.

FS-L retains a bounded pool of chunks by key norm and recency, then reads four selected chunks. Its attention scales linearly with sequence length. This 743M research base model was trained on FineWeb-Edu at a 4,096-token context.

Try it

hf download zekun-shi/FreeSolo-L-743M freesolo-source.zip --local-dir ./FreeSolo-L-743M
python -m pip install ./FreeSolo-L-743M/freesolo-source.zip
freesolo-generate --model zekun-shi/FreeSolo-L-743M --output generation.json

The short retrieval example asks for the hidden number 5437923. Start with CUDA-enabled PyTorch; Usage gives the tested installation and speed-replication command.

The idea

FreeSolo learns simple chunk mean keys that support search, norm pruning, and hierarchical summarization. RoPE stays in the local window; long-range attention reads selected original tokens without rotating their queries and keys.

Results

Context	2k	4k	8k	16k
RULER mean	37.26	31.76	25.95	22.71

Common 12-task set at every length, excluding QA-2. 100 examples per task, seed 42. These are checkpoint evaluation results; the figures show the paper's model-family comparison.

This is a base model without instruction tuning. Retrieval is task dependent, especially when several facts or values must be combined. Training details, artifact checksums, and citation are in Usage.