nanoFLY decoder: MaleCNS v1.0 connectome as a frozen recurrent layer

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+connectome.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,115 @@
+---
+license: cc-by-4.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+datasets:
+- roneneldan/TinyStories
+tags:
+- connectome
+- reservoir-computing
+- echo-state-network
+- fruit-fly
+- drosophila
+- malecns
+- tinystories
+- custom_code
+---
+
+# nanofly-decoder-en
+
+![MaleCNS soma cloud coloured by this model's neuron states](connectome.png)
+
+<sub>43,993 neurons at their measured MaleCNS v1.0 coordinates, coloured by this checkpoint's state at one tick while writing *"…there was a girl named Lily."* Orange excited, blue inhibited, grey at rest. Frontal view; the optic lobes flank the central brain.</sub>
+
+A language model whose recurrent layer is the measured wiring of a fruit fly. The connectome is a frozen [echo state network](https://en.wikipedia.org/wiki/Echo_state_network) reservoir — **no synapse is trained**. Only the input projection, per-neuron gain/bias/leak, one global scale and the readout learn.
+
+- **Code:** [github.com/igorktech/nanoFLY](https://github.com/igorktech/nanoFLY)
+- **Russian sibling:** [`igorktech/nanofly-decoder-ru`](https://huggingface.co/igorktech/nanofly-decoder-ru)
+
+## Usage
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+repo = "igorktech/nanofly-decoder-en"
+tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).eval()
+model = model.to("cuda" if torch.cuda.is_available() else "cpu")
+
+ids = tok("Once upon a time, there was a", return_tensors="pt").input_ids
+ids = torch.cat([torch.tensor([[model.config.bos_token_id]]), ids], dim=1).to(model.device)
+out = model.generate(ids, max_new_tokens=80, do_sample=True, top_k=50, temperature=0.7)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+
+- Prepend BOS: every training example started with it.
+- Greedy: `do_sample=False`, and drop `top_k` or `transformers` warns.
+- Beam search and assisted generation are unsupported (stateful model). Greedy, sampling, top-k, top-p work.
+- ~960 forward passes/s on an RTX 5080, ~10/s on a laptop CPU.
+
+## Architecture
+
+| | |
+|---|---|
+| connectome | MaleCNS v1.0 central brain — `cb_sensory`, `cb_intrinsic`, `visual_projection`, `descending_neuron`, `ascending_neuron` |
+| neurons / edges | 49,393 / 9,055,280 signed (623,728 dropped: modulatory or unknown transmitter) |
+| edge weight | sign of the presynaptic transmitter × synapse count, rows normalised to unit absolute weight. ACh +1; GABA, Glu, His −1; others 0 |
+| token input | 11,434 sensory-facing neurons, 8-slot delay line (slot *j* gets token *t−j*). No attention, no positional encoding |
+| held out | the 2,635 ORNs stay out of the token input, so the encoder-decoder variant can start from these weights |
+| dynamics | `x ← (1−a)·x + a·tanh(ρ·g·(Wx) + u + b)`, 2 ticks per token; `a` learned per neuron (init 0.5), `ρ` learned global (init 1.0 → **4.26**) |
+| readout | all 49,393 states → `Linear(49393→256)` → `LayerNorm` → `Linear(256→1024)` |
+| trainable | **16.25M** — readout 12.90M, input projection 2.93M, embedding 0.26M, per-neuron scalars 0.15M |
+
+## Training
+
+| | |
+|---|---|
+| data | TinyStories, 98,024 stories / 1,976 held out, 28,417,062 tokens, byte-level BPE vocab 1,024 |
+| objective | next-token cross entropy, truncated BPTT over 32-token windows, state carried across windows |
+| optimiser | AdamW — body 2e-3 (no decay), readout 5e-4 (decay 0.01), warmup 200 then cosine to 10%, clip 1.0 |
+| schedule | 3 epochs, 22,115 updates, batch 128 |
+| hardware | 1 × RTX 5080, 1.74 h at ~13,700 tok/s |
+
+## Evaluation
+
+Held-out loss. Identical data, recipe, seed and budget; only the recurrent layer differs:
+
+| recurrent layer | val loss | ppl | trainable |
+|---|---|---|---|
+| degree-matched shuffled wiring, frozen | 1.979 | 7.2 | 16.25M |
+| **this model** — real connectome, frozen | **1.933** | **6.9** | 16.25M |
+| real connectome, synapse strengths trained | 1.913 | 6.8 | 25.30M |
+
+The control keeps every neuron's in- and out-degree, the transmitter signs and Dale's law, and randomises only which neuron pairs with which. Two results, each holding at all ten intermediate evaluations:
+
+- **The fly's specific wiring is worth 0.046 nats** over a random graph with its degrees — and that margin grows through training (0.031 → 0.046).
+- **Making all 9,055,280 synapse strengths trainable is worth 0.020 nats** on top — and that margin shrinks through training (0.032 → 0.020), at 4× the compute. It buys faster convergence more than a higher ceiling. Trained strengths stay close to the measured ones (Pearson r = 0.94).
+
+One seed per condition, so treat the margins as indicative rather than significant. Validation was still improving at the end in all three runs; none is converged.
+
+Greedy samples, prompt in bold:
+
+> **Once upon a time, there was a** little girl named Lily. She loved to play outside in the sunshine. One day, she saw a big, red ball that was very pretty. She wanted to play with it, but it was too high for her to reach.
+
+> **Tom and his dog** were best friends. They liked to play in the park. One day, they saw a big dog with a big bag.
+
+## Limitations
+
+- 16M trainable parameters over 85M tokens of children's stories. It writes about Lily and Tom and nothing else.
+- 8-token delay line plus a short leaky recurrent memory. Loss stops improving with context by roughly position 32; names and objects drift within a paragraph.
+- A `tanh` rate neuron is not a spiking model: no spikes, no synaptic delays, no neuromodulation — modulatory edges are removed outright.
+- Central brain only; the optic lobes and ventral nerve cord of the 166,700-neuron CNS are absent.
+- Synapse count is a proxy for strength, and rows are normalised. Neither is measured physiology — see the third row of the evaluation table for what happens when the strengths are fit to the task instead.
+
+## Credits
+
+- **Connectome:** MaleCNS v1.0 — FlyEM / HHMI Janelia, University of Cambridge, MRC LMB, Google Research. CC BY 4.0. The published buffers derive from that release; keep the attribution when redistributing.
+- **Transmitter signs:** Shiu et al., *Nature* 2024.
+- **Connectome as reservoir:** Costi, Hadjiivanov, Dold, Hale, Izzo, 2025.
+- **Prior art:** [`ngxson/fly-llm-hf`](https://huggingface.co/ngxson/fly-llm-hf), whose graph subset this reproduces.
+- **Data:** Eldan and Li, *TinyStories*, 2023.
+
+Weights CC BY 4.0, matching the connectome. Modeling code Apache-2.0.

config.json

@@ -0,0 +1,44 @@
+{
+  "arch": "decoder",
+  "architectures": [
+    "FlyForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_fly.FlyConfig",
+    "AutoModel": "modeling_fly.FlyModel",
+    "AutoModelForCausalLM": "modeling_fly.FlyForCausalLM"
+  },
+  "bos_token_id": 1,
+  "connectome": "MaleCNS v1.0, FlyEM / HHMI Janelia, University of Cambridge, MRC LMB, Google Research. CC BY 4.0.",
+  "d_emb": 256,
+  "delay": 8,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_type": "lowrank",
+  "min_syn": 1,
+  "mode": "gains",
+  "model_type": "fly",
+  "modulatory_sign": 0.0,
+  "n_edges": 9055280,
+  "n_neurons": 49393,
+  "n_news_input": 0,
+  "n_reserved": 2635,
+  "n_token_input": 11434,
+  "news_dim": 0,
+  "news_encoder": "none",
+  "news_glom": 0,
+  "news_group": "orn",
+  "news_mode": "glomeruli",
+  "news_prefix": "",
+  "pad_token_id": 0,
+  "readout": "all",
+  "readout_rank": 256,
+  "readout_size": 49393,
+  "source_repo": "",
+  "ticks": 2,
+  "tie_word_embeddings": false,
+  "token_input": "cb_sensory,visual_projection",
+  "transformers_version": "5.17.0",
+  "use_cache": true,
+  "vocab_size": 1024
+}

configuration_fly.py

@@ -0,0 +1,103 @@
+"""Config for the Fly connectome language models (published with trust_remote_code)."""
+from transformers import PretrainedConfig
+
+CONNECTOME = ("MaleCNS v1.0, FlyEM / HHMI Janelia, University of Cambridge, MRC LMB, "
+              "Google Research. CC BY 4.0.")
+
+
+class FlyConfig(PretrainedConfig):
+    """A fruit-fly connectome as the recurrent layer of a language model.
+
+    Two architectures share this config:
+
+    * ``decoder`` — token in, token out, nothing else. Comparable to ngxson/fly-llm-hf.
+    * ``encoder_decoder`` — the post being answered is encoded once by a frozen sentence encoder
+      (``news_encoder``) and injected as a constant current into the olfactory neurons for the whole
+      generation, so the fly "smells" the post while it writes. Only the projection into those
+      neurons is trained here; the sentence encoder is a separate model and is not shipped.
+
+    The recurrent weights are anatomy, not learned: ``w_values`` are signed, row-normalised synapse
+    counts between the neurons of the release above. ``mode`` records whether the published values
+    are the raw connectome (``gains``) or connectome-masked learned strengths (``edges``).
+    """
+
+    model_type = "fly"
+    keys_to_ignore_at_inference = ["cache_params"]
+
+    def __init__(
+        self,
+        arch="decoder",
+        vocab_size=2048,
+        d_emb=256,
+        delay=8,
+        ticks=2,
+        mode="gains",
+        token_input="cb_sensory",
+        n_neurons=1,
+        n_edges=0,
+        n_token_input=1,
+        n_news_input=0,
+        readout="all",
+        readout_size=1,
+        readout_rank=256,
+        head_type="lowrank",
+        news_dim=0,
+        news_group="orn",
+        n_reserved=0,
+        news_mode="direct",
+        news_glom=0,
+        news_encoder="none",
+        news_prefix="",
+        min_syn=1,
+        modulatory_sign=0.0,
+        connectome=CONNECTOME,
+        source_repo="",
+        use_cache=True,
+        pad_token_id=0,
+        bos_token_id=1,
+        eos_token_id=2,
+        tie_word_embeddings=False,
+        **kwargs,
+    ):
+        self.arch = arch
+        self.vocab_size = vocab_size
+        self.d_emb = d_emb
+        self.delay = delay
+        self.ticks = ticks
+        self.mode = mode
+        self.token_input = token_input      # which population the delay line writes into
+        self.n_neurons = n_neurons
+        self.n_edges = n_edges
+        self.n_token_input = n_token_input
+        self.n_news_input = n_news_input
+        self.readout = readout
+        self.readout_size = readout_size
+        self.readout_rank = readout_rank
+        self.head_type = head_type          # lowrank: Linear->LayerNorm->Linear, linear: LayerNorm->Linear
+        self.news_dim = news_dim
+        self.news_group = news_group        # population carved out for the post channel
+        self.n_reserved = n_reserved        # its size, even when this variant does not use it
+        self.news_mode = news_mode
+        self.news_glom = news_glom
+        self.news_encoder = news_encoder
+        self.news_prefix = news_prefix
+        self.min_syn = min_syn
+        self.modulatory_sign = modulatory_sign
+        self.connectome = connectome
+        self.source_repo = source_repo
+        self.use_cache = use_cache
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )
+
+    @property
+    def hidden_size(self):
+        return self.readout_size
+
+    @property
+    def conditional(self):
+        return self.arch == "encoder_decoder" and self.news_dim > 0

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.17.0",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de8d7646db82560e0f3c4ed706d7f00c8b2bf48476918e87a013f7aee1f00dc3
+size 138110864

modeling_fly.py

@@ -0,0 +1,324 @@
+"""Fly: a fruit-fly connectome as the recurrent layer of a language model.
+
+State update, one line per neuron i, repeated `ticks` times per token:
+
+    x_i <- (1 - a_i) * x_i + a_i * tanh( rho * g_i * sum_j W_ij x_j + u_i + b_i )
+
+`W` is anatomy and is not trained here: signed, row-normalised synapse counts from MaleCNS v1.0.
+`u` is the input current — the token embedding fanned into sensory neurons through a delay line,
+plus, for the encoder-decoder model, a constant current on the olfactory neurons that carries the
+post being answered. The readout is a linear head on a subset of neurons (all, descending, motor…).
+
+This file is the inference/fine-tuning copy that ships with the weights. Training lives in the
+project repo (see `config.source_repo`), where the connectome is rebuilt from the release tables.
+"""
+from dataclasses import dataclass
+from typing import Optional, Tuple
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import GenerationMixin, PreTrainedModel
+from transformers.utils import ModelOutput
+
+from .configuration_fly import FlyConfig
+
+# This file is the single source for both published variants. `export_hf.py::write_code` ships it
+# unchanged for the encoder-decoder and, for the decoder, rewrites the `Fly` prefix to `Fly`
+# and drops the conditional tail, which starts at the top-level definition of `_hash_embed` and runs
+# to the end of the file. Three invariants:
+#   1. the prefix is spelled `Fly` (class) and `fly` (module) nowhere except where it must
+#      be renamed — never put either literal in a URL or a message meant to survive;
+#   2. everything below that definition belongs to the post channel, and nothing above it refers to
+#      anything below;
+#   3. the relative import above is rewritten too, so the pair always travels together.
+# The export writes the folder, loads it back and compares logits, so a broken rename fails loudly —
+# as it did the first time this very comment was written with the marker spelled out in full.
+
+
+class FlyCache:
+    """What carries over between generate() steps: neuron state and the delay line of token ids."""
+
+    is_compileable = False
+
+    def __init__(self, state, last_tokens, seq_len=0):
+        self.state = state              # [B, N] neuron activations
+        self.last_tokens = last_tokens  # [B, delay], most recent token first
+        self.seq_len = seq_len
+
+    def get_seq_length(self, layer_idx=0):
+        return self.seq_len
+
+    def get_max_cache_shape(self):
+        return None
+
+    def reorder_cache(self, beam_idx):
+        self.state = self.state.index_select(0, beam_idx.to(self.state.device))
+        self.last_tokens = self.last_tokens.index_select(0, beam_idx.to(self.last_tokens.device))
+        return self
+
+
+@dataclass
+class FlyOutput(ModelOutput):
+    last_hidden_state: torch.FloatTensor = None
+    cache_params: Optional[FlyCache] = None
+    hidden_states: Optional[Tuple[torch.FloatTensor, ...]] = None
+
+
+@dataclass
+class FlyCausalLMOutput(ModelOutput):
+    loss: Optional[torch.FloatTensor] = None
+    logits: torch.FloatTensor = None
+    cache_params: Optional[FlyCache] = None
+    hidden_states: Optional[Tuple[torch.FloatTensor, ...]] = None
+
+
+class _SpMM(torch.autograd.Function):
+    """y = W @ x with W a fixed sparse CSR matrix; the backward pass reuses a cached transpose."""
+
+    @staticmethod
+    def forward(ctx, x, W, WT):
+        ctx.WT = WT
+        return torch.sparse.mm(W, x)
+
+    @staticmethod
+    def backward(ctx, g):
+        return torch.sparse.mm(ctx.WT, g.contiguous()), None, None
+
+
+class FlyPreTrainedModel(PreTrainedModel):
+    config_class = FlyConfig
+    base_model_prefix = "brain"
+    supports_gradient_checkpointing = False
+    _is_stateful = True
+    _no_split_modules = []
+
+    @classmethod
+    def _supports_default_dynamic_cache(cls):
+        # the model keeps its own FlyCache; generate() must not build a KV cache for it
+        return False
+
+    def _init_weights(self, module):
+        if isinstance(module, nn.Linear):
+            nn.init.normal_(module.weight, std=min(0.02, module.in_features ** -0.5))
+            if module.bias is not None:
+                nn.init.zeros_(module.bias)
+        elif isinstance(module, nn.Embedding):
+            nn.init.normal_(module.weight, std=1.0)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+        elif isinstance(module, FlyModel):
+            # the raw parameters need their own init: a zeroed in_proj means no current ever reaches
+            # the neurons, so a model built from the config alone would be deaf. A reservoir wants
+            # O(1) drive, not transformer-scale.
+            nn.init.normal_(module.in_proj, std=module.config.d_emb ** -0.5)
+            nn.init.ones_(module.gain)
+            nn.init.zeros_(module.bias)
+            nn.init.zeros_(module.leak_logit)      # sigmoid(0) = 0.5
+            nn.init.zeros_(module.log_rho)         # exp(0) = 1
+
+
+class FlyModel(FlyPreTrainedModel):
+    """The brain: connectome recurrence, token and post input, neuron-subset readout."""
+
+    def __init__(self, config: FlyConfig):
+        super().__init__(config)
+        N, E, d = config.n_neurons, config.n_edges, config.d_emb
+        self.emb = nn.Embedding(config.vocab_size, d, padding_idx=config.pad_token_id)
+        self.in_proj = nn.Parameter(torch.empty(config.n_token_input, d))
+        self.bias = nn.Parameter(torch.zeros(N, 1))
+        self.gain = nn.Parameter(torch.ones(N, 1))
+        self.log_rho = nn.Parameter(torch.zeros(()))
+        self.leak_logit = nn.Parameter(torch.zeros(N, 1))
+        self.news_proj = None
+        if config.conditional:
+            out = config.news_glom if config.news_mode == "glomeruli" else config.n_news_input
+            self.news_proj = nn.Linear(config.news_dim, out)
+            if config.news_mode == "glomeruli":
+                self.register_buffer("news_glom_of", torch.zeros(config.n_news_input, dtype=torch.long))
+
+        # connectome in CSR by target neuron: crow [N+1], col = source neuron, w = signed strength
+        self.register_buffer("crow", torch.zeros(N + 1, dtype=torch.int32))
+        self.register_buffer("col", torch.zeros(E, dtype=torch.int32))
+        self.register_buffer("w", torch.zeros(E))
+        self.register_buffer("token_idx", torch.zeros(config.n_token_input, dtype=torch.long))
+        self.register_buffer("news_idx", torch.zeros(config.n_news_input, dtype=torch.long))
+        self.register_buffer("readout_idx", torch.zeros(config.readout_size, dtype=torch.long))
+        # delay line: slot j covers token_idx[bounds[j]:bounds[j+1]] and sees token t-j
+        self.register_buffer("bounds", torch.zeros(config.delay + 1, dtype=torch.long))
+        self._wt_key = None
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.emb
+
+    def set_input_embeddings(self, value):
+        self.emb = value
+
+    def connectome(self):
+        N = self.config.n_neurons
+        W = torch.sparse_csr_tensor(self.crow, self.col, self.w, size=(N, N))
+        key = (self.w.data_ptr(), self.w._version, self.w.device)
+        if self._wt_key != key:
+            self._wt = W.t().to_sparse_csr()
+            self._wt_key = key
+        return W, self._wt
+
+    def drive_from_tokens(self, prev_ids, inputs_embeds):
+        """Delay-line current: input group j receives the embedding of token t-j. -> [T, n_in, B]"""
+        k = self.config.delay
+        B, T, _ = inputs_embeds.shape
+        ext = torch.cat([self.emb(prev_ids.flip(1)), inputs_embeds], dim=1)   # oldest first, then new
+        bounds = self.bounds.tolist()
+        parts = []
+        for j in range(k):
+            e_j = ext[:, k - j: k - j + T]                                    # token t-j for t in [0, T)
+            parts.append(e_j @ self.in_proj[bounds[j]:bounds[j + 1]].t())
+        return torch.cat(parts, dim=-1).permute(1, 2, 0).contiguous()
+
+    def forward(
+        self,
+        input_ids=None,
+        attention_mask=None,
+        inputs_embeds=None,
+        news_embeds=None,
+        cache_params: Optional[FlyCache] = None,
+        use_cache=None,
+        output_hidden_states=None,
+        return_dict=None,
+        **kwargs,
+    ):
+        cfg = self.config
+        use_cache = use_cache if use_cache is not None else cfg.use_cache
+        return_dict = return_dict if return_dict is not None else getattr(cfg, "return_dict", True)
+        if inputs_embeds is None:
+            inputs_embeds = self.emb(input_ids)
+        B, T, _ = inputs_embeds.shape
+        N, k, dev = cfg.n_neurons, cfg.delay, inputs_embeds.device
+
+        if cache_params is None:
+            x = inputs_embeds.new_zeros(N, B)
+            prev_ids = torch.full((B, k), cfg.pad_token_id, dtype=torch.long, device=dev)
+            seq_len = 0
+        else:
+            x = cache_params.state.t().contiguous()
+            prev_ids = cache_params.last_tokens
+            seq_len = cache_params.seq_len
+
+        drive = self.drive_from_tokens(prev_ids, inputs_embeds)
+        news_u = None
+        if self.news_proj is not None and news_embeds is not None:
+            news_u = self.news_proj(news_embeds.to(inputs_embeds.dtype))
+            if cfg.news_mode == "glomeruli":
+                news_u = news_u.index_select(1, self.news_glom_of)
+            news_u = news_u.t()                                               # [n_news_input, B]
+
+        W, WT = self.connectome()
+        a = torch.sigmoid(self.leak_logit)
+        scale = torch.exp(self.log_rho) * self.gain
+        mask = attention_mask.to(x.dtype).t() if attention_mask is not None else None
+        outs = []
+        for t in range(T):
+            u = torch.zeros(N, B, device=dev, dtype=x.dtype)
+            u = u.index_add(0, self.token_idx, drive[t])
+            if news_u is not None:
+                u = u.index_add(0, self.news_idx, news_u)
+            u = u + self.bias
+            for _ in range(cfg.ticks):
+                new = (1 - a) * x + a * torch.tanh(scale * _SpMM.apply(x, W, WT) + u)
+                if mask is not None:                                          # padding freezes the state
+                    m = mask[t][None, :]
+                    new = m * new + (1 - m) * x
+                x = new
+            outs.append(x.index_select(0, self.readout_idx))
+        hidden = torch.stack(outs, dim=0).permute(2, 0, 1)                    # [B, T, readout_size]
+
+        cache = None
+        if use_cache:
+            ids = prev_ids if input_ids is None else torch.cat([input_ids.flip(1), prev_ids], dim=1)[:, :k]
+            cache = FlyCache(x.t().contiguous(), ids, seq_len + T)
+        if not return_dict:
+            return tuple(v for v in [hidden, cache, (hidden,) if output_hidden_states else None] if v is not None)
+        return FlyOutput(last_hidden_state=hidden, cache_params=cache,
+                              hidden_states=(hidden,) if output_hidden_states else None)
+
+
+def _build_head(config: FlyConfig):
+    if config.head_type == "lowrank":
+        return nn.Sequential(
+            nn.Linear(config.readout_size, config.readout_rank, bias=False),
+            nn.LayerNorm(config.readout_rank),
+            nn.Linear(config.readout_rank, config.vocab_size),
+        )
+    return nn.Sequential(nn.LayerNorm(config.readout_size), nn.Linear(config.readout_size, config.vocab_size))
+
+
+class FlyForCausalLM(FlyPreTrainedModel, GenerationMixin):
+    """Decoder-only: tokens in, tokens out. `news_embeds` is accepted but is None for this arch."""
+
+    def __init__(self, config: FlyConfig):
+        super().__init__(config)
+        self.brain = FlyModel(config)
+        self.head = _build_head(config)
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.brain.emb
+
+    def set_input_embeddings(self, value):
+        self.brain.emb = value
+
+    def get_output_embeddings(self):
+        return self.head[-1]
+
+    def prepare_inputs_for_generation(self, input_ids, cache_params=None, use_cache=None,
+                                      attention_mask=None, news_embeds=None, **kwargs):
+        if cache_params is not None:                       # feed only what the state has not seen
+            input_ids = input_ids[:, cache_params.seq_len:]
+            attention_mask = None
+        model_inputs = {
+            "input_ids": input_ids,
+            "attention_mask": attention_mask,
+            "cache_params": cache_params,
+            "use_cache": use_cache if use_cache is not None else self.config.use_cache,
+        }
+        if news_embeds is not None:
+            model_inputs["news_embeds"] = news_embeds
+        return model_inputs
+
+    def forward(
+        self,
+        input_ids=None,
+        attention_mask=None,
+        inputs_embeds=None,
+        news_embeds=None,
+        cache_params: Optional[FlyCache] = None,
+        labels=None,
+        use_cache=None,
+        output_hidden_states=None,
+        return_dict=None,
+        logits_to_keep=0,
+        **kwargs,
+    ):
+        return_dict = return_dict if return_dict is not None else getattr(self.config, "return_dict", True)
+        out = self.brain(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            inputs_embeds=inputs_embeds,
+            news_embeds=news_embeds,
+            cache_params=cache_params,
+            use_cache=use_cache,
+            output_hidden_states=output_hidden_states,
+            return_dict=True,
+        )
+        h = out.last_hidden_state
+        sl = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) and logits_to_keep > 0 else slice(None)
+        logits = self.head(h[:, sl])
+        loss = None
+        if labels is not None:
+            loss = F.cross_entropy(logits[:, :-1].reshape(-1, logits.size(-1)).float(),
+                                   labels[:, 1:].reshape(-1).to(logits.device), ignore_index=-100)
+        if not return_dict:
+            return tuple(v for v in [loss, logits, out.cache_params, out.hidden_states] if v is not None)
+        return FlyCausalLMOutput(loss=loss, logits=logits, cache_params=out.cache_params,
+                                      hidden_states=out.hidden_states)

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+ "pad_token": "<pad>",
+ "bos_token": "<s>",
+ "eos_token": "</s>"
+}

tokenizer.json

@@ -0,0 +1,4146 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<pad>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "</s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "ByteLevel",
+    "add_prefix_space": false,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "post_processor": null,
+  "decoder": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": null,
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": false,
+    "byte_fallback": false,
+    "ignore_merges": false,
+    "vocab": {
+      "<pad>": 0,
+      "<s>": 1,
+      "</s>": 2,
+      "!": 3,
+      "\"": 4,
+      "#": 5,
+      "$": 6,
+      "%": 7,
+      "&": 8,
+      "'": 9,
+      "(": 10,
+      ")": 11,
+      "*": 12,
+      "+": 13,
+      ",": 14,
+      "-": 15,
+      ".": 16,
+      "/": 17,
+      "0": 18,
+      "1": 19,
+      "2": 20,
+      "3": 21,
+      "4": 22,
+      "5": 23,
+      "6": 24,
+      "7": 25,
+      "8": 26,
+      "9": 27,
+      ":": 28,
+      ";": 29,
+      "<": 30,
+      "=": 31,
+      ">": 32,
+      "?": 33,
+      "@": 34,
+      "A": 35,
+      "B": 36,
+      "C": 37,
+      "D": 38,
+      "E": 39,
+      "F": 40,
+      "G": 41,
+      "H": 42,
+      "I": 43,
+      "J": 44,
+      "K": 45,
+      "L": 46,
+      "M": 47,
+      "N": 48,
+      "O": 49,
+      "P": 50,
+      "Q": 51,
+      "R": 52,
+      "S": 53,
+      "T": 54,
+      "U": 55,
+      "V": 56,
+      "W": 57,
+      "X": 58,
+      "Y": 59,
+      "Z": 60,
+      "[": 61,
+      "\\": 62,
+      "]": 63,
+      "^": 64,
+      "_": 65,
+      "`": 66,
+      "a": 67,
+      "b": 68,
+      "c": 69,
+      "d": 70,
+      "e": 71,
+      "f": 72,
+      "g": 73,
+      "h": 74,
+      "i": 75,
+      "j": 76,
+      "k": 77,
+      "l": 78,
+      "m": 79,
+      "n": 80,
+      "o": 81,
+      "p": 82,
+      "q": 83,
+      "r": 84,
+      "s": 85,
+      "t": 86,
+      "u": 87,
+      "v": 88,
+      "w": 89,
+      "x": 90,
+      "y": 91,
+      "z": 92,
+      "{": 93,
+      "|": 94,
+      "}": 95,
+      "~": 96,
+      "¡": 97,
+      "¢": 98,
+      "£": 99,
+      "¤": 100,
+      "¥": 101,
+      "¦": 102,
+      "§": 103,
+      "¨": 104,
+      "©": 105,
+      "ª": 106,
+      "«": 107,
+      "¬": 108,
+      "®": 109,
+      "¯": 110,
+      "°": 111,
+      "±": 112,
+      "²": 113,
+      "³": 114,
+      "´": 115,
+      "µ": 116,
+      "¶": 117,
+      "·": 118,
+      "¸": 119,
+      "¹": 120,
+      "º": 121,
+      "»": 122,
+      "¼": 123,
+      "½": 124,
+      "¾": 125,
+      "¿": 126,
+      "À": 127,
+      "Á": 128,
+      "Â": 129,
+      "Ã": 130,
+      "Ä": 131,
+      "Å": 132,
+      "Æ": 133,
+      "Ç": 134,
+      "È": 135,
+      "É": 136,
+      "Ê": 137,
+      "Ë": 138,
+      "Ì": 139,
+      "Í": 140,
+      "Î": 141,
+      "Ï": 142,
+      "Ð": 143,
+      "Ñ": 144,
+      "Ò": 145,
+      "Ó": 146,
+      "Ô": 147,
+      "Õ": 148,
+      "Ö": 149,
+      "×": 150,
+      "Ø": 151,
+      "Ù": 152,
+      "Ú": 153,
+      "Û": 154,
+      "Ü": 155,
+      "Ý": 156,
+      "Þ": 157,
+      "ß": 158,
+      "à": 159,
+      "á": 160,
+      "â": 161,
+      "ã": 162,
+      "ä": 163,
+      "å": 164,
+      "æ": 165,
+      "ç": 166,
+      "è": 167,
+      "é": 168,
+      "ê": 169,
+      "ë": 170,
+      "ì": 171,
+      "í": 172,
+      "î": 173,
+      "ï": 174,
+      "ð": 175,
+      "ñ": 176,
+      "ò": 177,
+      "ó": 178,
+      "ô": 179,
+      "õ": 180,
+      "ö": 181,
+      "÷": 182,
+      "ø": 183,
+      "ù": 184,
+      "ú": 185,
+      "û": 186,
+      "ü": 187,
+      "ý": 188,
+      "þ": 189,
+      "ÿ": 190,
+      "Ā": 191,
+      "ā": 192,
+      "Ă": 193,
+      "ă": 194,
+      "Ą": 195,
+      "ą": 196,
+      "Ć": 197,
+      "ć": 198,
+      "Ĉ": 199,
+      "ĉ": 200,
+      "Ċ": 201,
+      "ċ": 202,
+      "Č": 203,
+      "č": 204,
+      "Ď": 205,
+      "ď": 206,
+      "Đ": 207,
+      "đ": 208,
+      "Ē": 209,
+      "ē": 210,
+      "Ĕ": 211,
+      "ĕ": 212,
+      "Ė": 213,
+      "ė": 214,
+      "Ę": 215,
+      "ę": 216,
+      "Ě": 217,
+      "ě": 218,
+      "Ĝ": 219,
+      "ĝ": 220,
+      "Ğ": 221,
+      "ğ": 222,
+      "Ġ": 223,
+      "ġ": 224,
+      "Ģ": 225,
+      "ģ": 226,
+      "Ĥ": 227,
+      "ĥ": 228,
+      "Ħ": 229,
+      "ħ": 230,
+      "Ĩ": 231,
+      "ĩ": 232,
+      "Ī": 233,
+      "ī": 234,
+      "Ĭ": 235,
+      "ĭ": 236,
+      "Į": 237,
+      "į": 238,
+      "İ": 239,
+      "ı": 240,
+      "IJ": 241,
+      "ij": 242,
+      "Ĵ": 243,
+      "ĵ": 244,
+      "Ķ": 245,
+      "ķ": 246,
+      "ĸ": 247,
+      "Ĺ": 248,
+      "ĺ": 249,
+      "Ļ": 250,
+      "ļ": 251,
+      "Ľ": 252,
+      "ľ": 253,
+      "Ŀ": 254,
+      "ŀ": 255,
+      "Ł": 256,
+      "ł": 257,
+      "Ń": 258,
+      "he": 259,
+      "Ġt": 260,
+      "Ġa": 261,
+      "Ġs": 262,
+      "Ġw": 263,
+      "nd": 264,
+      "Ġthe": 265,
+      "ed": 266,
+      "Ġand": 267,
+      "Ġto": 268,
+      "Ġb": 269,
+      "in": 270,
+      "Ġh": 271,
+      "Ġwa": 272,
+      "re": 273,
+      "Ġf": 274,
+      "it": 275,
+      "ou": 276,
+      "Ġl": 277,
+      "Ġc": 278,
+      "er": 279,
+      "Ġhe": 280,
+      "Ġd": 281,
+      "Ġwas": 282,
+      "Ġm": 283,
+      "Ġp": 284,
+      "om": 285,
+      "Ġo": 286,
+      "ĠT": 287,
+      "ay": 288,
+      "ar": 289,
+      "ing": 290,
+      "is": 291,
+      "Ġg": 292,
+      "il": 293,
+      "id": 294,
+      "at": 295,
+      "en": 296,
+      "Ġn": 297,
+      "Ġha": 298,
+      "Ġsa": 299,
+      "ĠS": 300,
+      "an": 301,
+      "ĠThe": 302,
+      "im": 303,
+      "or": 304,
+      "on": 305,
+      "Ġit": 306,
+      "Ġth": 307,
+      "ll": 308,
+      "le": 309,
+      "ĠH": 310,
+      "Ġher": 311,
+      "ot": 312,
+      "et": 313,
+      "ir": 314,
+      "ĠShe": 315,
+      "ĠHe": 316,
+      "ver": 317,
+      "es": 318,
+      "Ġin": 319,
+      "ow": 320,
+      "ut": 321,
+      "Ġe": 322,
+      "ck": 323,
+      "ld": 324,
+      "Ġu": 325,
+      "ĠThey": 326,
+      "oo": 327,
+      "ig": 328,
+      "Ġst": 329,
+      "Ġr": 330,
+      "Ġbe": 331,
+      "ily": 332,
+      "am": 333,
+      "Ġsaid": 334,
+      "Ġy": 335,
+      "ce": 336,
+      "Ġshe": 337,
+      "Ġ\"": 338,
+      "ke": 339,
+      "pp": 340,
+      "ĠI": 341,
+      "ith": 342,
+      "On": 343,
+      "ve": 344,
+      "Ġwith": 345,
+      "Ġof": 346,
+      "Ġso": 347,
+      "Ġon": 348,
+      "Lily": 349,
+      "ked": 350,
+      "ri": 351,
+      "Ġhis": 352,
+      "very": 353,
+      "nt": 354,
+      "ad": 355,
+      "Ġpl": 356,
+      "Ġday": 357,
+      "Ġthat": 358,
+      "Ġyou": 359,
+      "Ġup": 360,
+      "Ġhad": 361,
+      "st": 362,
+      "el": 363,
+      "Ġthey": 364,
+      "Ġplay": 365,
+      "ĠLily": 366,
+      "ould": 367,
+      "Ġfor": 368,
+      "Ġmom": 369,
+      "un": 370,
+      "Ġwe": 371,
+      "my": 372,
+      "The": 373,
+      "'s": 374,
+      "ĠB": 375,
+      "itt": 376,
+      "ent": 377,
+      "ch": 378,
+      "Ġhapp": 379,
+      "Ġli": 380,
+      "out": 381,
+      "Ġsh": 382,
+      "Ġwant": 383,
+      "her": 384,
+      "ly": 385,
+      "ound": 386,
+      "Ġvery": 387,
+      "ittle": 388,
+      "all": 389,
+      "ime": 390,
+      "Ġlittle": 391,
+      "ome": 392,
+      "al": 393,
+      "Ġthere": 394,
+      "Ġdo": 395,
+      "Ġtime": 396,
+      "se": 397,
+      "Ġwh": 398,
+      "Ġk": 399,
+      "end": 400,
+      "Ġnot": 401,
+      "Ġne": 402,
+      "ht": 403,
+      "Ġre": 404,
+      "Ġhappy": 405,
+      "Ġsm": 406,
+      "Ġbig": 407,
+      "ĠIt": 408,
+      "ĠĊ": 409,
+      "Ġbut": 410,
+      "Ġsaw": 411,
+      "Ġan": 412,
+      "ra": 413,
+      "Ġis": 414,
+      "ack": 415,
+      "riend": 416,
+      "Ġfriend": 417,
+      "ĠM": 418,
+      "One": 419,
+      "Ġas": 420,
+      "ide": 421,
+      "ry": 422,
+      "ved": 423,
+      ".\"": 424,
+      "Once": 425,
+      "ore": 426,
+      "Ġloo": 427,
+      "ake": 428,
+      "'t": 429,
+      "ec": 430,
+      "ug": 431,
+      "ter": 432,
+      "Ġwere": 433,
+      "Ġlo": 434,
+      "!\"": 435,
+      "irl": 436,
+      "Ġhim": 437,
+      "Ġse": 438,
+      "Ġgirl": 439,
+      "Ġbo": 440,
+      "Ġtoo": 441,
+      "ĠTim": 442,
+      "Ġupon": 443,
+      "Ġgo": 444,
+      "Ġwanted": 445,
+      "Ġj": 446,
+      "Ġout": 447,
+      "ard": 448,
+      "ill": 449,
+      "ind": 450,
+      "way": 451,
+      "Ġsp": 452,
+      "Ġat": 453,
+      "fu": 454,
+      "Ġthem": 455,
+      "Ġcould": 456,
+      "ur": 457,
+      "hen": 458,
+      "Ġsmil": 459,
+      "Ġare": 460,
+      "ĠA": 461,
+      "Ġex": 462,
+      "Ġdid": 463,
+      "hed": 464,
+      "art": 465,
+      "ĠJ": 466,
+      "Ġar": 467,
+      "Ġwent": 468,
+      "ain": 469,
+      "ic": 470,
+      "rom": 471,
+      "Ġtheir": 472,
+      "Ġhave": 473,
+      "hing": 474,
+      "ful": 475,
+      "one": 476,
+      "Ġall": 477,
+      "Ġnam": 478,
+      "lp": 479,
+      "ight": 480,
+      "ood": 481,
+      "Ġkn": 482,
+      "Ġfriends": 483,
+      "Ġcan": 484,
+      "Ġhelp": 485,
+      "Ġcl": 486,
+      "Ġback": 487,
+      "ark": 488,
+      "ĠBen": 489,
+      "?\"": 490,
+      "Ġfun": 491,
+      "um": 492,
+      "Ġle": 493,
+      "are": 494,
+      "side": 495,
+      "oug": 496,
+      "Ġsc": 497,
+      "Ġno": 498,
+      "Ġnamed": 499,
+      "ĠOne": 500,
+      "Ġloved": 501,
+      "op": 502,
+      "elt": 503,
+      "Ġal": 504,
+      "ick": 505,
+      "Ġsmiled": 506,
+      "ĠTom": 507,
+      "Ġaround": 508,
+      "Ġsay": 509,
+      "ĠTimmy": 510,
+      "Ġfelt": 511,
+      "Ġfe": 512,
+      "You": 513,
+      "Ġman": 514,
+      "Ġsee": 515,
+      "omet": 516,
+      "Ġlooked": 517,
+      "Ġbr": 518,
+      "Ġasked": 519,
+      "ame": 520,
+      "Ġlike": 521,
+      "Ġtoy": 522,
+      "Ġstart": 523,
+      "Ġboy": 524,
+      "Ġbir": 525,
+      "Ġsome": 526,
+      "get": 527,
+      "ss": 528,
+      "ure": 529,
+      "Ġwould": 530,
+      "ĠW": 531,
+      "Ġsomet": 532,
+      "ong": 533,
+      "Ġro": 534,
+      "Ġme": 535,
+      "Ġbird": 536,
+      "dd": 537,
+      "king": 538,
+      "ade": 539,
+      "ree": 540,
+      "ings": 541,
+      "Ġwor": 542,
+      "ie": 543,
+      "Ġstarted": 544,
+      "â": 545,
+      "âĤ": 546,
+      "âĤ¬": 547,
+      "ice": 548,
+      "Ġaway": 549,
+      "ag": 550,
+      "Ġfa": 551,
+      "Ġag": 552,
+      "own": 553,
+      "Ġran": 554,
+      "Ġwhat": 555,
+      "Ġsomething": 556,
+      "gether": 557,
+      "Ġmake": 558,
+      "ared": 559,
+      "Ġmade": 560,
+      "if": 561,
+      "oud": 562,
+      "ited": 563,
+      "Ġco": 564,
+      "ĠBut": 565,
+      "Ġtogether": 566,
+      "Ġexc": 567,
+      "ther": 568,
+      "Ġcar": 569,
+      "Ġfrom": 570,
+      "Ġnew": 571,
+      "Ġpark": 572,
+      "Ġother": 573,
+      "Ġsad": 574,
+      ",\"": 575,
+      "Ġwhen": 576,
+      "ro": 577,
+      "ĠE": 578,
+      "Ġput": 579,
+      "ell": 580,
+      "Ġhome": 581,
+      "Ġla": 582,
+      "qu": 583,
+      "Ġdec": 584,
+      "Ġwho": 585,
+      "Ġfound": 586,
+      "Ġhug": 587,
+      "ĠYou": 588,
+      "Ġsays": 589,
+      "ried": 590,
+      "ble": 591,
+      "Ġmu": 592,
+      "Ġthen": 593,
+      "ach": 594,
+      "est": 595,
+      "Ġgood": 596,
+      "Ġagain": 597,
+      "ep": 598,
+      "Ġfl": 599,
+      "ĠL": 600,
+      "Ġwal": 601,
+      "Ġch": 602,
+      "ook": 603,
+      "pped": 604,
+      "ought": 605,
+      "pl": 606,
+      "ous": 607,
+      "hat": 608,
+      "Ġsw": 609,
+      "ally": 610,
+      "Ġthings": 611,
+      "Ġgot": 612,
+      "Ġplaying": 613,
+      "Ġexcited": 614,
+      "ided": 615,
+      "Ġpr": 616,
+      "Ġevery": 617,
+      "Ġget": 618,
+      "Ġdecided": 619,
+      "as": 620,
+      "ny": 621,
+      "Ġbl": 622,
+      "Ġcame": 623,
+      "Ġscared": 624,
+      "Ġbu": 625,
+      "Ġfind": 626,
+      "ave": 627,
+      "Ġbec": 628,
+      "Ġfeel": 629,
+      "Ġv": 630,
+      "ust": 631,
+      "Ġone": 632,
+      "udd": 633,
+      "Ġdown": 634,
+      "Ġliked": 635,
+      "ouse": 636,
+      "Ġlot": 637,
+      "na": 638,
+      "ue": 639,
+      "Ġcare": 640,
+      "ĠD": 641,
+      "Ġthan": 642,
+      "Ġdad": 643,
+      "Ġtree": 644,
+      "arn": 645,
+      "ways": 646,
+      "ap": 647,
+      "Ġgra": 648,
+      "Ġke": 649,
+      "Ġmommy": 650,
+      "eci": 651,
+      "nder": 652,
+      "Ġalways": 653,
+      "Ġdog": 654,
+      "Ġyour": 655,
+      "bb": 656,
+      "Ġoutside": 657,
+      "fe": 658,
+      "ched": 659,
+      "ant": 660,
+      "Ġtook": 661,
+      "Ġab": 662,
+      "Ġpo": 663,
+      "Ġlook": 664,
+      "ise": 665,
+      "ers": 666,
+      "Ġdidn": 667,
+      "Ġgr": 668,
+      "our": 669,
+      "ĠMom": 670,
+      "Ġmy": 671,
+      "Ġspeci": 672,
+      "Ġspecial": 673,
+      "They": 674,
+      "ara": 675,
+      "Ġkne": 676,
+      "ened": 677,
+      "Ġabout": 678,
+      "nna": 679,
+      "ite": 680,
+      "Ġthought": 681,
+      "ist": 682,
+      "ess": 683,
+      "ax": 684,
+      "Ġball": 685,
+      "uck": 686,
+      "Ġho": 687,
+      "Ġlearn": 688,
+      "ĠF": 689,
+      "Ġmore": 690,
+      "udden": 691,
+      "Ġwill": 692,
+      "Ġtoys": 693,
+      "ĠSam": 694,
+      "Ġint": 695,
+      "Ġold": 696,
+      "uddenly": 697,
+      "Ġknow": 698,
+      "reat": 699,
+      "Ġtake": 700,
+      "ma": 701,
+      "Ġlaug": 702,
+      "ged": 703,
+      "Ġhow": 704,
+      "Ġknew": 705,
+      "Ġta": 706,
+      "Ġshow": 707,
+      "fter": 708,
+      "au": 709,
+      "ret": 710,
+      "Ġany": 711,
+      "Ġpe": 712,
+      "Ġcat": 713,
+      "ish": 714,
+      "Ġmuch": 715,
+      "ĠEvery": 716,
+      "ven": 717,
+      "ive": 718,
+      "ink": 719,
+      "Ġtw": 720,
+      "Ġsk": 721,
+      "Ġclo": 722,
+      "Ġinside": 723,
+      "Ġnever": 724,
+      "Ġhand": 725,
+      "But": 726,
+      "Ġcouldn": 727,
+      "Ġrun": 728,
+      "Ġmo": 729,
+      "Ġsun": 730,
+      "Ġhouse": 731,
+      "ge": 732,
+      "Ġen": 733,
+      "Ġstr": 734,
+      "Ġpu": 735,
+      "Ġqu": 736,
+      "Ġsor": 737,
+      "Ġop": 738,
+      "Ġtried": 739,
+      "ia": 740,
+      "ank": 741,
+      "ace": 742,
+      "Ġproud": 743,
+      "Ġsl": 744,
+      "Ġtold": 745,
+      "Ġinto": 746,
+      "Ġtr": 747,
+      "ty": 748,
+      "and": 749,
+      "Ġexpl": 750,
+      "ion": 751,
+      "Ġwater": 752,
+      "ause": 753,
+      "Ġpick": 754,
+      "Ġcom": 755,
+      "ĠMax": 756,
+      "Ġor": 757,
+      "Ġthis": 758,
+      "Ġheard": 759,
+      "Ġover": 760,
+      "here": 761,
+      "by": 762,
+      "Ġif": 763,
+      "Ġgave": 764,
+      "Ġroom": 765,
+      "ump": 766,
+      "Ġwat": 767,
+      "etter": 768,
+      "Ġeat": 769,
+      "ough": 770,
+      "Ġfo": 771,
+      "Ġbecause": 772,
+      "Ġsorry": 773,
+      "Ġplayed": 774,
+      "oth": 775,
+      "other": 776,
+      "Ġlong": 777,
+      "Ġjust": 778,
+      "dy": 779,
+      "Ġoff": 780,
+      "Ġhugged": 781,
+      "imal": 782,
+      "Ġsmall": 783,
+      "Ġeach": 784,
+      "Ġanimal": 785,
+      "ine": 786,
+      "ile": 787,
+      "Ġbear": 788,
+      "Ġneed": 789,
+      "Ġbox": 790,
+      "Ġunt": 791,
+      "ĠAnna": 792,
+      "ort": 793,
+      "ĠJack": 794,
+      "ĠWe": 795,
+      "ĠJo": 796,
+      "Ġjo": 797,
+      "Ġmany": 798,
+      "ild": 799,
+      "Ġuntil": 800,
+      "uc": 801,
+      "Ġtry": 802,
+      "sed": 803,
+      "ate": 804,
+      "iz": 805,
+      "Ġpret": 806,
+      "sel": 807,
+      "ving": 808,
+      "Tom": 809,
+      "ak": 810,
+      "Ġend": 811,
+      "Ġnice": 812,
+      "ĠSara": 813,
+      "ft": 814,
+      "Ġlearned": 815,
+      "Mom": 816,
+      "Ġbra": 817,
+      "Ġflow": 818,
+      "Ġcall": 819,
+      "Ġbetter": 820,
+      "Ġkind": 821,
+      "gry": 822,
+      "Ġsn": 823,
+      "ars": 824,
+      "He": 825,
+      "ber": 826,
+      "urn": 827,
+      "Ġlots": 828,
+      "ĠSo": 829,
+      "Ġim": 830,
+      "aut": 831,
+      "Ġbest": 832,
+      "lly": 833,
+      "'m": 834,
+      "em": 835,
+      "pt": 836,
+      "Ġdre": 837,
+      "urp": 838,
+      "Ġra": 839,
+      "urt": 840,
+      "hn": 841,
+      "Ġbeaut": 842,
+      "arden": 843,
+      "Ġgarden": 844,
+      "ady": 845,
+      "Ġfam": 846,
+      "Ġthanked": 847,
+      "Ġad": 848,
+      "iny": 849,
+      "ĠThen": 850,
+      "Ġcle": 851,
+      "Ġby": 852,
+      "vent": 853,
+      "ves": 854,
+      "Ġlaughed": 855,
+      "ane": 856,
+      "self": 857,
+      "lew": 858,
+      "Ġeven": 859,
+      "Ġbeauti": 860,
+      "Ġcareful": 861,
+      "Ġanimals": 862,
+      "ĠHer": 863,
+      "wn": 864,
+      "Ġbeautiful": 865,
+      "Ġche": 866,
+      "Ġfi": 867,
+      "Ġwo": 868,
+      "Ġhard": 869,
+      "Ġsky": 870,
+      "Ġgl": 871,
+      "Th": 872,
+      "ies": 873,
+      "ĠAnd": 874,
+      "Ġway": 875,
+      "Ġfast": 876,
+      "Ġcol": 877,
+      "Ġdon": 878,
+      "Ġstay": 879,
+      "Ġrem": 880,
+      "Åĵ": 881,
+      "Ġtwo": 882,
+      "Ġjump": 883,
+      "Ġits": 884,
+      "Ġcome": 885,
+      "Ġbrave": 886,
+      "Ġlived": 887,
+      "ĠFrom": 888,
+      "Yes": 889,
+      "Ġlove": 890,
+      "It": 891,
+      "Ġloud": 892,
+      "Ġte": 893,
+      "ĠHis": 894,
+      "Ġboth": 895,
+      "Ġfamily": 896,
+      "ect": 897,
+      "Ġunder": 898,
+      "ning": 899,
+      "ool": 900,
+      "Ġsafe": 901,
+      "Ġstill": 902,
+      "ast": 903,
+      "Ġhurt": 904,
+      "ĠMia": 905,
+      "ase": 906,
+      "Ġcu": 907,
+      "Ġbad": 908,
+      "fore": 909,
+      "sh": 910,
+      "ĠC": 911,
+      "ople": 912,
+      "Ġcalled": 913,
+      "Ġimp": 914,
+      "Ġdan": 915,
+      "Ġstor": 916,
+      "Ġkept": 917,
+      "Ġflew": 918,
+      "Ġred": 919,
+      "Ġâ": 920,
+      "Ġlist": 921,
+      "urpr": 922,
+      "When": 923,
+      "Ġfore": 924,
+      "Ġsurpr": 925,
+      "Ġfin": 926,
+      "Ġsoon": 927,
+      "Ġdoll": 928,
+      "led": 929,
+      "Ġfly": 930,
+      "ĠWhen": 931,
+      "ct": 932,
+      "Ġlet": 933,
+      "Ġwalked": 934,
+      "Ġpeople": 935,
+      "ip": 936,
+      "ished": 937,
+      "ucy": 938,
+      "hes": 939,
+      "Ġshould": 940,
+      "Tim": 941,
+      "Ġbook": 942,
+      "ary": 943,
+      "Ġwind": 944,
+      "Ġbefore": 945,
+      "ised": 946,
+      "Ġangry": 947,
+      "Ġexplore": 948,
+      "Ġfeeling": 949,
+      "Ġrock": 950,
+      "xt": 951,
+      "Ġpic": 952,
+      "igh": 953,
+      "âĤ¬â": 954,
+      "Ġgoing": 955,
+      "ob": 956,
+      "Ġdoor": 957,
+      "Ġkeep": 958,
+      "ĠP": 959,
+      "Ġcon": 960,
+      "Let": 961,
+      "Ġopened": 962,
+      "Ġpretty": 963,
+      "Ġcolor": 964,
+      "Ġthr": 965,
+      "be": 966,
+      "Ġshiny": 967,
+      "Ġwar": 968,
+      "Ġey": 969,
+      "kay": 970,
+      "illy": 971,
+      "Ġwalking": 972,
+      "Ġpicked": 973,
+      "Ġdra": 974,
+      "Ġsmile": 975,
+      "ied": 976,
+      "Ġnow": 977,
+      "Ġfar": 978,
+      "Ġnoise": 979,
+      "Ġground": 980,
+      "age": 981,
+      "Ġsto": 982,
+      "dded": 983,
+      "Ġclean": 984,
+      "oon": 985,
+      "Ġturn": 986,
+      "ock": 987,
+      "Ġadvent": 988,
+      "No": 989,
+      "th": 990,
+      "Ġshare": 991,
+      "Ħ¢": 992,
+      "âĤ¬âĦ¢": 993,
+      "Ġma": 994,
+      "thing": 995,
+      "iced": 996,
+      "uch": 997,
+      "Ġclos": 998,
+      "Ġfr": 999,
+      "Ġcry": 1000,
+      "Ġreal": 1001,
+      "Ġblue": 1002,
+      "What": 1003,
+      "kes": 1004,
+      "ture": 1005,
+      "Ġnext": 1006,
+      "ummy": 1007,
+      "oy": 1008,
+      "ember": 1009,
+      "Ġun": 1010,
+      "so": 1011,
+      "Ġdif": 1012,
+      "imb": 1013,
+      "Ġide": 1014,
+      "Ġclimb": 1015,
+      "Ġeyes": 1016,
+      "ĠSuddenly": 1017,
+      "Ġbed": 1018,
+      "Ġhas": 1019,
+      "les": 1020,
+      "Ġalso": 1021,
+      "That": 1022,
+      "Ġgreat": 1023
+    },
+    "merges": [
+      [
+        "h",
+        "e"
+      ],
+      [
+        "Ġ",
+        "t"
+      ],
+      [
+        "Ġ",
+        "a"
+      ],
+      [
+        "Ġ",
+        "s"
+      ],
+      [
+        "Ġ",
+        "w"
+      ],
+      [
+        "n",
+        "d"
+      ],
+      [
+        "Ġt",
+        "he"
+      ],
+      [
+        "e",
+        "d"
+      ],
+      [
+        "Ġa",
+        "nd"
+      ],
+      [
+        "Ġt",
+        "o"
+      ],
+      [
+        "Ġ",
+        "b"
+      ],
+      [
+        "i",
+        "n"
+      ],
+      [
+        "Ġ",
+        "h"
+      ],
+      [
+        "Ġw",
+        "a"
+      ],
+      [
+        "r",
+        "e"
+      ],
+      [
+        "Ġ",
+        "f"
+      ],
+      [
+        "i",
+        "t"
+      ],
+      [
+        "o",
+        "u"
+      ],
+      [
+        "Ġ",
+        "l"
+      ],
+      [
+        "Ġ",
+        "c"
+      ],
+      [
+        "e",
+        "r"
+      ],
+      [
+        "Ġ",
+        "he"
+      ],
+      [
+        "Ġ",
+        "d"
+      ],
+      [
+        "Ġwa",
+        "s"
+      ],
+      [
+        "Ġ",
+        "m"
+      ],
+      [
+        "Ġ",
+        "p"
+      ],
+      [
+        "o",
+        "m"
+      ],
+      [
+        "Ġ",
+        "o"
+      ],
+      [
+        "Ġ",
+        "T"
+      ],
+      [
+        "a",
+        "y"
+      ],
+      [
+        "a",
+        "r"
+      ],
+      [
+        "in",
+        "g"
+      ],
+      [
+        "i",
+        "s"
+      ],
+      [
+        "Ġ",
+        "g"
+      ],
+      [
+        "i",
+        "l"
+      ],
+      [
+        "i",
+        "d"
+      ],
+      [
+        "a",
+        "t"
+      ],
+      [
+        "e",
+        "n"
+      ],
+      [
+        "Ġ",
+        "n"
+      ],
+      [
+        "Ġh",
+        "a"
+      ],
+      [
+        "Ġs",
+        "a"
+      ],
+      [
+        "Ġ",
+        "S"
+      ],
+      [
+        "a",
+        "n"
+      ],
+      [
+        "ĠT",
+        "he"
+      ],
+      [
+        "i",
+        "m"
+      ],
+      [
+        "o",
+        "r"
+      ],
+      [
+        "o",
+        "n"
+      ],
+      [
+        "Ġ",
+        "it"
+      ],
+      [
+        "Ġt",
+        "h"
+      ],
+      [
+        "l",
+        "l"
+      ],
+      [
+        "l",
+        "e"
+      ],
+      [
+        "Ġ",
+        "H"
+      ],
+      [
+        "Ġhe",
+        "r"
+      ],
+      [
+        "o",
+        "t"
+      ],
+      [
+        "e",
+        "t"
+      ],
+      [
+        "i",
+        "r"
+      ],
+      [
+        "ĠS",
+        "he"
+      ],
+      [
+        "ĠH",
+        "e"
+      ],
+      [
+        "v",
+        "er"
+      ],
+      [
+        "e",
+        "s"
+      ],
+      [
+        "Ġ",
+        "in"
+      ],
+      [
+        "o",
+        "w"
+      ],
+      [
+        "u",
+        "t"
+      ],
+      [
+        "Ġ",
+        "e"
+      ],
+      [
+        "c",
+        "k"
+      ],
+      [
+        "l",
+        "d"
+      ],
+      [
+        "Ġ",
+        "u"
+      ],
+      [
+        "ĠThe",
+        "y"
+      ],
+      [
+        "o",
+        "o"
+      ],
+      [
+        "i",
+        "g"
+      ],
+      [
+        "Ġs",
+        "t"
+      ],
+      [
+        "Ġ",
+        "r"
+      ],
+      [
+        "Ġb",
+        "e"
+      ],
+      [
+        "il",
+        "y"
+      ],
+      [
+        "a",
+        "m"
+      ],
+      [
+        "Ġsa",
+        "id"
+      ],
+      [
+        "Ġ",
+        "y"
+      ],
+      [
+        "c",
+        "e"
+      ],
+      [
+        "Ġs",
+        "he"
+      ],
+      [
+        "Ġ",
+        "\""
+      ],
+      [
+        "k",
+        "e"
+      ],
+      [
+        "p",
+        "p"
+      ],
+      [
+        "Ġ",
+        "I"
+      ],
+      [
+        "it",
+        "h"
+      ],
+      [
+        "O",
+        "n"
+      ],
+      [
+        "v",
+        "e"
+      ],
+      [
+        "Ġw",
+        "ith"
+      ],
+      [
+        "Ġo",
+        "f"
+      ],
+      [
+        "Ġs",
+        "o"
+      ],
+      [
+        "��o",
+        "n"
+      ],
+      [
+        "L",
+        "ily"
+      ],
+      [
+        "k",
+        "ed"
+      ],
+      [
+        "r",
+        "i"
+      ],
+      [
+        "Ġh",
+        "is"
+      ],
+      [
+        "ver",
+        "y"
+      ],
+      [
+        "n",
+        "t"
+      ],
+      [
+        "a",
+        "d"
+      ],
+      [
+        "Ġp",
+        "l"
+      ],
+      [
+        "Ġd",
+        "ay"
+      ],
+      [
+        "Ġth",
+        "at"
+      ],
+      [
+        "Ġy",
+        "ou"
+      ],
+      [
+        "Ġu",
+        "p"
+      ],
+      [
+        "Ġha",
+        "d"
+      ],
+      [
+        "s",
+        "t"
+      ],
+      [
+        "e",
+        "l"
+      ],
+      [
+        "Ġthe",
+        "y"
+      ],
+      [
+        "Ġpl",
+        "ay"
+      ],
+      [
+        "Ġ",
+        "Lily"
+      ],
+      [
+        "ou",
+        "ld"
+      ],
+      [
+        "Ġf",
+        "or"
+      ],
+      [
+        "Ġm",
+        "om"
+      ],
+      [
+        "u",
+        "n"
+      ],
+      [
+        "Ġw",
+        "e"
+      ],
+      [
+        "m",
+        "y"
+      ],
+      [
+        "T",
+        "he"
+      ],
+      [
+        "'",
+        "s"
+      ],
+      [
+        "Ġ",
+        "B"
+      ],
+      [
+        "it",
+        "t"
+      ],
+      [
+        "en",
+        "t"
+      ],
+      [
+        "c",
+        "h"
+      ],
+      [
+        "Ġha",
+        "pp"
+      ],
+      [
+        "Ġl",
+        "i"
+      ],
+      [
+        "ou",
+        "t"
+      ],
+      [
+        "Ġs",
+        "h"
+      ],
+      [
+        "Ġwa",
+        "nt"
+      ],
+      [
+        "he",
+        "r"
+      ],
+      [
+        "l",
+        "y"
+      ],
+      [
+        "ou",
+        "nd"
+      ],
+      [
+        "Ġ",
+        "very"
+      ],
+      [
+        "itt",
+        "le"
+      ],
+      [
+        "a",
+        "ll"
+      ],
+      [
+        "im",
+        "e"
+      ],
+      [
+        "Ġl",
+        "ittle"
+      ],
+      [
+        "om",
+        "e"
+      ],
+      [
+        "a",
+        "l"
+      ],
+      [
+        "Ġthe",
+        "re"
+      ],
+      [
+        "Ġd",
+        "o"
+      ],
+      [
+        "Ġt",
+        "ime"
+      ],
+      [
+        "s",
+        "e"
+      ],
+      [
+        "Ġw",
+        "h"
+      ],
+      [
+        "Ġ",
+        "k"
+      ],
+      [
+        "e",
+        "nd"
+      ],
+      [
+        "Ġn",
+        "ot"
+      ],
+      [
+        "Ġn",
+        "e"
+      ],
+      [
+        "h",
+        "t"
+      ],
+      [
+        "Ġ",
+        "re"
+      ],
+      [
+        "Ġhapp",
+        "y"
+      ],
+      [
+        "Ġs",
+        "m"
+      ],
+      [
+        "Ġb",
+        "ig"
+      ],
+      [
+        "ĠI",
+        "t"
+      ],
+      [
+        "Ġ",
+        "Ċ"
+      ],
+      [
+        "Ġb",
+        "ut"
+      ],
+      [
+        "Ġsa",
+        "w"
+      ],
+      [
+        "Ġa",
+        "n"
+      ],
+      [
+        "r",
+        "a"
+      ],
+      [
+        "Ġ",
+        "is"
+      ],
+      [
+        "a",
+        "ck"
+      ],
+      [
+        "ri",
+        "end"
+      ],
+      [
+        "Ġf",
+        "riend"
+      ],
+      [
+        "Ġ",
+        "M"
+      ],
+      [
+        "On",
+        "e"
+      ],
+      [
+        "Ġa",
+        "s"
+      ],
+      [
+        "id",
+        "e"
+      ],
+      [
+        "r",
+        "y"
+      ],
+      [
+        "v",
+        "ed"
+      ],
+      [
+        ".",
+        "\""
+      ],
+      [
+        "On",
+        "ce"
+      ],
+      [
+        "o",
+        "re"
+      ],
+      [
+        "Ġl",
+        "oo"
+      ],
+      [
+        "a",
+        "ke"
+      ],
+      [
+        "'",
+        "t"
+      ],
+      [
+        "e",
+        "c"
+      ],
+      [
+        "u",
+        "g"
+      ],
+      [
+        "t",
+        "er"
+      ],
+      [
+        "Ġwe",
+        "re"
+      ],
+      [
+        "Ġl",
+        "o"
+      ],
+      [
+        "!",
+        "\""
+      ],
+      [
+        "ir",
+        "l"
+      ],
+      [
+        "Ġh",
+        "im"
+      ],
+      [
+        "Ġs",
+        "e"
+      ],
+      [
+        "Ġg",
+        "irl"
+      ],
+      [
+        "Ġb",
+        "o"
+      ],
+      [
+        "Ġto",
+        "o"
+      ],
+      [
+        "ĠT",
+        "im"
+      ],
+      [
+        "Ġup",
+        "on"
+      ],
+      [
+        "Ġg",
+        "o"
+      ],
+      [
+        "Ġwant",
+        "ed"
+      ],
+      [
+        "Ġ",
+        "j"
+      ],
+      [
+        "Ġ",
+        "out"
+      ],
+      [
+        "ar",
+        "d"
+      ],
+      [
+        "il",
+        "l"
+      ],
+      [
+        "i",
+        "nd"
+      ],
+      [
+        "w",
+        "ay"
+      ],
+      [
+        "Ġs",
+        "p"
+      ],
+      [
+        "Ġa",
+        "t"
+      ],
+      [
+        "f",
+        "u"
+      ],
+      [
+        "Ġthe",
+        "m"
+      ],
+      [
+        "Ġc",
+        "ould"
+      ],
+      [
+        "u",
+        "r"
+      ],
+      [
+        "he",
+        "n"
+      ],
+      [
+        "Ġsm",
+        "il"
+      ],
+      [
+        "Ġa",
+        "re"
+      ],
+      [
+        "Ġ",
+        "A"
+      ],
+      [
+        "Ġe",
+        "x"
+      ],
+      [
+        "Ġd",
+        "id"
+      ],
+      [
+        "he",
+        "d"
+      ],
+      [
+        "ar",
+        "t"
+      ],
+      [
+        "Ġ",
+        "J"
+      ],
+      [
+        "Ġa",
+        "r"
+      ],
+      [
+        "Ġw",
+        "ent"
+      ],
+      [
+        "a",
+        "in"
+      ],
+      [
+        "i",
+        "c"
+      ],
+      [
+        "r",
+        "om"
+      ],
+      [
+        "Ġthe",
+        "ir"
+      ],
+      [
+        "Ġha",
+        "ve"
+      ],
+      [
+        "h",
+        "ing"
+      ],
+      [
+        "fu",
+        "l"
+      ],
+      [
+        "on",
+        "e"
+      ],
+      [
+        "Ġa",
+        "ll"
+      ],
+      [
+        "Ġn",
+        "am"
+      ],
+      [
+        "l",
+        "p"
+      ],
+      [
+        "ig",
+        "ht"
+      ],
+      [
+        "oo",
+        "d"
+      ],
+      [
+        "Ġk",
+        "n"
+      ],
+      [
+        "Ġfriend",
+        "s"
+      ],
+      [
+        "Ġc",
+        "an"
+      ],
+      [
+        "Ġhe",
+        "lp"
+      ],
+      [
+        "Ġc",
+        "l"
+      ],
+      [
+        "Ġb",
+        "ack"
+      ],
+      [
+        "ar",
+        "k"
+      ],
+      [
+        "ĠB",
+        "en"
+      ],
+      [
+        "?",
+        "\""
+      ],
+      [
+        "Ġf",
+        "un"
+      ],
+      [
+        "u",
+        "m"
+      ],
+      [
+        "Ġl",
+        "e"
+      ],
+      [
+        "a",
+        "re"
+      ],
+      [
+        "s",
+        "ide"
+      ],
+      [
+        "ou",
+        "g"
+      ],
+      [
+        "Ġs",
+        "c"
+      ],
+      [
+        "Ġn",
+        "o"
+      ],
+      [
+        "Ġnam",
+        "ed"
+      ],
+      [
+        "Ġ",
+        "One"
+      ],
+      [
+        "Ġlo",
+        "ved"
+      ],
+      [
+        "o",
+        "p"
+      ],
+      [
+        "el",
+        "t"
+      ],
+      [
+        "Ġa",
+        "l"
+      ],
+      [
+        "i",
+        "ck"
+      ],
+      [
+        "Ġsmil",
+        "ed"
+      ],
+      [
+        "ĠT",
+        "om"
+      ],
+      [
+        "Ġar",
+        "ound"
+      ],
+      [
+        "Ġs",
+        "ay"
+      ],
+      [
+        "ĠTim",
+        "my"
+      ],
+      [
+        "Ġf",
+        "elt"
+      ],
+      [
+        "Ġf",
+        "e"
+      ],
+      [
+        "Y",
+        "ou"
+      ],
+      [
+        "Ġm",
+        "an"
+      ],
+      [
+        "Ġse",
+        "e"
+      ],
+      [
+        "om",
+        "et"
+      ],
+      [
+        "Ġloo",
+        "ked"
+      ],
+      [
+        "Ġb",
+        "r"
+      ],
+      [
+        "Ġas",
+        "ked"
+      ],
+      [
+        "am",
+        "e"
+      ],
+      [
+        "Ġli",
+        "ke"
+      ],
+      [
+        "Ġto",
+        "y"
+      ],
+      [
+        "Ġst",
+        "art"
+      ],
+      [
+        "Ġbo",
+        "y"
+      ],
+      [
+        "Ġb",
+        "ir"
+      ],
+      [
+        "Ġs",
+        "ome"
+      ],
+      [
+        "g",
+        "et"
+      ],
+      [
+        "s",
+        "s"
+      ],
+      [
+        "u",
+        "re"
+      ],
+      [
+        "Ġw",
+        "ould"
+      ],
+      [
+        "Ġ",
+        "W"
+      ],
+      [
+        "Ġs",
+        "omet"
+      ],
+      [
+        "on",
+        "g"
+      ],
+      [
+        "Ġr",
+        "o"
+      ],
+      [
+        "Ġm",
+        "e"
+      ],
+      [
+        "Ġbir",
+        "d"
+      ],
+      [
+        "d",
+        "d"
+      ],
+      [
+        "k",
+        "ing"
+      ],
+      [
+        "ad",
+        "e"
+      ],
+      [
+        "re",
+        "e"
+      ],
+      [
+        "ing",
+        "s"
+      ],
+      [
+        "Ġw",
+        "or"
+      ],
+      [
+        "i",
+        "e"
+      ],
+      [
+        "Ġstart",
+        "ed"
+      ],
+      [
+        "Ã",
+        "¢"
+      ],
+      [
+        "â",
+        "Ĥ"
+      ],
+      [
+        "âĤ",
+        "¬"
+      ],
+      [
+        "i",
+        "ce"
+      ],
+      [
+        "Ġa",
+        "way"
+      ],
+      [
+        "a",
+        "g"
+      ],
+      [
+        "Ġf",
+        "a"
+      ],
+      [
+        "Ġa",
+        "g"
+      ],
+      [
+        "ow",
+        "n"
+      ],
+      [
+        "Ġr",
+        "an"
+      ],
+      [
+        "Ġwh",
+        "at"
+      ],
+      [
+        "Ġsomet",
+        "hing"
+      ],
+      [
+        "get",
+        "her"
+      ],
+      [
+        "Ġm",
+        "ake"
+      ],
+      [
+        "ar",
+        "ed"
+      ],
+      [
+        "Ġm",
+        "ade"
+      ],
+      [
+        "i",
+        "f"
+      ],
+      [
+        "ou",
+        "d"
+      ],
+      [
+        "it",
+        "ed"
+      ],
+      [
+        "Ġc",
+        "o"
+      ],
+      [
+        "ĠB",
+        "ut"
+      ],
+      [
+        "Ġto",
+        "gether"
+      ],
+      [
+        "Ġex",
+        "c"
+      ],
+      [
+        "t",
+        "her"
+      ],
+      [
+        "Ġc",
+        "ar"
+      ],
+      [
+        "Ġf",
+        "rom"
+      ],
+      [
+        "Ġne",
+        "w"
+      ],
+      [
+        "Ġp",
+        "ark"
+      ],
+      [
+        "Ġo",
+        "ther"
+      ],
+      [
+        "Ġsa",
+        "d"
+      ],
+      [
+        ",",
+        "\""
+      ],
+      [
+        "Ġw",
+        "hen"
+      ],
+      [
+        "r",
+        "o"
+      ],
+      [
+        "Ġ",
+        "E"
+      ],
+      [
+        "Ġp",
+        "ut"
+      ],
+      [
+        "e",
+        "ll"
+      ],
+      [
+        "Ġh",
+        "ome"
+      ],
+      [
+        "Ġl",
+        "a"
+      ],
+      [
+        "q",
+        "u"
+      ],
+      [
+        "Ġd",
+        "ec"
+      ],
+      [
+        "Ġwh",
+        "o"
+      ],
+      [
+        "Ġf",
+        "ound"
+      ],
+      [
+        "Ġh",
+        "ug"
+      ],
+      [
+        "Ġ",
+        "You"
+      ],
+      [
+        "Ġsay",
+        "s"
+      ],
+      [
+        "ri",
+        "ed"
+      ],
+      [
+        "b",
+        "le"
+      ],
+      [
+        "Ġm",
+        "u"
+      ],
+      [
+        "Ġthe",
+        "n"
+      ],
+      [
+        "a",
+        "ch"
+      ],
+      [
+        "es",
+        "t"
+      ],
+      [
+        "Ġg",
+        "ood"
+      ],
+      [
+        "Ġag",
+        "ain"
+      ],
+      [
+        "e",
+        "p"
+      ],
+      [
+        "Ġf",
+        "l"
+      ],
+      [
+        "Ġ",
+        "L"
+      ],
+      [
+        "Ġwa",
+        "l"
+      ],
+      [
+        "Ġc",
+        "h"
+      ],
+      [
+        "oo",
+        "k"
+      ],
+      [
+        "pp",
+        "ed"
+      ],
+      [
+        "oug",
+        "ht"
+      ],
+      [
+        "p",
+        "l"
+      ],
+      [
+        "ou",
+        "s"
+      ],
+      [
+        "h",
+        "at"
+      ],
+      [
+        "Ġs",
+        "w"
+      ],
+      [
+        "all",
+        "y"
+      ],
+      [
+        "Ġth",
+        "ings"
+      ],
+      [
+        "Ġg",
+        "ot"
+      ],
+      [
+        "Ġplay",
+        "ing"
+      ],
+      [
+        "Ġexc",
+        "ited"
+      ],
+      [
+        "id",
+        "ed"
+      ],
+      [
+        "Ġp",
+        "r"
+      ],
+      [
+        "Ġe",
+        "very"
+      ],
+      [
+        "Ġg",
+        "et"
+      ],
+      [
+        "Ġdec",
+        "ided"
+      ],
+      [
+        "a",
+        "s"
+      ],
+      [
+        "n",
+        "y"
+      ],
+      [
+        "Ġb",
+        "l"
+      ],
+      [
+        "Ġc",
+        "ame"
+      ],
+      [
+        "Ġsc",
+        "ared"
+      ],
+      [
+        "Ġb",
+        "u"
+      ],
+      [
+        "Ġf",
+        "ind"
+      ],
+      [
+        "a",
+        "ve"
+      ],
+      [
+        "Ġbe",
+        "c"
+      ],
+      [
+        "Ġfe",
+        "el"
+      ],
+      [
+        "Ġ",
+        "v"
+      ],
+      [
+        "u",
+        "st"
+      ],
+      [
+        "Ġon",
+        "e"
+      ],
+      [
+        "u",
+        "dd"
+      ],
+      [
+        "Ġd",
+        "own"
+      ],
+      [
+        "Ġli",
+        "ked"
+      ],
+      [
+        "ou",
+        "se"
+      ],
+      [
+        "Ġl",
+        "ot"
+      ],
+      [
+        "n",
+        "a"
+      ],
+      [
+        "u",
+        "e"
+      ],
+      [
+        "Ġc",
+        "are"
+      ],
+      [
+        "Ġ",
+        "D"
+      ],
+      [
+        "Ġth",
+        "an"
+      ],
+      [
+        "Ġd",
+        "ad"
+      ],
+      [
+        "Ġt",
+        "ree"
+      ],
+      [
+        "ar",
+        "n"
+      ],
+      [
+        "way",
+        "s"
+      ],
+      [
+        "a",
+        "p"
+      ],
+      [
+        "Ġg",
+        "ra"
+      ],
+      [
+        "Ġ",
+        "ke"
+      ],
+      [
+        "Ġmom",
+        "my"
+      ],
+      [
+        "ec",
+        "i"
+      ],
+      [
+        "nd",
+        "er"
+      ],
+      [
+        "Ġal",
+        "ways"
+      ],
+      [
+        "Ġdo",
+        "g"
+      ],
+      [
+        "Ġyou",
+        "r"
+      ],
+      [
+        "b",
+        "b"
+      ],
+      [
+        "Ġout",
+        "side"
+      ],
+      [
+        "f",
+        "e"
+      ],
+      [
+        "c",
+        "hed"
+      ],
+      [
+        "an",
+        "t"
+      ],
+      [
+        "Ġtoo",
+        "k"
+      ],
+      [
+        "Ġa",
+        "b"
+      ],
+      [
+        "Ġp",
+        "o"
+      ],
+      [
+        "Ġloo",
+        "k"
+      ],
+      [
+        "is",
+        "e"
+      ],
+      [
+        "er",
+        "s"
+      ],
+      [
+        "Ġdid",
+        "n"
+      ],
+      [
+        "Ġg",
+        "r"
+      ],
+      [
+        "ou",
+        "r"
+      ],
+      [
+        "ĠM",
+        "om"
+      ],
+      [
+        "Ġm",
+        "y"
+      ],
+      [
+        "Ġsp",
+        "eci"
+      ],
+      [
+        "Ġspeci",
+        "al"
+      ],
+      [
+        "The",
+        "y"
+      ],
+      [
+        "ar",
+        "a"
+      ],
+      [
+        "Ġkn",
+        "e"
+      ],
+      [
+        "en",
+        "ed"
+      ],
+      [
+        "Ġab",
+        "out"
+      ],
+      [
+        "n",
+        "na"
+      ],
+      [
+        "it",
+        "e"
+      ],
+      [
+        "Ġth",
+        "ought"
+      ],
+      [
+        "is",
+        "t"
+      ],
+      [
+        "es",
+        "s"
+      ],
+      [
+        "a",
+        "x"
+      ],
+      [
+        "Ġb",
+        "all"
+      ],
+      [
+        "u",
+        "ck"
+      ],
+      [
+        "Ġh",
+        "o"
+      ],
+      [
+        "Ġle",
+        "arn"
+      ],
+      [
+        "Ġ",
+        "F"
+      ],
+      [
+        "Ġm",
+        "ore"
+      ],
+      [
+        "udd",
+        "en"
+      ],
+      [
+        "Ġw",
+        "ill"
+      ],
+      [
+        "Ġtoy",
+        "s"
+      ],
+      [
+        "ĠS",
+        "am"
+      ],
+      [
+        "Ġin",
+        "t"
+      ],
+      [
+        "Ġo",
+        "ld"
+      ],
+      [
+        "udden",
+        "ly"
+      ],
+      [
+        "Ġkn",
+        "ow"
+      ],
+      [
+        "re",
+        "at"
+      ],
+      [
+        "Ġt",
+        "ake"
+      ],
+      [
+        "m",
+        "a"
+      ],
+      [
+        "Ġla",
+        "ug"
+      ],
+      [
+        "g",
+        "ed"
+      ],
+      [
+        "Ġh",
+        "ow"
+      ],
+      [
+        "Ġkne",
+        "w"
+      ],
+      [
+        "Ġt",
+        "a"
+      ],
+      [
+        "Ġsh",
+        "ow"
+      ],
+      [
+        "f",
+        "ter"
+      ],
+      [
+        "a",
+        "u"
+      ],
+      [
+        "re",
+        "t"
+      ],
+      [
+        "Ġan",
+        "y"
+      ],
+      [
+        "Ġp",
+        "e"
+      ],
+      [
+        "Ġc",
+        "at"
+      ],
+      [
+        "is",
+        "h"
+      ],
+      [
+        "Ġmu",
+        "ch"
+      ],
+      [
+        "ĠE",
+        "very"
+      ],
+      [
+        "v",
+        "en"
+      ],
+      [
+        "i",
+        "ve"
+      ],
+      [
+        "in",
+        "k"
+      ],
+      [
+        "Ġt",
+        "w"
+      ],
+      [
+        "Ġs",
+        "k"
+      ],
+      [
+        "Ġcl",
+        "o"
+      ],
+      [
+        "Ġin",
+        "side"
+      ],
+      [
+        "Ġne",
+        "ver"
+      ],
+      [
+        "Ġha",
+        "nd"
+      ],
+      [
+        "B",
+        "ut"
+      ],
+      [
+        "Ġcould",
+        "n"
+      ],
+      [
+        "Ġr",
+        "un"
+      ],
+      [
+        "Ġm",
+        "o"
+      ],
+      [
+        "Ġs",
+        "un"
+      ],
+      [
+        "Ġh",
+        "ouse"
+      ],
+      [
+        "g",
+        "e"
+      ],
+      [
+        "Ġ",
+        "en"
+      ],
+      [
+        "Ġst",
+        "r"
+      ],
+      [
+        "Ġp",
+        "u"
+      ],
+      [
+        "Ġ",
+        "qu"
+      ],
+      [
+        "Ġs",
+        "or"
+      ],
+      [
+        "Ġo",
+        "p"
+      ],
+      [
+        "Ġt",
+        "ried"
+      ],
+      [
+        "i",
+        "a"
+      ],
+      [
+        "an",
+        "k"
+      ],
+      [
+        "a",
+        "ce"
+      ],
+      [
+        "Ġpr",
+        "oud"
+      ],
+      [
+        "Ġs",
+        "l"
+      ],
+      [
+        "Ġto",
+        "ld"
+      ],
+      [
+        "Ġint",
+        "o"
+      ],
+      [
+        "Ġt",
+        "r"
+      ],
+      [
+        "t",
+        "y"
+      ],
+      [
+        "a",
+        "nd"
+      ],
+      [
+        "Ġex",
+        "pl"
+      ],
+      [
+        "i",
+        "on"
+      ],
+      [
+        "Ġwa",
+        "ter"
+      ],
+      [
+        "au",
+        "se"
+      ],
+      [
+        "Ġp",
+        "ick"
+      ],
+      [
+        "Ġc",
+        "om"
+      ],
+      [
+        "ĠM",
+        "ax"
+      ],
+      [
+        "Ġo",
+        "r"
+      ],
+      [
+        "Ġth",
+        "is"
+      ],
+      [
+        "Ġhe",
+        "ard"
+      ],
+      [
+        "Ġo",
+        "ver"
+      ],
+      [
+        "he",
+        "re"
+      ],
+      [
+        "b",
+        "y"
+      ],
+      [
+        "Ġ",
+        "if"
+      ],
+      [
+        "Ġg",
+        "ave"
+      ],
+      [
+        "Ġro",
+        "om"
+      ],
+      [
+        "um",
+        "p"
+      ],
+      [
+        "Ġwa",
+        "t"
+      ],
+      [
+        "et",
+        "ter"
+      ],
+      [
+        "Ġe",
+        "at"
+      ],
+      [
+        "oug",
+        "h"
+      ],
+      [
+        "Ġf",
+        "o"
+      ],
+      [
+        "Ġbec",
+        "ause"
+      ],
+      [
+        "Ġsor",
+        "ry"
+      ],
+      [
+        "Ġplay",
+        "ed"
+      ],
+      [
+        "ot",
+        "h"
+      ],
+      [
+        "ot",
+        "her"
+      ],
+      [
+        "Ġl",
+        "ong"
+      ],
+      [
+        "Ġj",
+        "ust"
+      ],
+      [
+        "d",
+        "y"
+      ],
+      [
+        "Ġof",
+        "f"
+      ],
+      [
+        "Ġhug",
+        "ged"
+      ],
+      [
+        "im",
+        "al"
+      ],
+      [
+        "Ġsm",
+        "all"
+      ],
+      [
+        "Ġe",
+        "ach"
+      ],
+      [
+        "Ġan",
+        "imal"
+      ],
+      [
+        "in",
+        "e"
+      ],
+      [
+        "il",
+        "e"
+      ],
+      [
+        "Ġbe",
+        "ar"
+      ],
+      [
+        "Ġne",
+        "ed"
+      ],
+      [
+        "Ġbo",
+        "x"
+      ],
+      [
+        "Ġu",
+        "nt"
+      ],
+      [
+        "ĠA",
+        "nna"
+      ],
+      [
+        "or",
+        "t"
+      ],
+      [
+        "ĠJ",
+        "ack"
+      ],
+      [
+        "ĠW",
+        "e"
+      ],
+      [
+        "ĠJ",
+        "o"
+      ],
+      [
+        "Ġj",
+        "o"
+      ],
+      [
+        "Ġman",
+        "y"
+      ],
+      [
+        "il",
+        "d"
+      ],
+      [
+        "Ġunt",
+        "il"
+      ],
+      [
+        "u",
+        "c"
+      ],
+      [
+        "Ġt",
+        "ry"
+      ],
+      [
+        "s",
+        "ed"
+      ],
+      [
+        "at",
+        "e"
+      ],
+      [
+        "i",
+        "z"
+      ],
+      [
+        "Ġp",
+        "ret"
+      ],
+      [
+        "s",
+        "el"
+      ],
+      [
+        "v",
+        "ing"
+      ],
+      [
+        "T",
+        "om"
+      ],
+      [
+        "a",
+        "k"
+      ],
+      [
+        "Ġe",
+        "nd"
+      ],
+      [
+        "Ġn",
+        "ice"
+      ],
+      [
+        "ĠS",
+        "ara"
+      ],
+      [
+        "f",
+        "t"
+      ],
+      [
+        "Ġlearn",
+        "ed"
+      ],
+      [
+        "M",
+        "om"
+      ],
+      [
+        "Ġb",
+        "ra"
+      ],
+      [
+        "Ġfl",
+        "ow"
+      ],
+      [
+        "Ġc",
+        "all"
+      ],
+      [
+        "Ġb",
+        "etter"
+      ],
+      [
+        "Ġk",
+        "ind"
+      ],
+      [
+        "g",
+        "ry"
+      ],
+      [
+        "Ġs",
+        "n"
+      ],
+      [
+        "ar",
+        "s"
+      ],
+      [
+        "H",
+        "e"
+      ],
+      [
+        "b",
+        "er"
+      ],
+      [
+        "ur",
+        "n"
+      ],
+      [
+        "Ġlot",
+        "s"
+      ],
+      [
+        "ĠS",
+        "o"
+      ],
+      [
+        "Ġ",
+        "im"
+      ],
+      [
+        "a",
+        "ut"
+      ],
+      [
+        "Ġb",
+        "est"
+      ],
+      [
+        "ll",
+        "y"
+      ],
+      [
+        "'",
+        "m"
+      ],
+      [
+        "e",
+        "m"
+      ],
+      [
+        "p",
+        "t"
+      ],
+      [
+        "Ġd",
+        "re"
+      ],
+      [
+        "ur",
+        "p"
+      ],
+      [
+        "Ġr",
+        "a"
+      ],
+      [
+        "ur",
+        "t"
+      ],
+      [
+        "h",
+        "n"
+      ],
+      [
+        "Ġbe",
+        "aut"
+      ],
+      [
+        "ard",
+        "en"
+      ],
+      [
+        "Ġg",
+        "arden"
+      ],
+      [
+        "ad",
+        "y"
+      ],
+      [
+        "Ġf",
+        "am"
+      ],
+      [
+        "Ġthan",
+        "ked"
+      ],
+      [
+        "Ġa",
+        "d"
+      ],
+      [
+        "in",
+        "y"
+      ],
+      [
+        "ĠThe",
+        "n"
+      ],
+      [
+        "Ġc",
+        "le"
+      ],
+      [
+        "Ġb",
+        "y"
+      ],
+      [
+        "v",
+        "ent"
+      ],
+      [
+        "v",
+        "es"
+      ],
+      [
+        "Ġlaug",
+        "hed"
+      ],
+      [
+        "an",
+        "e"
+      ],
+      [
+        "sel",
+        "f"
+      ],
+      [
+        "le",
+        "w"
+      ],
+      [
+        "Ġe",
+        "ven"
+      ],
+      [
+        "Ġbeaut",
+        "i"
+      ],
+      [
+        "Ġcare",
+        "ful"
+      ],
+      [
+        "Ġanimal",
+        "s"
+      ],
+      [
+        "ĠH",
+        "er"
+      ],
+      [
+        "w",
+        "n"
+      ],
+      [
+        "Ġbeauti",
+        "ful"
+      ],
+      [
+        "Ġc",
+        "he"
+      ],
+      [
+        "Ġf",
+        "i"
+      ],
+      [
+        "Ġw",
+        "o"
+      ],
+      [
+        "Ġh",
+        "ard"
+      ],
+      [
+        "Ġsk",
+        "y"
+      ],
+      [
+        "Ġg",
+        "l"
+      ],
+      [
+        "T",
+        "h"
+      ],
+      [
+        "i",
+        "es"
+      ],
+      [
+        "ĠA",
+        "nd"
+      ],
+      [
+        "Ġwa",
+        "y"
+      ],
+      [
+        "Ġfa",
+        "st"
+      ],
+      [
+        "Ġco",
+        "l"
+      ],
+      [
+        "Ġd",
+        "on"
+      ],
+      [
+        "Ġst",
+        "ay"
+      ],
+      [
+        "Ġre",
+        "m"
+      ],
+      [
+        "Å",
+        "ĵ"
+      ],
+      [
+        "Ġtw",
+        "o"
+      ],
+      [
+        "Ġj",
+        "ump"
+      ],
+      [
+        "Ġit",
+        "s"
+      ],
+      [
+        "Ġc",
+        "ome"
+      ],
+      [
+        "Ġbra",
+        "ve"
+      ],
+      [
+        "Ġli",
+        "ved"
+      ],
+      [
+        "ĠF",
+        "rom"
+      ],
+      [
+        "Y",
+        "es"
+      ],
+      [
+        "Ġlo",
+        "ve"
+      ],
+      [
+        "I",
+        "t"
+      ],
+      [
+        "Ġl",
+        "oud"
+      ],
+      [
+        "Ġt",
+        "e"
+      ],
+      [
+        "ĠH",
+        "is"
+      ],
+      [
+        "Ġb",
+        "oth"
+      ],
+      [
+        "Ġfam",
+        "ily"
+      ],
+      [
+        "ec",
+        "t"
+      ],
+      [
+        "Ġu",
+        "nder"
+      ],
+      [
+        "n",
+        "ing"
+      ],
+      [
+        "oo",
+        "l"
+      ],
+      [
+        "Ġsa",
+        "fe"
+      ],
+      [
+        "Ġst",
+        "ill"
+      ],
+      [
+        "a",
+        "st"
+      ],
+      [
+        "Ġh",
+        "urt"
+      ],
+      [
+        "ĠM",
+        "ia"
+      ],
+      [
+        "a",
+        "se"
+      ],
+      [
+        "Ġc",
+        "u"
+      ],
+      [
+        "Ġb",
+        "ad"
+      ],
+      [
+        "f",
+        "ore"
+      ],
+      [
+        "s",
+        "h"
+      ],
+      [
+        "Ġ",
+        "C"
+      ],
+      [
+        "op",
+        "le"
+      ],
+      [
+        "Ġcall",
+        "ed"
+      ],
+      [
+        "Ġim",
+        "p"
+      ],
+      [
+        "Ġd",
+        "an"
+      ],
+      [
+        "Ġst",
+        "or"
+      ],
+      [
+        "Ġke",
+        "pt"
+      ],
+      [
+        "Ġf",
+        "lew"
+      ],
+      [
+        "Ġr",
+        "ed"
+      ],
+      [
+        "Ġ",
+        "â"
+      ],
+      [
+        "Ġl",
+        "ist"
+      ],
+      [
+        "urp",
+        "r"
+      ],
+      [
+        "W",
+        "hen"
+      ],
+      [
+        "Ġf",
+        "ore"
+      ],
+      [
+        "Ġs",
+        "urpr"
+      ],
+      [
+        "Ġf",
+        "in"
+      ],
+      [
+        "Ġso",
+        "on"
+      ],
+      [
+        "Ġdo",
+        "ll"
+      ],
+      [
+        "l",
+        "ed"
+      ],
+      [
+        "Ġf",
+        "ly"
+      ],
+      [
+        "ĠW",
+        "hen"
+      ],
+      [
+        "c",
+        "t"
+      ],
+      [
+        "Ġl",
+        "et"
+      ],
+      [
+        "Ġwal",
+        "ked"
+      ],
+      [
+        "Ġpe",
+        "ople"
+      ],
+      [
+        "i",
+        "p"
+      ],
+      [
+        "is",
+        "hed"
+      ],
+      [
+        "uc",
+        "y"
+      ],
+      [
+        "he",
+        "s"
+      ],
+      [
+        "Ġsh",
+        "ould"
+      ],
+      [
+        "T",
+        "im"
+      ],
+      [
+        "Ġb",
+        "ook"
+      ],
+      [
+        "ar",
+        "y"
+      ],
+      [
+        "Ġw",
+        "ind"
+      ],
+      [
+        "Ġbe",
+        "fore"
+      ],
+      [
+        "is",
+        "ed"
+      ],
+      [
+        "Ġan",
+        "gry"
+      ],
+      [
+        "Ġexpl",
+        "ore"
+      ],
+      [
+        "Ġfeel",
+        "ing"
+      ],
+      [
+        "Ġro",
+        "ck"
+      ],
+      [
+        "x",
+        "t"
+      ],
+      [
+        "Ġp",
+        "ic"
+      ],
+      [
+        "ig",
+        "h"
+      ],
+      [
+        "âĤ¬",
+        "â"
+      ],
+      [
+        "Ġgo",
+        "ing"
+      ],
+      [
+        "o",
+        "b"
+      ],
+      [
+        "Ġdo",
+        "or"
+      ],
+      [
+        "Ġke",
+        "ep"
+      ],
+      [
+        "Ġ",
+        "P"
+      ],
+      [
+        "Ġc",
+        "on"
+      ],
+      [
+        "L",
+        "et"
+      ],
+      [
+        "Ġop",
+        "ened"
+      ],
+      [
+        "Ġpret",
+        "ty"
+      ],
+      [
+        "Ġcol",
+        "or"
+      ],
+      [
+        "Ġth",
+        "r"
+      ],
+      [
+        "b",
+        "e"
+      ],
+      [
+        "Ġsh",
+        "iny"
+      ],
+      [
+        "Ġwa",
+        "r"
+      ],
+      [
+        "Ġe",
+        "y"
+      ],
+      [
+        "k",
+        "ay"
+      ],
+      [
+        "il",
+        "ly"
+      ],
+      [
+        "Ġwal",
+        "king"
+      ],
+      [
+        "Ġpick",
+        "ed"
+      ],
+      [
+        "Ġd",
+        "ra"
+      ],
+      [
+        "Ġsmil",
+        "e"
+      ],
+      [
+        "i",
+        "ed"
+      ],
+      [
+        "Ġn",
+        "ow"
+      ],
+      [
+        "Ġf",
+        "ar"
+      ],
+      [
+        "Ġno",
+        "ise"
+      ],
+      [
+        "Ġgr",
+        "ound"
+      ],
+      [
+        "ag",
+        "e"
+      ],
+      [
+        "Ġst",
+        "o"
+      ],
+      [
+        "dd",
+        "ed"
+      ],
+      [
+        "Ġcle",
+        "an"
+      ],
+      [
+        "o",
+        "on"
+      ],
+      [
+        "Ġt",
+        "urn"
+      ],
+      [
+        "o",
+        "ck"
+      ],
+      [
+        "Ġad",
+        "vent"
+      ],
+      [
+        "N",
+        "o"
+      ],
+      [
+        "t",
+        "h"
+      ],
+      [
+        "Ġsh",
+        "are"
+      ],
+      [
+        "Ħ",
+        "¢"
+      ],
+      [
+        "âĤ¬â",
+        "Ħ¢"
+      ],
+      [
+        "Ġm",
+        "a"
+      ],
+      [
+        "t",
+        "hing"
+      ],
+      [
+        "ic",
+        "ed"
+      ],
+      [
+        "u",
+        "ch"
+      ],
+      [
+        "Ġclo",
+        "s"
+      ],
+      [
+        "Ġf",
+        "r"
+      ],
+      [
+        "Ġc",
+        "ry"
+      ],
+      [
+        "Ġre",
+        "al"
+      ],
+      [
+        "Ġbl",
+        "ue"
+      ],
+      [
+        "W",
+        "hat"
+      ],
+      [
+        "k",
+        "es"
+      ],
+      [
+        "t",
+        "ure"
+      ],
+      [
+        "Ġne",
+        "xt"
+      ],
+      [
+        "um",
+        "my"
+      ],
+      [
+        "o",
+        "y"
+      ],
+      [
+        "em",
+        "ber"
+      ],
+      [
+        "Ġu",
+        "n"
+      ],
+      [
+        "s",
+        "o"
+      ],
+      [
+        "Ġd",
+        "if"
+      ],
+      [
+        "im",
+        "b"
+      ],
+      [
+        "Ġ",
+        "ide"
+      ],
+      [
+        "Ġcl",
+        "imb"
+      ],
+      [
+        "Ġey",
+        "es"
+      ],
+      [
+        "ĠS",
+        "uddenly"
+      ],
+      [
+        "Ġb",
+        "ed"
+      ],
+      [
+        "Ġha",
+        "s"
+      ],
+      [
+        "le",
+        "s"
+      ],
+      [
+        "Ġal",
+        "so"
+      ],
+      [
+        "T",
+        "hat"
+      ],
+      [
+        "Ġg",
+        "reat"
+      ]
+    ]
+  }
+}

tokenizer_config.json

@@ -0,0 +1,8 @@
+{
+ "tokenizer_class": "PreTrainedTokenizerFast",
+ "model_max_length": 512,
+ "pad_token": "<pad>",
+ "bos_token": "<s>",
+ "eos_token": "</s>",
+ "clean_up_tokenization_spaces": false
+}