Notes

Determinism that is easy to miss

I tried two repositories, p-zielinski/test-llm and p-zielinski/test-llm-2. I wanted to see where text that looks like an idea comes from.

In a large model that determinism is easy to miss, so the output looks creative. I already believed in determinism. The experiment made the comparison sharper: a human brain is like the model, or the model is like the brain, and creativity comes from outside.

  1. Prompt The starting sentence enters as the sequence the rest of the text is counted from.

  2. Next token Each next piece is whatever most often follows the expression before it.

  3. Outside pressure A body, the world, and competition for resources force a search for a new way. Sometimes that search produces a discovery.

The loop I saw then is short.

pseudocode

token = prompt
while not finished:
    token = next_token(token)

A person is not in a vacuum. They have a body, they hear the world, and competition for resources forces a search for new ways. Sometimes that search produces a discovery. That is the whole secret, and I may be overstating it.