A compressor like gzip can be used for language modeling without neural networks by scoring continuations based on their compressed length. Gzip's DEFLATE algorithm uses a sliding window to find matches and compresses them, allowing it to predict the next bytes in a text prompt.