Arto Bendiken artob

How large are large language models? (2025)

This aims to be factual information about the size of large language models. None of this document was written by AI. I do not include any information from leaks or rumors. The focus of this document is on base models (the raw text continuation engines, not 'helpful chatbot/assistants'). This is a view from a few years ago to today of one very tiny fraction of the larger LLM story that's happening.

History

GPT-2,-medium,-large,-xl (2019): 137M, 380M, 812M, 1.61B. Source: openai-community/gpt2. Trained on the unreleased WebText dataset said to 40GB of Internet text - I estimate that to be roughly 10B tokens. You can see a list of the websites that went into that data set here domains.txt.
GPT-3 aka davinci, davinci-002 (2020): 175B parameters. There is a good breakdown of how those parameters are 'spent' here [How d

Fast Discrete Program Search 2

I am investigating how to use Bend (a parallel language) to accelerate Symbolic AI; in special, Discrete Program Search. Basically, think of it as an alternative to LLMs, GPTs, NNs, that is also capable of generating code, but by entirely different means. This kind of approach was never scaled with mass compute before - it wasn't possible! - but Bend changes this. So, my idea was to do it, and see where it goes.

Now, while I was implementing some candidate algorithms on Bend, I realized that, rather than mass parallelism, I could use an entirely different mechanism to speed things up: SUP Nodes. Basically, it is a feature that Bend inherited from its underlying model ("Interaction Combinators") that, in simple terms, allows us to combine multiple functions into a single superposed one, and apply them all to an argument "at the same time". In short, it allows us to call N functions at a fraction of the expected cost. Or, in simple terms: why parallelize when we can share?

A

There are a few lists already, I thought I'd create another one so we can have a long list of lists someday. 🤠 Ordered by approximately which ones I actually use often.

Modern CLI Replacements

zoxide - A smarter cd command. Supports all major shells.
mcfly - Fly through your shell history. Great Scott!
Procs - A modern replacement for ps written in Rust
sd - Intuitive find & replace CLI (sed alternative)
dust - A more intuitive version of du in rust
hyperfine - A command-line benchmarking tool

This is the predecessor of Mazeppa.

Supercompilation is a deep program transformation technique due to V. F. Turchin, a prominent computer scientist, cybernetician, physicist, and Soviet dissident. He described the concept as follows [^supercompiler-concept]:

A supercompiler is a program transformer of a certain type. The usual way of thinking about program transformation is in terms of some set of rules which preserve the functional meaning of the program, and a step-by-step application of these rules to the initial program. ... The concept of a supercompiler is a product of cybernetic thinking. A program is seen as a machine. To make sense of it, one must observe its operation. So a supercompiler does not transform the program by steps; it controls and observes (SUPERvises) the running of the machine that is represented by th

Personal notes building zig 0.15.0-dev.xxxx on Macbook M3 Max

20250711 Sequoia 15.5, latest XCode Commandline Tools, homebrew-installed LLVM 20.1.8 (llvm@20)

This way of building probably works on all Apple Silicon Macs, but you must check other models yourself. Some Intel Macs should also work (default homebrew path is the only significant difference). But I don't test that anymore.

Here is a list of some previously successful M3 Max builds

DO YOU REALLY NEED TO BUILD FROM SOURCE?

We need to talk about group.

I'm sorry to tell you, but group is gone.

If you check out the latest k.d on shakti.com as at 28 March 2023, you'll see that 'unary' = is now 'freq' (frequency).

Group had a long life: it's been around since at least k2, or 1996.

So why did group go? And what should we use instead?

The reverse turing test

I asked GPT-4 to come up with 10 questions to determine if the answerer was AI or human.

I provided my own answers for these questions and I also asked ChatGPT to answer them.

The result is that GPT-4 was able to correctly differentiate between AI and Human.

Purpose

Bootstrap knowledge of LLMs ASAP. With a bias/focus to GPT.

Avoid being a link dump. Try to provide only valuable well tuned information.

Prelude

Neural network links before starting with transformers.

	#!/usr/bin/env python3

	import os
	import json

	from Crypto.Protocol.KDF import PBKDF2
	from Crypto.Hash import SHA1
	from Crypto.Cipher import AES
	from Crypto.Util.Padding import unpad

	import 'dart:ffi';
	import 'dart:isolate';
	import 'dart:io';
	import 'package:ffi/ffi.dart';

	void newIsolate(SendPort sendPort) {
	final ptr = calloc.allocate<Int>(100);
	ptr[0] = 42;
	sendPort.send(ptr.address);
	sleep(const Duration(seconds: 1));