Simon Willison · Tech & AI
TIER 4 2026-02-04
<p>I've been exploring Go for building small, fast and self-contained binary applications recently. I'm enjoying how there's generally one obvious way to do things and the resulting code is boring and readable - and something that LLMs are very competent at writing. The one catch is distribution, but it turns out publishing Go binaries to PyPI means any Go binary can be just a <code>uvx package-name</code> call away.</p>
<h4 id="sqlite-scanner">sqlite-scanner</h4>
<p><a href="https://github.com/simonw/sqlite-scanner">sqlite-scanner</a> is my new Go CLI tool for scanning a filesystem for SQLite database files.</p>
<p>It works by checking if the first 16 bytes of the file exactly match the SQLite magic number sequence <code>SQLite format 3\x00</code>. It can search one or more folders recursively, spinning up concurrent goroutines to accelerate the scan. It streams out results as it finds them in plain text, JSON or newline-delimited JSON. It can optionally display the file sizes as well.</p>
<p>To try it out you can download a release from the <a href="https://github.com/simonw/sqlite-scanner/releases">GitHub releases</a> - and then <a href="https://support.apple.com/en-us/102445">jump through macOS hoops</a> to execute an "unsafe" binary. Or you can clone the repo and compile it with Go. Or... you can run the binary like this:</p>
<pre><code>uvx sqlite-scanner
</code></pre>
<p>By default this will search your current directory for SQLite databases. You can pass one or more directories as arguments:</p>
<pre><code>uvx sqlite-scanner ~ /tmp
</code></pre>
<p>Add <code>--json</code> for JSON output, <code>--size</code> to include file sizes or <code>--jsonl</code> for newline-delimited JSON. Here's a demo:</p>
<pre><code>uvx sqlite-scanner ~ --jsonl --size
</code></pre>
<p><img src="https://static.simonwillison.net/static/2025/sqlite-scanner-demo.gif" alt="running that command produces a sequence of JSON objects, each with a path and a size key" style="max-width: 100%;" /></p>
<p>If you haven't been uv-pilled yet you can instead install <code>sqlite-scanner</code> using <code>pip install sqlite-scanner</code> and then run <code>sqlite-scanner</code>.</p>
<p>To get a permanent copy with <code>uv</code> use <code>uv tool install sqlite-scanner</code>.</p>
<h4 id="how-the-python-package-works">How the Python package works</h4>
<p>The reason this is worth doing is that <code>pip</code>, <code>uv</code> and <a href="https://pypi.org/">PyPI</a> will work together to identify the correct compiled binary for your operating system and architecture.</p>
<p>This is driven by file names. If you visit <a href="https://pypi.org/project/sqlite-scanner/#files">the PyPI downloads for sqlite-scanner</a> you'll see the following files:</p>
<ul>
<li><code>sqlite_scanner-0.1.1-py3-none-win_arm64.whl</code></li>
<li><code>sqlite_scanner-0.1.1-py3-none-win_amd64.whl</code></li>
<li><code>sqlite_scanner-0.1.1-py3-none-musllinux_1_2_x86_64.whl</code></li>
<li><code>sqlite_scanner-0.1.1-py3-none-musllinux_1_2_aarch64.whl</code></li>
<li><code>sqlite_scanner-0.1.1-py3-none-manylinux_2_17_x86_64.whl</code></li>
<li><code>sqlite_scanner-0.1.1-py3-none-manylinux_2_17_aarch64.whl</code></li>
<li><code>sqlite_scanner-0.1.1-py3-none-macosx_11_0_arm64.whl</code></li>
<li><code>sqlite_scanner-0.1.1-py3-none-macosx_10_9_x86_64.whl</code></li>
</ul>
<p>When I run <code>pip install sqlite-scanner</code> or <code>uvx sqlite-scanner</code> on my Apple Silicon Mac laptop Python's packaging magic ensures I get that <code>macosx_11_0_arm64.whl</code> variant.</p>
<p>Here's <a href="https://tools.simonwillison.net/zip-wheel-explorer?url=https%3A%2F%2Ffiles.pythonhosted.org%2Fpackages%2F88%2Fb1%2F17a716635d2733fec53ba0a8267f85bd6b6cf882c6b29301bc711fba212c%2Fsqlite_scanner-0.1.1-py3-none-macosx_11_0_arm64.whl#sqlite_scanner/__init__.py">what's in the wheel</a>, which is a zip file with a <code>.whl</code> extension.</p>
<p>In addition to the <code>bin/sqlite-scanner</code> the most important file is <code>sqlite_scanner/__init__.py</code> which includes the following:</p>
<pre><span class="pl-k">def</span> <span class="pl-en">get_binary_path</span>():
<span class="pl-s">"""Return the path to the bundled binary."""</span>
<span class="pl-s1">binary</span> <span class="pl-c1">=</span> <span class="pl-s1">os</span>.<span class="pl-c1">path</span>.<span class="pl-c1">join</span>(<span class="pl-s1">os</span>.<span class="pl-c1">path</span>.<span class="pl-c1">dirname</span>(<span class="pl-s1">__file__</span>), <span class="pl-s">"bin"</span>, <span class="pl-s">"sqlite-scanner"</span>)
<span class="pl-c"># Ensure binary is executable on Unix</span>
<span class="pl-k">if</span> <span class="pl-s1">sys</span>.<span class="pl-c1">platform</span> <span class="pl-c1">!=</span> <span class="pl-s">"win32"</span>:
<span class="pl-s1">current_mode</span> <span class="pl-c1">=</span> <span class="pl-s1">os</span>.<span class="pl-c1">stat</span>(<span class="pl-s1">binary</span>).<span class="pl-c1">st_mode</span>
<span class="pl-k">if</span> <span class="pl-c1">not</span> (<span class="pl-s1">current_mode</span> <span class="pl-c1">&</span> <span class="pl-s1">stat</span>.<span class="pl-c1">S_IXUSR</span>):
<span class="pl-s1">os</span>.<span class="pl-c1">chmod</span>(<span class="pl-s1">binary</span>, <span class="pl-s1">current_mode</span> <span class="pl-c1">|</span> <span class="pl-s1">stat</span>.<span class="pl-c1">S_IXUSR</span> <span class="pl-c1">|</span> <span class="pl-s1">stat</span>.<span class="pl-c1">S_IXGRP</span> <span class="pl-c1">|</span> <span class="pl-s1">stat</span>.<span class="pl-c1">S_IXOTH</span>)
<span class="pl-k">return</span> <span class="pl-s1">binary</span>
<span class="pl-k">def</span> <span class="pl-en">main</span>():
<span class="pl-s">"""Execute the bundled binary."""</span>
<span class="pl-s1">binary</span> <span class="pl-c1">=</span> <span class="pl-en">get_binary_path</span>()
<span class="pl-k">if</span> <span class="pl-s1">sys</span>.<span class="pl-c1">platform</span> <span class="pl-c1">==</span> <span class="pl-s">"win32"</span>:
<span class="pl-c"># On Windows, use subprocess to properly handle signals</span>
<span class="pl-s1">sys</span>.<span class="pl-c1">exit</span>(<span class="pl-s1">subprocess</span>.<span class="pl-c1">call</span>([<span class="pl-s1">binary</span>] <span class="pl-c1">+</span> <span class="pl-s1">sys</span>.<span class="pl-c1">argv</span>[<span class="pl-c1">1</span>:]))
<span class="pl-k">else</span>:
<span class="pl-c"># On Unix, exec replaces the process</span>
<span class="pl-s1">os</span>.<span class="pl-c1">execvp</span>(<span class="pl-s1">binary</span>, [<span class="pl-s1">binary</span>] <span class="pl-c1">+</span> <span class="pl-s1">sys</span>.<span class="pl-c1">argv</span>[<span class="pl-c1">1</span>:])</pre>
<p>That <code>main()</code> method - also called from <code>sqlite_scanner/__main__.py</code> - locates the binary and executes it when the Python package itself is executed, using the <code>sqlite-scanner = sqlite_scanner:main</code> entry point defined in the wheel.</p>
<h4 id="which-means-we-can-use-it-as-a-dependency">Which means we can use it as a dependency</h4>
<p>Using PyPI as a distribution platform for Go binaries feels a tiny bit abusive, albeit <a href="https://simonwillison.net/2022/May/23/bundling-binary-tools-in-python-wheels/">there is plenty of precedent</a>.</p>
<p>I’ll justify it by pointing out that this means <strong>we can use Go binaries as dependencies</strong> for other Python packages now.</p>
<p>That's genuinely useful! It means that any functionality which is available in a cross-platform Go binary can now be subsumed into a Python package. Python is really good at running subprocesses so this opens up a whole world of useful tricks that we can bake into our Python tools.</p>
<p>To demonstrate this, I built <a href="https://github.com/simonw/datasette-scan">datasette-scan</a> - a new Datasette plugin which depends on <code>sqlite-scanner</code> and then uses that Go binary to scan a folder for SQLite databases and attach them to a Datasette instance.</p>
<p>Here's how to use that (without even installing anything first, thanks <code>uv</code>) to explore any SQLite databases in your Downloads folder:</p>
<div class="highlight highlight-source-shell"><pre>uv run --with datasette-scan datasette scan <span class="pl-k">~</span>/Downloads</pre></div>
<p>If you peek at the code you'll see it <a href="https://github.com/simonw/datasette-scan/blob/1a2b6d1e6b04c8cd05f5676ff7daa877efd99f08/pyproject.toml#L14">depends on sqlite-scanner</a> in <code>pyproject.toml</code> and calls it using <code>subprocess.run()</code> against <code>sqlite_scanner.get_binary_path()</code> in its own <a href="https://github.com/simonw/datasette-scan/blob/1a2b6d1e6b04c8cd05f5676ff7daa877efd99f08/datasette_scan/__init__.py#L38-L58">scan_directories() function</a>.</p>
<p>I've been exploring this pattern for other, non-Go binaries recently - here's <a href="https://github.com/simonw/tools/blob/main/python/livestream-gif.py">a recent script</a> that depends on <a href="https://pypi.org/project/static-ffmpeg/">static-ffmpeg</a> to ensure that <code>ffmpeg</code> is available for the script to use.</p>
<h4 id="building-python-wheels-from-go-packages-with-go-to-wheel">Building Python wheels from Go packages with go-to-wheel</h4>
<p>After trying this pattern myself a couple of times I realized it would be useful to have a tool to automate the process.</p>
<p>I first <a href="https://claude.ai/share/2d9ced56-b3e8-4651-83cc-860b9b419187">brainstormed with Claude</a> to check that there was no existing tool to do this. It pointed me to <a href="https://www.maturin.rs/bindings.html#bin">maturin bin</a> which helps distribute Rust projects using Python wheels, and <a href="https://github.com/Bing-su/pip-binary-factory">pip-binary-factory</a> which bundles all sorts of other projects, but did not identify anything that addressed the exact problem I was looking to solve.</p>
<p>So I <a href="https://gisthost.github.io/?41f04e4eb823b1ceb888d9a28c2280dd/index.html">had Claude Code for web build the first version</a>, then refined the code locally on my laptop with the help of more Claude Code and a little bit of OpenAI Codex too, just to mix things up.</p>
<p>The full documentation is in the <a href="https://github.com/simonw/go-to-wheel">simonw/go-to-wheel</a> repository. I've published that tool to PyPI so now you can run it using:</p>
<div class="highlight highlight-source-shell"><pre>uvx go-to-wheel --help</pre></div>
<p>The <code>sqlite-scanner</code> package you can <a href="https://pypi.org/project/sqlite-scanner/">see on PyPI</a> was built using <code>go-to-wheel</code> like this:</p>
<div class="highlight highlight-source-shell"><pre>uvx go-to-wheel <span class="pl-k">~</span>/dev/sqlite-scanner \
--set-version-var main.version \
--version 0.1.1 \
--readme README.md \
--author <span class="pl-s"><span class="pl-pds">'</span>Simon Willison<span class="pl-pds">'</span></span> \
--url https://github.com/simonw/sqlite-scanner \
--description <span class="pl-s"><span class="pl-pds">'</span>Scan directories for SQLite databases<span class="pl-pds">'</span></span></pre></div>
<p>This created a set of wheels in the <code>dist/</code> folder. I tested one of them like this:</p>
<div class="highlight highlight-source-shell"><pre>uv run --with dist/sqlite_scanner-0.1.1-py3-none-macosx_11_0_arm64.whl \
sqlite-scanner --version</pre></div>
<p>When that spat out the correct version number I was confident everything had worked as planned, so I pushed the whole set of wheels to PyPI using <code>twine upload</code> like this:</p>
<div class="highlight highlight-source-shell"><pre>uvx twine upload dist/<span class="pl-k">*</span></pre></div>
<p>I had to paste in a PyPI API token I had saved previously.</p>
<h4 id="i-expect-to-use-this-pattern-a-lot">I expect to use this pattern a lot</h4>
<p><code>sqlite-scanner</code> is very clearly meant as a proof-of-concept for this wider pattern - Python is very much capable of recursively crawling a directory structure looking for files that start with a specific byte prefix on its own!</p>
<p>That said, I think there's a <em>lot</em> to be said for this pattern. Go is a great complement to Python - it's fast, compiles to small self-contained binaries, has excellent concurrency support and a rich ecosystem of libraries.</p>
<p>Go is similar to Python in that it has a strong standard library. Go is particularly good for HTTP tooling - I've built several HTTP proxies in the past using Go's excellent <code>net/http/httputil.ReverseProxy</code> handler.</p>
<p>I've also been experimenting with <a href="https://github.com/wazero/wazero">wazero</a>, Go's robust and mature zero dependency WebAssembly runtime as part of my ongoing quest for the ideal sandbox for running untrusted code. <a href="https://github.com/simonw/research/tree/main/wasm-repl-cli">Here's my latest experiment</a> with that library.</p>
<p>Being able to seamlessly integrate Go binaries into Python projects without the end user having to think about Go at all - they <code>pip install</code> and everything Just Works - feels like a valuable addition to my toolbox.</p>