emjupy is a multi-file Emacs package, which was developed with use of
Claude Opus 5.5 (and other LLMs). To build a tar and install it:
make package # generates emjupy-i.j.k.tar
and install from Emacs: M-x package-install-file RET emjupy-i.j.k.tar
RET with i.j.k the current version.
The only external dependency is websocket (1.15+); Emacs 30.1+ is
required. Installed as a package, the commands you start from –
emjupy-login, emjupy-open-notebook, emjupy-server-dashboard – are
autoloaded, so they work without any require.
Which Emacs to develop against
Development targets Emacs 30.1, which is also the minimum the package declares: what CI runs, and what most users are on. 29 was supported until Eglot's differences there became a source of bugs – one sent every language server request to a local process, answering about the wrong machine's files, and stayed invisible for weeks because 29 was where it was being tested.
make versions says which Emacs is in use and whether it is the target.
Source layout
The sources are in src/, listed here from the bottom layer up. Each
file requires only files earlier in this table, never later ones.
| File | Contains |
|---|---|
emjupy-core.el |
structs, state, server registry, lookups |
emjupy-http.el |
REST transport, URL parsing, the HTTP error types |
emjupy-render.el |
faces, colours, outlines, drawing output, text helpers |
emjupy-cells.el |
buffer<->struct syncing, cell editing and cell commands |
emjupy-kernel.el |
kernel WebSocket, execution, message routing |
emjupy-figures.el |
opening an output on its own: plotly figures, HTML, images |
emjupy-widgets.el |
widget controls in the buffer, and what they cause |
emjupy-widget-page.el |
widgets that draw themselves, live in a page, through Emacs |
emjupy-remote.el |
where a server's files live: roots, SSH tunnels, TRAMP |
emjupy-lsp.el |
language-server WebSocket transport, shadow document text |
emjupy-eglot.el |
shadow buffer, Eglot integration, completion, eldoc |
emjupy-mode.el |
emjupy-mode, its keymap and menu |
emjupy-notebook.el |
login, the notebook browser, .ipynb parse and serialize |
emjupy.el |
package header, version, and the entry point that loads it |
emjupy-pkg.el |
package metadata for make package (MELPA makes its own) |
Dependencies run one way
No file calls a function defined in a file later in the table, and
there are no declare-function forms for emjupy's own functions. Every
module can be loaded on its own, which is what makes that checkable:
(require 'emjupy-lsp) in a fresh Emacs must work without anything else
having been loaded first.
When a lower layer needs to tell a higher one that something happened, it runs a hook rather than naming the function. There are these:
emjupy-kernel-connected-functions, run by the kernel layer when a kernel attaches.emjupy-modehangs two handlers on it: one asks the kernel where it runs, for the language server; the other tells the kernel the cell width.emjupy-box-width-changed-functions, run by the drawing layer when the width the cells are drawn at changes, so the kernel is told the new width.emjupy-comm-functions, run by the kernel layer with each comm message, andemjupy-output-owner-functions, asked by it which cell output answering a request no cell waits on belongs to. The widgets layer keeps its models from the first, and claims with the second the output a moved slider causes. It also setsemjupy-widget-view-function, which the drawing layer calls to draw a widget.
Both are wired when a notebook buffer starts, not when a file is loaded – a package should not add hooks just by being loaded.
A keymap may name a command defined further down the table, with a plain
quote rather than #': a binding needs the name, and resolves it when
the key is pressed.
Errors
Catch the failure you expect, and let everything else through. There is
no ignore-errors in the package: it hides the failure that was
expected and every bug alongside it, and a function that has silently
done nothing for months looks exactly like one that works.
- An expected failure that has a precondition is checked, not caught –
window-live-p,buffer-live-p,websocket-pandwebsocket-openp,jsonrpc-running-p,xref-file-location-p. - One that can only be found out by trying is caught by its type:
json-error– the whole family: a reply nested too deep, or bytes that are not UTF-8, is ajson-errorbut not ajson-parse-error–jsonrpc-error,file-error. - A request to the Jupyter server that fails signals
emjupy-http-error, oremjupy-http-statuswhen the server answered with an error status. Both are children offile-error, the family Emacs uses for a resource it cannot reach, which also keeps their messages readable.emjupy--http-status-ofextracts the status;emjupy--http-exists-panswers "is this there" with nil for a 404 instead of signalling. - A path into the Contents API goes through
emjupy--contents-path, never a bareconcat: a notebook name can hold#or?, which in a URL start a fragment or a query, soa#b.ipynbwas asked for asa– and saved there. - Background work that must not abort the command it runs under – the
language server warming up while a notebook opens – uses
with-demoted-errors, so a failure is reported rather than lost.
Makefile targets
| Target | Effect |
|---|---|
make compile |
byte-compile the sources, with src/ first on the load path |
make test |
the unit tests; integration tests skip without a server |
make check |
unit and integration tests; refuses to run without a server |
make lint |
checkdoc, and package-lint if it is installed |
make docs |
export docs/*.org to the site in html-content/ |
make package |
build emjupy-VERSION.tar |
make install |
install that tar via package-install-file |
make coverage |
line coverage of the suite, as lcov, with a per-file table |
make set-version |
write a new version everywhere: make set-version VERSION=X.Y.Z |
make check-version |
the version header, emjupy-version and the pkg file agree |
make versions |
which Emacs this is, against the one development targets |
make clean |
remove .elc files and the tar |
make timestamps |
touch every source, to force a full rebuild |
Continuous integration
Each check runs once per event, in one workflow unless only another can do it:
| Event | Runs |
|---|---|
push to main, pull request |
build-package.yml: compile, lint, melpazoid, docs, and the tests – unit and integration, once |
push to main changing src/ |
and coverage.yml, for the Codecov badge |
| tag | release.yml only |
| by hand | set-version.yml, or any of the above from the Actions tab |
Pushes to other branches run nothing until a pull request is opened, and a newer push to the same branch cancels a run still going. Build also runs for changes to the documentation, the tools, the notebook corpus and the MELPA recipe, not only the Lisp.
The release tests what it ships, and is the only run a tag starts: the
version commit is often pushed with [skip ci], so nothing else has
tested it. Path filters do not apply to tag pushes, which is why a tag
used to start Build and Coverage as well – the suite four times over.
Running everything locally
What CI runs, in the order worth running it: the cheap checks first, so a typo does not cost a server start-up.
Once: what the checks need
- Emacs 30.1 –
make versionssays whether you have it. websocket.el, the one dependency. An installed copy is found; a checkout can be named instead withexport EMJUPY_WEBSOCKET_DIR=/path/to/emacs-websocket.For the integration tests, a Python environment with the server, a kernel and a language server:
bashpip install jupyter-server ipykernel jupyter-lsp 'python-lsp-server[all]'- For
make lintto run package-lint as well as checkdoc,M-x package-install RET package-lint. Without it the target says so and checks the rest. For melpazoid, a clone and Docker – melpazoid runs inside a container, which is what makes its result match MELPA's:
bashgit clone https://github.com/riscy/melpazoid ~/melpazoid
Every time
make compile # no warnings expected make test # unit tests, about ten seconds make lint # checkdoc and package-lint: nothing expected make check-version # three places agree on the version
Then the integration tests, against a server of your own:
mkdir -p /tmp/nbroot jupyter server --no-browser --port=8888 --ip=127.0.0.1 \ --IdentityProvider.token=abc --ServerApp.root_dir=/tmp/nbroot & EMJUPY_TEST_URL=127.0.0.1:8888 EMJUPY_TEST_TOKEN=abc \ EMJUPY_TEST_ROOT=/tmp/nbroot make check
EMJUPY_SKIP_UNSTABLE=1 leaves out the tests tagged :unstable, as CI
does; without it they run, and a failure there is real but known.
Then melpazoid, reading the same recipe MELPA will be sent:
RECIPE="$(cat .github/melpa-recipe)" LOCAL_REPO="$PWD" make -C ~/melpazoid
Coverage, when you want to see what the tests leave untouched. It needs
undercover.el and its two dependencies, installed or checked out:
git clone https://github.com/undercover-el/undercover.el ~/src/undercover git clone https://github.com/magnars/dash.el ~/src/dash git clone https://github.com/rejeep/shut-up.el ~/src/shut-up export EMJUPY_UNDERCOVER_PATH=~/src/undercover:~/src/dash:~/src/shut-up make coverage # prints a per-file table; lcov in coverage/
With the integration variables set it covers the server-facing code as
well, which is how CI runs it. Tests tagged :timing are left out: they
assert on elapsed time, and every function is several times slower when
instrumented.
And the documentation, with the link check CI runs – every page and every anchor a link names must exist:
make docs # then open html-content/index.html python3 tools/check-links.py
Both the build and the release run that one script; a link to a section
of another page is checked to its anchor, so link to a heading by a
CUSTOM_ID – Org cannot resolve a heading searched for in another file.
Compiling one file by hand
M-x byte-compile-file compiles against whatever load-path holds. With
an emjupy installed from a package archive, (require 'emjupy-core) finds
the INSTALLED copy, and a module added since then is not found at all –
"Cannot open load file: emjupy-remote" is that, not a fault in the file.
make compile puts src/ first and avoids it; by hand, add the checkout
to the front of load-path, or uninstall the package while working on
it.
emjupy-run-tests.el, lint.el, emjupy-pkg.el and publish.el are not
compiled at all – each says no-byte-compile – because they are scripts
and metadata rather than code to load. publish.el matters most:
package-vc compiles everything in a checkout, so without it an install
from git would warn about the site's build script.
Unit tests
The self-contained suite requires only websocket:
make test # which is emacs -batch -Q -L src -l src/emjupy-run-tests.el
websocket is resolved from load-path first, then from
EMJUPY_WEBSOCKET_DIR, and only then from GNU ELPA – so the suite
also runs offline / in CI:
EMJUPY_WEBSOCKET_DIR=/path/to/emacs-websocket make test
Integration tests (real server, real kernel, real LSP)
emjupy-integration-test.el talks to a live Jupyter server and runs
real Python in a real kernel. These are opt-in and ert-skip unless
the environment points at a running server, so the default run stays
fast and hermetic.
Some things only break against a live server – an image arriving that this Emacs cannot display, a WebSocket URL that drops its base path, a saved notebook no other tool will open – so they are worth running before trusting a change.
| Variable | Meaning |
|---|---|
EMJUPY_TEST_URL |
host:port of a running server (the LOCAL end of a tunnel) |
EMJUPY_TEST_TOKEN |
its token |
EMJUPY_TEST_ROOT |
server root dir, for the nbformat-validation test |
EMJUPY_TEST_URL2 |
a SECOND server, for the multi-server tests |
EMJUPY_TEST_TOKEN2 |
its token (defaults to EMJUPY_TEST_TOKEN) |
To exercise the remote path, point EMJUPY_TEST_URL at the local end
of an ssh tunnel rather than at the server directly:
# server side: jupyter listens only on loopback jupyter server --no-browser --port=8888 --ip=127.0.0.1 --IdentityProvider.token=abc # client side: forward a local port through ssh ssh -N -L 18888:127.0.0.1:8888 user@remote-host & EMJUPY_TEST_URL=127.0.0.1:18888 EMJUPY_TEST_TOKEN=abc \ EMJUPY_TEST_ROOT=/path/to/notebook/root make check
The language-server tests come in two kinds. Those of the WebSocket
transport need jupyter-lsp and python-lsp-server in the environment
running the server – the pip line above installs both. Those of the
local fallback need a language server on this machine's PATH (pylsp,
pyright-langserver or jedi-language-server), and skip if none is
found.
Robustness tests
src/emjupy-robustness-test.el holds the tests that make their own
inputs. They run with the rest of make test, and each found a bug
the hand-written tests had not.
| Test | What it tries |
|---|---|
| message order | random interleavings of a kernel's replies and outputs |
| random edits | random sequences of cell commands, invariants after each |
| undo everything | random edits, then undo to the start |
| round-trip | every notebook in src/test-corpus/, saved and reloaded |
| hostile content | escape floods, huge SVGs, deep JSON, awkward file names |
| leaks | many sessions; timers, processes, hooks, tables after |
Seeds
Each run draws a seed, and every failure of a randomised test names it:
Info: EMJUPY_FUZZ_SEED=4, case 5 Info: after (insert-above insert-below delete-char split move-down)
Set it to replay the run exactly; EMJUPY_FUZZ_ITERATIONS sets how many
cases each test tries – by default from ten, for the slow ones, to two
hundred, for the message orders:
EMJUPY_FUZZ_SEED=4 EMJUPY_FUZZ_ITERATIONS=500 make test
A failing sequence is usually long. Before working on one, shrink it: replay it with a step removed, and keep the removal while it still fails. The separator bug the edit fuzz found was some twenty steps as found and five once shrunk, and understood from those, three: insert a cell, insert another before it, backspace.
Expected failures
A test for a fault that is known but not yet fixed is marked
:expected-result :failed, with the reason in its docstring. The suite stays green, the fault stays visible, and
a fix shows up as an unexpected pass – the cue to take the marking off.
The notebook corpus
src/test-corpus/ is written by tools/make-notebook-corpus.py, with
nbformat itself, so each notebook is valid by Jupyter's own
definition. tools/check-roundtrip.py compares a notebook with emjupy's
save of it – every cell, source, output, metadata and attachment – and
validates the save; the round-trip test calls it when Python has
nbformat, and otherwise checks only that saving twice changes nothing.
Add a notebook to the generator, not by hand, when a new shape turns up.
Network faults
tools/faultproxy.py sits between emjupy and the server and misbehaves
on command: it drops every connection, stalls, delays or splits what it
forwards. The emjupy-int-fault-* tests start it in front of the test
server, so they run with the integration tests and need python3. It
can be driven by hand as well:
python3 tools/faultproxy.py 127.0.0.1 8888 # prints its port and control port echo stall | nc 127.0.0.1 CONTROL-PORT # or sever, resume, delay 200, chunk 7, reset
Compiled files
Emacs loads a .elc in preference to its .el, newer or not. To check
a test fails without a fix, run make clean first – otherwise the old
code is the compiled copy of the new, and the test passes for no reason.
Test hygiene
A unit test must give the same answer on every machine. Two things in emjupy make that easy to lose: the host of an SSH tunnel is found by reading the process list, and whether LaTeX can be rendered is decided by looking for the programs that render it. Both are correct in use and wrong in a test.
emjupy-probe-environment governs both, and the unit test runner binds it
to nil. A test that needs the detection tested – as opposed to merely
used – binds it back to t and stubs what it examines.
This is not hypothetical: two tests asking what a tunnelled server resolves to passed here and failed on a machine that happened to be forwarding that port, and a LaTeX test depended on whether LaTeX was installed.
Other rules worth following, each learned from a test that passed while the thing it covered was broken:
- Test the key binding, not only the function behind it. A command
bound to
DELalone is not reached in a graphical Emacs that sends<backspace>. - Do not wrap the action under test in
ignore-errors. A test that guards against remote connections passed because the connection it should have caught raised an error which was then swallowed. - When a stub replaces a function used by Emacs itself, call through for the cases the test is not about.
- Prefer asserting an invariant over asserting a mechanism; mechanisms differ between Emacs versions, invariants do not.
Invariants
M-x emjupy-debug-check-invariants reports how a buffer and its cells
disagree: which cell, and whether its overlay is missing, overlapping,
out of order, detached from its output box, or showing text the cell
does not hold. The last means what is displayed and what would be saved
have come apart.
Tests call it after every operation. It has caught overlays left overlapping after a split, and a sync that turned an empty cell into one containing a newline.
Performance
Three tests guard against regressions rather than measuring speed:
- work done after every command does not grow with the notebook;
- redrawing output does not grow with the notebook, since only the cell concerned is redrawn;
- nothing on a path that runs after a command touches a remote file.
The last matters most. A remote operation costs a round trip, TRAMP blocks Emacs while it waits, and anything on the per-command path pays that on every keystroke. The test asserts the absence of such calls rather than a duration, since a duration would depend on a network.
The per-command costs these keep small are measured in Language server support.
Releasing
- Set the version. From GitHub: Actions, "Set version", "Run
workflow", and give it the new version as
X.Y.Z. It writes the version everywhere the current one appears – the;; Version:header,emjupy-version,src/emjupy-pkg.eland the sample output indocs/usage.org– checks they agree, and commits the change. Or locally, the same script:make set-version VERSION=X.Y.Z.
Tag the commit with the same version and push the tag. That starts the release: the tests, the tarball on the release page, and the documentation site, all from that commit. The workflow does not tag for you: a tag pushed from a workflow does not start other workflows, so the release would not run.
A commit whose message contains
[skip ci]is skipped for tag pushes too. Tag a commit without it, or the release never starts.
The package's own description
C-h P emjupy shows the ;;; Commentary: section of src/emjupy.el:
what emjupy is, what it needs, and how to start. It is written for
someone who has only just installed the package and has not seen the
repository.
package.el prefers an installed package's README to its Commentary, and
the README is a web page – badges, screenshots, links into the
repository – so the tarball make package builds carries no README.
MELPA leaves it out for the same reason, so both describe themselves
alike. An install from git with package-vc is the whole checkout, and
shows the README.
The opening paragraph is the same in three places – the Commentary, the
README and docs/index.org – and is worth keeping so when one changes.
The one-line summary is the same in the header of src/emjupy.el and in
src/emjupy-pkg.el.
Submitting to MELPA
The recipe must list the files explicitly. A recipe without :files
takes every .el in the repository, which here would ship the tests to
users and include emjupy-run-tests.el, a script with no provide form
that package-lint rejects.
The recipe used is kept at .github/melpa-recipe:
(emjupy :fetcher github :repo "mathren/emjupy" :files ("src/emjupy.el" "src/emjupy-core.el" "src/emjupy-http.el" "src/emjupy-render.el" "src/emjupy-cells.el" "src/emjupy-kernel.el" "src/emjupy-figures.el" "src/emjupy-widgets.el" "src/emjupy-widget-page.el" "src/emjupy-figure-window.py" "src/emjupy-remote.el" "src/emjupy-lsp.el" "src/emjupy-eglot.el" "src/emjupy-mode.el" "src/emjupy-notebook.el"))
The recipe lists the sources by name rather than by wildcard, so a
development file added to src/ does not ship by accident. The
melpazoid job in CI reads this same file, so there is one list.
Before submitting:
make check # unit and integration tests make lint # checkdoc and package-lint make package # builds the tar
and check that:
- melpazoid reports nothing – its CI job runs it against the recipe above, with warnings treated as errors;
- the tar installs into an empty
package-user-dirandM-x emjupy-modeworks there; - each module loads on its own in a fresh Emacs, since that is how an accidental dependency on load order shows up.
