emjupy is a multi-file Emacs package, which was developed with use of Claude Opus 5.5 (and other LLMs). To build a tar and install it:

bash
make package                 # generates emjupy-i.j.k.tar

and install from Emacs: M-x package-install-file RET emjupy-i.j.k.tar RET with i.j.k the current version.

The only external dependency is websocket (1.15+); Emacs 30.1+ is required. Installed as a package, the commands you start from – emjupy-login, emjupy-open-notebook, emjupy-server-dashboard – are autoloaded, so they work without any require.

Which Emacs to develop against

Development targets Emacs 30.1, which is also the minimum the package declares: what CI runs, and what most users are on. 29 was supported until Eglot's differences there became a source of bugs – one sent every language server request to a local process, answering about the wrong machine's files, and stayed invisible for weeks because 29 was where it was being tested.

make versions says which Emacs is in use and whether it is the target.

Source layout

The sources are in src/, listed here from the bottom layer up. Each file requires only files earlier in this table, never later ones.

File Contains
emjupy-core.el structs, state, server registry, lookups
emjupy-http.el REST transport, URL parsing, the HTTP error types
emjupy-render.el faces, colours, outlines, drawing output, text helpers
emjupy-cells.el buffer<->struct syncing, cell editing and cell commands
emjupy-kernel.el kernel WebSocket, execution, message routing
emjupy-figures.el opening an output on its own: plotly figures, HTML, images
emjupy-widgets.el widget controls in the buffer, and what they cause
emjupy-widget-page.el widgets that draw themselves, live in a page, through Emacs
emjupy-remote.el where a server's files live: roots, SSH tunnels, TRAMP
emjupy-lsp.el language-server WebSocket transport, shadow document text
emjupy-eglot.el shadow buffer, Eglot integration, completion, eldoc
emjupy-mode.el emjupy-mode, its keymap and menu
emjupy-notebook.el login, the notebook browser, .ipynb parse and serialize
emjupy.el package header, version, and the entry point that loads it
emjupy-pkg.el package metadata for make package (MELPA makes its own)

Dependencies run one way

No file calls a function defined in a file later in the table, and there are no declare-function forms for emjupy's own functions. Every module can be loaded on its own, which is what makes that checkable: (require 'emjupy-lsp) in a fresh Emacs must work without anything else having been loaded first.

When a lower layer needs to tell a higher one that something happened, it runs a hook rather than naming the function. There are these:

  • emjupy-kernel-connected-functions, run by the kernel layer when a kernel attaches. emjupy-mode hangs two handlers on it: one asks the kernel where it runs, for the language server; the other tells the kernel the cell width.
  • emjupy-box-width-changed-functions, run by the drawing layer when the width the cells are drawn at changes, so the kernel is told the new width.
  • emjupy-comm-functions, run by the kernel layer with each comm message, and emjupy-output-owner-functions, asked by it which cell output answering a request no cell waits on belongs to. The widgets layer keeps its models from the first, and claims with the second the output a moved slider causes. It also sets emjupy-widget-view-function, which the drawing layer calls to draw a widget.

Both are wired when a notebook buffer starts, not when a file is loaded – a package should not add hooks just by being loaded.

A keymap may name a command defined further down the table, with a plain quote rather than #': a binding needs the name, and resolves it when the key is pressed.

Errors

Catch the failure you expect, and let everything else through. There is no ignore-errors in the package: it hides the failure that was expected and every bug alongside it, and a function that has silently done nothing for months looks exactly like one that works.

  • An expected failure that has a precondition is checked, not caught – window-live-p, buffer-live-p, websocket-p and websocket-openp, jsonrpc-running-p, xref-file-location-p.
  • One that can only be found out by trying is caught by its type: json-error – the whole family: a reply nested too deep, or bytes that are not UTF-8, is a json-error but not a json-parse-error – jsonrpc-error, file-error.
  • A request to the Jupyter server that fails signals emjupy-http-error, or emjupy-http-status when the server answered with an error status. Both are children of file-error, the family Emacs uses for a resource it cannot reach, which also keeps their messages readable. emjupy--http-status-of extracts the status; emjupy--http-exists-p answers "is this there" with nil for a 404 instead of signalling.
  • A path into the Contents API goes through emjupy--contents-path, never a bare concat: a notebook name can hold # or ?, which in a URL start a fragment or a query, so a#b.ipynb was asked for as a – and saved there.
  • Background work that must not abort the command it runs under – the language server warming up while a notebook opens – uses with-demoted-errors, so a failure is reported rather than lost.

Makefile targets

Target Effect
make compile byte-compile the sources, with src/ first on the load path
make test the unit tests; integration tests skip without a server
make check unit and integration tests; refuses to run without a server
make lint checkdoc, and package-lint if it is installed
make docs export docs/*.org to the site in html-content/
make package build emjupy-VERSION.tar
make install install that tar via package-install-file
make coverage line coverage of the suite, as lcov, with a per-file table
make set-version write a new version everywhere: make set-version VERSION=X.Y.Z
make check-version the version header, emjupy-version and the pkg file agree
make versions which Emacs this is, against the one development targets
make clean remove .elc files and the tar
make timestamps touch every source, to force a full rebuild

Continuous integration

Each check runs once per event, in one workflow unless only another can do it:

Event Runs
push to main, pull request build-package.yml: compile, lint, melpazoid, docs, and the tests – unit and integration, once
push to main changing src/ and coverage.yml, for the Codecov badge
tag release.yml only
by hand set-version.yml, or any of the above from the Actions tab

Pushes to other branches run nothing until a pull request is opened, and a newer push to the same branch cancels a run still going. Build also runs for changes to the documentation, the tools, the notebook corpus and the MELPA recipe, not only the Lisp.

The release tests what it ships, and is the only run a tag starts: the version commit is often pushed with [skip ci], so nothing else has tested it. Path filters do not apply to tag pushes, which is why a tag used to start Build and Coverage as well – the suite four times over.

Running everything locally

What CI runs, in the order worth running it: the cheap checks first, so a typo does not cost a server start-up.

Once: what the checks need

  • Emacs 30.1 – make versions says whether you have it.
  • websocket.el, the one dependency. An installed copy is found; a checkout can be named instead with export EMJUPY_WEBSOCKET_DIR=/path/to/emacs-websocket.
  • For the integration tests, a Python environment with the server, a kernel and a language server:

    bash
    pip install jupyter-server ipykernel jupyter-lsp 'python-lsp-server[all]'
    
  • For make lint to run package-lint as well as checkdoc, M-x package-install RET package-lint. Without it the target says so and checks the rest.
  • For melpazoid, a clone and Docker – melpazoid runs inside a container, which is what makes its result match MELPA's:

    bash
    git clone https://github.com/riscy/melpazoid ~/melpazoid
    

Every time

bash
make compile                # no warnings expected
make test                   # unit tests, about ten seconds
make lint                   # checkdoc and package-lint: nothing expected
make check-version          # three places agree on the version

Then the integration tests, against a server of your own:

bash
mkdir -p /tmp/nbroot
jupyter server --no-browser --port=8888 --ip=127.0.0.1 \
  --IdentityProvider.token=abc --ServerApp.root_dir=/tmp/nbroot &

EMJUPY_TEST_URL=127.0.0.1:8888 EMJUPY_TEST_TOKEN=abc \
EMJUPY_TEST_ROOT=/tmp/nbroot make check

EMJUPY_SKIP_UNSTABLE=1 leaves out the tests tagged :unstable, as CI does; without it they run, and a failure there is real but known.

Then melpazoid, reading the same recipe MELPA will be sent:

bash
RECIPE="$(cat .github/melpa-recipe)" LOCAL_REPO="$PWD" make -C ~/melpazoid

Coverage, when you want to see what the tests leave untouched. It needs undercover.el and its two dependencies, installed or checked out:

bash
git clone https://github.com/undercover-el/undercover.el ~/src/undercover
git clone https://github.com/magnars/dash.el ~/src/dash
git clone https://github.com/rejeep/shut-up.el ~/src/shut-up
export EMJUPY_UNDERCOVER_PATH=~/src/undercover:~/src/dash:~/src/shut-up

make coverage               # prints a per-file table; lcov in coverage/

With the integration variables set it covers the server-facing code as well, which is how CI runs it. Tests tagged :timing are left out: they assert on elapsed time, and every function is several times slower when instrumented.

And the documentation, with the link check CI runs – every page and every anchor a link names must exist:

bash
make docs                   # then open html-content/index.html
python3 tools/check-links.py

Both the build and the release run that one script; a link to a section of another page is checked to its anchor, so link to a heading by a CUSTOM_ID – Org cannot resolve a heading searched for in another file.

Compiling one file by hand

M-x byte-compile-file compiles against whatever load-path holds. With an emjupy installed from a package archive, (require 'emjupy-core) finds the INSTALLED copy, and a module added since then is not found at all – "Cannot open load file: emjupy-remote" is that, not a fault in the file. make compile puts src/ first and avoids it; by hand, add the checkout to the front of load-path, or uninstall the package while working on it.

emjupy-run-tests.el, lint.el, emjupy-pkg.el and publish.el are not compiled at all – each says no-byte-compile – because they are scripts and metadata rather than code to load. publish.el matters most: package-vc compiles everything in a checkout, so without it an install from git would warn about the site's build script.

Unit tests

The self-contained suite requires only websocket:

bash
make test
# which is
emacs -batch -Q -L src -l src/emjupy-run-tests.el

websocket is resolved from load-path first, then from EMJUPY_WEBSOCKET_DIR, and only then from GNU ELPA – so the suite also runs offline / in CI:

bash
EMJUPY_WEBSOCKET_DIR=/path/to/emacs-websocket make test

Integration tests (real server, real kernel, real LSP)

emjupy-integration-test.el talks to a live Jupyter server and runs real Python in a real kernel. These are opt-in and ert-skip unless the environment points at a running server, so the default run stays fast and hermetic.

Some things only break against a live server – an image arriving that this Emacs cannot display, a WebSocket URL that drops its base path, a saved notebook no other tool will open – so they are worth running before trusting a change.

Variable Meaning
EMJUPY_TEST_URL host:port of a running server (the LOCAL end of a tunnel)
EMJUPY_TEST_TOKEN its token
EMJUPY_TEST_ROOT server root dir, for the nbformat-validation test
EMJUPY_TEST_URL2 a SECOND server, for the multi-server tests
EMJUPY_TEST_TOKEN2 its token (defaults to EMJUPY_TEST_TOKEN)

To exercise the remote path, point EMJUPY_TEST_URL at the local end of an ssh tunnel rather than at the server directly:

bash
# server side: jupyter listens only on loopback
jupyter server --no-browser --port=8888 --ip=127.0.0.1 --IdentityProvider.token=abc

# client side: forward a local port through ssh
ssh -N -L 18888:127.0.0.1:8888 user@remote-host &

EMJUPY_TEST_URL=127.0.0.1:18888 EMJUPY_TEST_TOKEN=abc \
EMJUPY_TEST_ROOT=/path/to/notebook/root make check

The language-server tests come in two kinds. Those of the WebSocket transport need jupyter-lsp and python-lsp-server in the environment running the server – the pip line above installs both. Those of the local fallback need a language server on this machine's PATH (pylsp, pyright-langserver or jedi-language-server), and skip if none is found.

Robustness tests

src/emjupy-robustness-test.el holds the tests that make their own inputs. They run with the rest of make test, and each found a bug the hand-written tests had not.

Test What it tries
message order random interleavings of a kernel's replies and outputs
random edits random sequences of cell commands, invariants after each
undo everything random edits, then undo to the start
round-trip every notebook in src/test-corpus/, saved and reloaded
hostile content escape floods, huge SVGs, deep JSON, awkward file names
leaks many sessions; timers, processes, hooks, tables after

Seeds

Each run draws a seed, and every failure of a randomised test names it:

Info: EMJUPY_FUZZ_SEED=4, case 5
Info: after (insert-above insert-below delete-char split move-down)

Set it to replay the run exactly; EMJUPY_FUZZ_ITERATIONS sets how many cases each test tries – by default from ten, for the slow ones, to two hundred, for the message orders:

bash
EMJUPY_FUZZ_SEED=4 EMJUPY_FUZZ_ITERATIONS=500 make test

A failing sequence is usually long. Before working on one, shrink it: replay it with a step removed, and keep the removal while it still fails. The separator bug the edit fuzz found was some twenty steps as found and five once shrunk, and understood from those, three: insert a cell, insert another before it, backspace.

Expected failures

A test for a fault that is known but not yet fixed is marked :expected-result :failed, with the reason in its docstring. The suite stays green, the fault stays visible, and a fix shows up as an unexpected pass – the cue to take the marking off.

The notebook corpus

src/test-corpus/ is written by tools/make-notebook-corpus.py, with nbformat itself, so each notebook is valid by Jupyter's own definition. tools/check-roundtrip.py compares a notebook with emjupy's save of it – every cell, source, output, metadata and attachment – and validates the save; the round-trip test calls it when Python has nbformat, and otherwise checks only that saving twice changes nothing. Add a notebook to the generator, not by hand, when a new shape turns up.

Network faults

tools/faultproxy.py sits between emjupy and the server and misbehaves on command: it drops every connection, stalls, delays or splits what it forwards. The emjupy-int-fault-* tests start it in front of the test server, so they run with the integration tests and need python3. It can be driven by hand as well:

bash
python3 tools/faultproxy.py 127.0.0.1 8888     # prints its port and control port
echo stall | nc 127.0.0.1 CONTROL-PORT         # or sever, resume, delay 200, chunk 7, reset

Compiled files

Emacs loads a .elc in preference to its .el, newer or not. To check a test fails without a fix, run make clean first – otherwise the old code is the compiled copy of the new, and the test passes for no reason.

Test hygiene

A unit test must give the same answer on every machine. Two things in emjupy make that easy to lose: the host of an SSH tunnel is found by reading the process list, and whether LaTeX can be rendered is decided by looking for the programs that render it. Both are correct in use and wrong in a test.

emjupy-probe-environment governs both, and the unit test runner binds it to nil. A test that needs the detection tested – as opposed to merely used – binds it back to t and stubs what it examines.

This is not hypothetical: two tests asking what a tunnelled server resolves to passed here and failed on a machine that happened to be forwarding that port, and a LaTeX test depended on whether LaTeX was installed.

Other rules worth following, each learned from a test that passed while the thing it covered was broken:

  • Test the key binding, not only the function behind it. A command bound to DEL alone is not reached in a graphical Emacs that sends <backspace>.
  • Do not wrap the action under test in ignore-errors. A test that guards against remote connections passed because the connection it should have caught raised an error which was then swallowed.
  • When a stub replaces a function used by Emacs itself, call through for the cases the test is not about.
  • Prefer asserting an invariant over asserting a mechanism; mechanisms differ between Emacs versions, invariants do not.

Invariants

M-x emjupy-debug-check-invariants reports how a buffer and its cells disagree: which cell, and whether its overlay is missing, overlapping, out of order, detached from its output box, or showing text the cell does not hold. The last means what is displayed and what would be saved have come apart.

Tests call it after every operation. It has caught overlays left overlapping after a split, and a sync that turned an empty cell into one containing a newline.

Performance

Three tests guard against regressions rather than measuring speed:

  • work done after every command does not grow with the notebook;
  • redrawing output does not grow with the notebook, since only the cell concerned is redrawn;
  • nothing on a path that runs after a command touches a remote file.

The last matters most. A remote operation costs a round trip, TRAMP blocks Emacs while it waits, and anything on the per-command path pays that on every keystroke. The test asserts the absence of such calls rather than a duration, since a duration would depend on a network.

The per-command costs these keep small are measured in Language server support.

Releasing

  1. Set the version. From GitHub: Actions, "Set version", "Run workflow", and give it the new version as X.Y.Z. It writes the version everywhere the current one appears – the ;; Version: header, emjupy-version, src/emjupy-pkg.el and the sample output in docs/usage.org – checks they agree, and commits the change. Or locally, the same script: make set-version VERSION=X.Y.Z.
  1. Tag the commit with the same version and push the tag. That starts the release: the tests, the tarball on the release page, and the documentation site, all from that commit. The workflow does not tag for you: a tag pushed from a workflow does not start other workflows, so the release would not run.

    A commit whose message contains [skip ci] is skipped for tag pushes too. Tag a commit without it, or the release never starts.

The package's own description

C-h P emjupy shows the ;;; Commentary: section of src/emjupy.el: what emjupy is, what it needs, and how to start. It is written for someone who has only just installed the package and has not seen the repository.

package.el prefers an installed package's README to its Commentary, and the README is a web page – badges, screenshots, links into the repository – so the tarball make package builds carries no README. MELPA leaves it out for the same reason, so both describe themselves alike. An install from git with package-vc is the whole checkout, and shows the README.

The opening paragraph is the same in three places – the Commentary, the README and docs/index.org – and is worth keeping so when one changes. The one-line summary is the same in the header of src/emjupy.el and in src/emjupy-pkg.el.

Submitting to MELPA

The recipe must list the files explicitly. A recipe without :files takes every .el in the repository, which here would ship the tests to users and include emjupy-run-tests.el, a script with no provide form that package-lint rejects.

The recipe used is kept at .github/melpa-recipe:

elisp
(emjupy :fetcher github
        :repo "mathren/emjupy"
        :files ("src/emjupy.el"
                "src/emjupy-core.el"
                "src/emjupy-http.el"
                "src/emjupy-render.el"
                "src/emjupy-cells.el"
                "src/emjupy-kernel.el"
                "src/emjupy-figures.el"
                "src/emjupy-widgets.el"
                "src/emjupy-widget-page.el"
                "src/emjupy-figure-window.py"
                "src/emjupy-remote.el"
                "src/emjupy-lsp.el"
                "src/emjupy-eglot.el"
                "src/emjupy-mode.el"
                "src/emjupy-notebook.el"))

The recipe lists the sources by name rather than by wildcard, so a development file added to src/ does not ship by accident. The melpazoid job in CI reads this same file, so there is one list.

Before submitting:

bash
make check                 # unit and integration tests
make lint                  # checkdoc and package-lint
make package               # builds the tar

and check that:

  • melpazoid reports nothing – its CI job runs it against the recipe above, with warnings treated as errors;
  • the tar installs into an empty package-user-dir and M-x emjupy-mode works there;
  • each module loads on its own in a fresh Emacs, since that is how an accidental dependency on load order shows up.