Replacing Jupyter with Orgmode

Background

• I dislike Jupyter notebooks (and JupyterHub) a lot
• EIN is really not much of a solution either

In the past I have written some posts on TeX with JupyterHub and discussed ways to use virtual Python with JupyterHub in a more reasonable manner.

However, I personally found that EIN was a huge pain to work with, and I mostly ended up working with the web-interface anyway.

It is a bit redundant to do so, given that at-least for my purposes, the end result was a LaTeX document. Breaking down the rest of my requirements went a bit like this:

What exports well to TeX?

Org, Markdown, anything which goes into pandoc

What displays code really well?

LaTeX, Markdown, Org

What allows easy visualization of code snippets?

Rmarkdown, RStudio, JupyterHub, Org with babel

Clearly, orgmode is the common denominator, and ergo, a perfect JupyterHub alternative.

Setup

Throughout this post I will assume the following structure:

1tree tmp
2mkdir -p tmp/images
3touch tmp/myFakeJupyter.org

As is evident, we have a folder tmp which will have all the things we need for dealing with our setup.

Virtual Python

Without waxing too eloquent on the whole reason behind doing this, since I will rant about virtual python management systems elsewhere, here I will simply describe my preferred method, which is using poetry.

1# In a folder above tmp
2poetry init
3poetry add numpy matplotlib scipy pandas

The next part is optional, but a good idea if you figure out using direnv and have configured layout_poetry as described here:

1# Same place as the poetry files
2echo "layout_poetry()" >> .envrc

Note:

• We can nest an arbitrary number of the tmp structures under a single place we define the poetry setup
• I prefer using direnv to ensure that I never forget to hook into the right environment

Orgmode

This is not an introduction to org, however in particular, there are some basic settings to keep in mind to make sure the set-up works as expected.

Indentation

Python is notoriously weird about whitespace, so we will ensure that our export process does not mangle whitespace and offend the python interpreter. We will have the following line at the top of our orgmode file:

1# -*- org-src-preserve-indentation: t; org-edit-src-content: 0; -*-

Note:

• this post is actually generating the file being discussed here by

tangling the file

• You can get the whole file here

TeX Settings

These are also basically optional, but at the very least you will need the following:

1#+author: Rohit Goswami
2#+title: Whatever
3#+subtitle: Wittier line about whatever
4#+date: \today
5#+OPTIONS: toc:nil

I actually use a lot of math using the TeX input mode in Emacs, so I like the following settings for math:

1# For math display
2#+LATEX_HEADER: \usepackage{amsfonts}
3#+LATEX_HEADER: \usepackage{unicode-math}

There are a bunch of other settings which may be used, but these are the bare minimum, more on that would be in a snippet anyway.

Note:

• rendering math in the orgmode file in this manner requires that we use XeTeX to compile the final file

Org-Python

We essentially need to ensure that:

• Babel uses our virtual python
• The same session is used for each block

We will get our poetry python pretty easily:

1which python

Now we will use this as a common header-arg passed into the property drawer to make sure we don’t need to set them in every code block.

We can use the following structure in our file:

1\* Python Stuff
2  :PROPERTIES:
3  :header-args:    :python /home/haozeke/.cache/pypoetry/virtualenvs/test-2aLV_5DQ-py3.8/bin/python :session One :results output :exports both
4  :END:
5Now we can simply work with code as we normally would
6\#+BEGIN_SRC python
7print("Hello World")
8\#+END_SRC

Note:

• For some reason, this property needs to be set on every heading (as of Feb 13 2020)
• In the actual file you will want to remove extraneous  \ symbols:
  • \* → *
  • \#+BEGIN_SRC → #+BEGIN_SRC
  • \#+END_SRC → #+END_SRC

Python Images and Orgmode

To view images in orgmode as we would in a JupyterLab notebook, we will use a slight trick.

• We will ensure that the code block returns a file object with the arguments

• The code block should end with a print statement to actually generate the file name

  So we want a code block like this:

 1#+BEGIN_SRC python :results output file :exports both
 2import matplotlib.pyplot as plt
 3from sklearn.datasets.samples_generator import make_circles
 4X, y = make_circles(100, factor=.1, noise=.1)
 5plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
 6plt.xlabel('x1')
 7plt.ylabel('x2')
 8plt.savefig('images/plotCircles.png', dpi = 300)
 9print('images/plotCircles.png') # return filename to org-mode
10#+end_src

Which would give the following when executed:

1#+RESULTS:
2[[file:images/plotCircles.png]]

Since that looks pretty ugly, this will actually look like this:

1import matplotlib.pyplot as plt
2from sklearn.datasets.samples_generator import make_circles
3X, y = make_circles(100, factor=.1, noise=.1)
4plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
5plt.xlabel('x1')
6plt.ylabel('x2')
7plt.savefig('images/plotCircles.png', dpi = 300)
8print('images/plotCircles.png') # return filename to org-mode

Bonus

A better way to simulate standard jupyter workflows is to just specify the properties once at the beginning.

1#+PROPERTY: header-args:python :python /home/haozeke/.cache/pypoetry/virtualenvs/test-2aLV_5DQ-py3.8/bin/python :session One :results output :exports both

This setup circumvents having to set the properties per sub-tree, though for very large projects, it is useful to use different processes.

Conclusions

• The last step is of course to export the file as to a TeX file and then compile that with something like latexmk -pdfxe -shell-escape file.tex

There are a million and one variations of this of course, but this is enough to get started.

The whole file is also reproduced here.