Co-authors: Gonzalo Peña-Castellanos, Eric Charles, Eric Kelly
At Quansight, we’ve been hosting a series of live streams
that feature our talented open source developers talking about the software they
contribute to and the communities around them. During a recent quirkshop our
incredible JupyterLab developers got together to discuss the upcoming major
version changes to JupyterLab. We discussed the toil that
core developers are investing to improve the experience of JupyterLab not only
for developers but for users as well.
See the video of this quirkshop,
"jupyterlab extensions quirkshop - migrating versions".
The extension system is a central concept to JupyterLab; all of the features
(e.g., notebook editor, file browser, menus, and status bar elements) are
extensions that interface into the greater JupyterLab ecosystem. The core
extensions that ship with JupyterLab that establish the baseline JupyterLab
interface. Third-party community extensions can then be installed to enhance and
augment the user experience. This model allows for development to happen inside
the tool itself, and co-development to happen by the surrounding ecosystem.
Very soon, JupyterLab will be achieving a major milestone by upgrading from
version 2 to 3. The next few sections will highlight the impacts of this upgrade
from individual and organizational perspectives.
JupyterLab Extensions Ecosystem
JupyterLab version 2 was a landmark in the interactive computing development
phase. During its time, the extensions ecosystem flourished. There are now
extensions for interacting with big data (e.g., Dask,
HoloViz), collaborating with others (e.g.,
jupyter-videochat), and
customizing styling. These connections are possible only
because JupyterLab connected modern, performant JavaScript with scientific
computing interfaces that improved users’ abilities to communicate and
collaborate. The development during this phase proved the extensibility of
JupyterLab and its ability to serve as a foundation for creating custom
interactive computing interfaces; this foundation allows it to fit the needs of
different users, disciplines, and workflows.
Version 3 upgrades will save your and your organization’s time and improve the
quality of collaborative work.
User Impacts of Version 3
As long-time Jupyter super users, the new changes to distributing JupyterLab
extensions are particularly exciting. Currently, JupyterLab users are required
to have a Node.js run-time in their environment in order to build and
install JupyterLab extensions (written in TypeScript/JavaScript and providing
different assets). This had to be performed every time a new extension was
installed! In JupyterLab 3, an extension developer will package the JavaScript
or CSS and ship that through PyPI with the prebuilt codes. This improvement
applies broadly to the JupyterLab ecosystem because all of the components are
extensions!
In JupyterLab 3, users will experience reduced installation times, and Python
users will rely on a familiar installation pattern by using pip.
At some point, the Jupyter Python community got very wrapped up in JavaScript
development. They said the browser was the future! This meant that already
complicated installation and packaging issues became more complicated. A common
problem in JupyterLab versions 1 and 2 was handling extensions. Users would have
to install these packages from combinations of npm and PyPI
packages. Fortunately, we have conda to help with some of these
concerns; but users still need Node.js and Python
environments to reap the benefits.
In JupyterLab 3, all of the packaging will be orchestrated by using pip, relying
on PyPI alone. So, how did this come to be? Well, this could not have
been achieved without advancements in webpack version 5, the
JavaScript library in charge of “bundling JavaScript files for usage in a
browser” and the new module federation functionality. Using
module federation with webpack 5 allows for a
JavaScript application to ship and load JavaScript modules on demand without
having to pack it in the initial application bundle (a different webpack build).
This means we can now dynamically load extensions without the need for a build.
This also means the possibility of using pip and PyPI packages to bundle the
assets and place them in the right place for JupyterLab to load extensions.
Since server-side extensions already depend on using Python, it was only natural
to extend the use to also bundle the frontend extension assets.
Python users, those typically using IPython or xeus-python kernels, will
only need a Python run-time to install packages!
But What About My npm Packages?
So, if you are an extension developer and already published npm packages, do you
need to change things? Well, it depends. The previous method for installing npm
packages will still work. And if you provide an extension that can be extended
then you still need to provide an npm package.
If your extension does not provide extension points, the new JupyterLab version
provides additional benefits, including faster installs and a single
distribution point, namely as PyPI packages for users. Managing the publication
of both npm and PyPI packages can be achieved with ease using Github Actions and
Continuous Integration, but more on that in another post!
The fact that now JupyterLab can load different webpack applications (JupyterLab
Extensions), also opens the door for serving (frontend) extensions on Content
Delivery Networks (CDNs) or your own private servers, so that installing an
extension would effectively become as easy as enabling or disabling the fetching
of assets from a given CDN!
If you are using conda, then the extension installation process will also be
simplified, as pip packages can be turned into conda packages with ease.
A Reason to Learn TypeScript
We’re primarily Python programmers who are also comfortable with distributing
packages. We get excited about designing experiences in JupyterLab, but versions
1 and 2 required us to understand a new build system, npm, and Node.js. That
became a chore and sapped motivation. Because of the changes in JupyterLab 3
we’re excited and happy to learn the necessary TypeScript and continue
generating more user-facing applications. We can now use our more familiar PyPI
workflows to distribute the finished work.
Faster, More Compact Binders
MyBinder is a critical service to the Jupyter community. It provides
on-demand, interactive Jupyter experiences from GitHub repositories. There are a
number of ways to define environments using Python, R, conda, or Docker by using
different configuration files. Conda has been, for some time, the most reliable
system for installing JupyterLab extensions on MyBinder. But extensions have
required a postbuild step to include extensions in a Binder,
thereby increasing build and deploy times.
The version 3 changes in packaging now mean that a lot of deployment complexity
is reduced. The fastest way to build and deploy a Binder is to use pip rather
than conda. In JupyterLab version 3, it will be possible to install a novel
JupyterLab experience on Binder purely through pip (if all of the packages are
available on the Python packaging index or on available version-control system
(VCS) installations). It will reduce Binder configuration complexity and
accelerate testing of Binder experiences; less time between Binder deploys
allows architects, users, and developers to iterate quicker.
Time Matters
Ultimately, the transition from JupyterLab 2 to 3 will improve organizational
efficiencies. These upgrades are going to drastically reduce installation times
and improve access for scientific Python developers. Users should find
themselves being more productive with less downtime and improved interactive
computing tools. In an organization that uses JupyterLab broadly, the
compounding effects of the time savings will ultimately result in better
communication through data and code.
In this post we wanted to highlight one of the features that in our opinion will
drastically improve the user experience. However, there are other features
coming with JupyterLab 3, including:
- Internationalization
- Debugging
- Single document mode, aka classic notebook mode
- Table of contents for notebooks
- File browser filtering
And many more! Please check the CHANGELOG and stay tuned for more information.
JupyterLab users will benefit greatly from the blood, sweat, and tears of the
development team. Those of us at Quansight would really like
to thank the team for all their hard work and thoughtfulness.
Want more Jupyter? Read about Quansight's new project, QHub, in our
announcement blog post.