Data scientists use different Jupyter notebooks every day — ranging from disposable ones for quick tasks to those shareable with clients. Over time, more and more notebooks accumulate, making it increasingly difficult to reuse them in whole or in part. To mitigate this problem and make the most relevant pieces of code quickly accessible to every data scientist, we developed JupyterLab Snippets at Feedzai — our take on leveraging code snippets directly on JupyterLab.
JupyterLab is a computational notebook platform that enables us to carry on data science work (and beyond) via notebooks. These notebooks, where code, text, and images come together, allow us to complete all kinds of tasks, keeping each input close to each output. At Feedzai, data scientists have access to different JupyterLab environments packed with custom notebooks and Python packages. Here they go from analyzing data to training models, from preparing reports to debugging the system — a “lab of all trades”, we could say.
Given the importance of JupyterLab and notebooks in the daily work of our data scientists, we started a research project that culminated in JupyterLab Snippets. First, we collected statistics from internal notebook repositories across teams and conducted user interviews with junior and senior data scientists. We needed to know more about how data scientists actually use JupyterLab and notebooks, and what their ideas are for a better platform.
After compiling the insights, several aspects became clear:
- JupyterLab and notebooks are heavily used for all sorts of data science and ad-hoc tasks (typically one per notebook). There is no single type of task (e.g., exploratory data analysis) that is more commonly addressed with notebooks than others.
- While file structures may look similar at first glance (each environment has at least a model training notebook, for example), data scientists have different and personal workflows for coding notebooks.
- Duplicating past notebooks available in the environment or getting them from repositories is a very common practice.
- Notebooks composed of code snippets is also an identified practice. These notebooks are not intended to be run, and contain code and text for future reference (imagine a notepad in notebook format). This case and the previous one imply removal of unnecessary parts and parameterization.
- Data scientists regularly communicate within teams and individually. Sharing notebooks/snippets is a common practice.
- No significant issues were identified with the JupyterLab/notebook interface.
It is true that data scientists have their own ways of finding whatever they need for their next notebook, and AI coding assistants are proliferating. Even so, would it be beneficial to improve the experience of reusing code in JupyterLab? After analyzing all the insights, we believe so. We believe in an interface that allows every data scientist to easily manage and use “their snippets” now and in one year, while also facilitating their sharing with other data scientists. JupyterLab Snippets aims to improve the developer experience, focusing on productivity and satisfaction, while having a low maintenance cost. Here is JupyterLab Snippets in its essence.
Introducing JupyterLab Snippets
JupyterLab Snippets is a JupyterLab extension for data scientists to create, manage, and use code snippets directly on JupyterLab, facilitating code reuse in their notebooks. To be more precise, the interface is a JupyterLab extension and the backend, for managing the snippet data, is a Jupyter Server one. Both communicate via a REST API.
The interface, a part of JupyterLab like notebooks and the like, is divided into three main components:
- Snippets Sidebar: The central hub for JupyterLab Snippets, available as a new tab in the right sidebar of JupyterLab. It allows the data scientist to manage and utilize the code snippets. Each snippet is available on a Snippet Card.
- Snippet Editor: The form-like interface for creating new snippets and editing existing ones. When opened, it is available as a tab in the main work area of JupyterLab.
- Shortcuts: Options for creating snippets from notebook cells. These options are available from the context menu after right-clicking on a cell, and the option to start a snippet from a cell is also available from the cell toolbar.
In the Snippets Sidebar, in addition to the Personal collection, data scientists will find a set of pre-made snippets: the Feedzai collection. They can switch between both (or view them all at once) by selecting the desired collection in the toolbar just above the list of available snippets. The Feedzai collection is a set of snippets for common actions performed in notebooks, prepared with the help of our data scientists. This collection is specially designed for newcomers who can take advantage of JupyterLab Snippets from day one. Additionally, it is available in a dedicated repository so that anyone can easily contribute to this collection and enrich it for everyone.
From the Snippets Sidebar, data scientists can export all of their snippets in JSON format (each snippet is exportable individually as well). This file makes it easy to share snippets among data scientists, as well as maintain them in repositories for reference. Each data scientist can then import one of these files into their environment, from the dropdown menu shown in the first screenshot, and have access to new snippets.
Usage
JupyterLab Snippets is designed to be a flexible tool that data scientists can integrate into their workflows in different ways and use as much or as little as they need. There is more than one way to create snippets, and each data scientist may have as few as five or as many as 50 snippets (or even more) at their disposal. Let’s look at a possible simple workflow:
- After launching JupyterLab, the data scientist starts by opening the notebook they are working on.
- Instead of copying and pasting code from a given source to the notebook directly, they open the Snippets Sidebar on the right side of JupyterLab and then the Snippet Editor to create a new snippet.
- Since the data scientist can imagine reusing this piece of code in the future, they create a new snippet with it, including a name, a brief description, and two tags that describe it appropriately.
- Now they insert the new snippet into the notebook and continue working from there.
- Finishing the current notebook, the data scientist clears all the output. Then, when they review the notebook and check that everything is okay, they notice a cell that would also make a useful snippet for the future.
- Since not all the cell content is relevant for a reusable snippet, the data scientist selects only what they want and starts a new snippet from there via the cell context menu.
- Similar to the first snippet, they assign a name and a tag.
- After a while, in a new notebook, the data scientist opens the Snippets Sidebar and quickly inserts both previously created snippets, as they will be useful in this new notebook as well.
What’s Next
JupyterLab Snippets is now available for data scientists to use across Feedzai. Our current focus is on helping data scientists incorporate JupyterLab Snippets into their daily workflow, addressing any issues that arise, and collecting feedback for future improvements.
If you have any questions or feedback, let us know in the comments section.
Beatriz Feliciano, Diogo Duarte, Ferran Pla Fernández, Iker Perez Lopez, João Bernardo Narciso, Rita Costa, and Sanjay Salomon: thank you so much for your thoughtful feedback that helped me shape this blog post!
Here and Now: Reusing Code at Feedzai with JupyterLab Snippets was originally published in Feedzai Techblog on Medium, where people are continuing the conversation by highlighting and responding to this story.
