23  Distributing and Sharing Julia Code

Authors

Alec Loudenback

MoJuWo Contributors

23.1 Chapter Overview

Applying software engineering best practices (Chapter 12) in Julia, including testing, documentation, and coverage metrics. Collaborating on code. Publishing packages for others to use.

23.2 Setup

A vast majority of Julia packages are hosted on GitHub (although less common, other options like GitLab are also possible). GitHub is a platform for collaborative software development, based on the version control system Git (see Chapter 12 for an introduction).

The first step is therefore creating an empty GitHub repository on GitHub (don’t add a README License, etc. at this step).

Tip

You should try to follow package naming guidelines and add a “.jl” extension at the end, like so: “MyAwesomePackage.jl”.

Locally, use PkgTemplates.jl (see Section 21.10.1) to then create the package’s folder locally on your computer, which will create a package with several subfolders (these will be described as the chapter progresses).

To sync this up with the newly created GitHub repository, you git push this new folder to the remote repository https://github.com/myuser/MyAwesomePackage.jl. GitHub should show you how to do this on the associated repository page (something like this):

# terminal commands from inside your new package directory:
git init
git add .
git commit -m "Initial commit"
git branch -M main 
git remote add origin https://github.com/myuser/MyAwesomePackage.jl.git
git push -u origin main

23.3 GitHub Actions

The most useful aspect of PkgTemplates.jl is that it automatically generates workflows for GitHub Actions. These are stored as YAML files in .github/workflows, with a slightly convoluted syntax that you don’t need to fully understand. For instance, the file CI.yml contains instructions that execute the tests of your package (see below) for each pull request, tag or push to the main branch. This is done on a GitHub server and should theoretically cost you money, but if your GitHub repository is public, you get an unlimited workflow budget for free.

A variety of workflows and functionalities are available through optional plugins. The interactive setting Template(..., interactive=true) allows you to select the ones you want for a given package. Otherwise, you will get the default selection, which you are encouraged to look at.

23.4 Testing

The purpose of the test subfolder in your package is unit testing: automatically checking that your code behaves the way you want it to. For instance, if you write your own square root function, you may want to test that it gives the correct results for positive numbers, and errors for negative numbers.

using Test

@test sqrt(4) ≈ 2

@testset "Invalid inputs" begin
    @test_throws DomainError sqrt(-1)
    @test_throws MethodError sqrt("abc")
end;

Such tests belong in test/runtests.jl, and they are executed with the ]test command (in the REPL’s Pkg mode). Unit testing may seem rather naive, or even superfluous, but as your code grows more complex, it becomes easier to break something without noticing. Testing each part separately will increase the reliability of the software you write.

Tip

To test the arguments provided to the functions within your code (for instance their sign or value), avoid @assert (which can be deactivated) and use ArgCheck.jl instead.

That is, avoid this:

function mysqrt(x)
    @assert x >= 0
    ...

And do this instead:

using ArgCheck
function mysqrt(x)
    @argcheck x >= 0 DomainError
    ...
end

At some point, your package may require test-specific dependencies. In essence, you give the test subfolder its own environment and Project.toml file. This often happens when you need to test compatibility with another package, on which you do not depend for the source code itself. Or it may simply be due to testing-specific packages like the ones we will encounter below. For interactive testing work, use TestEnv.jl to activate the full test environment (faster than running ]test repeatedly).

Tip

The Julia extension also offers a more advanced own testing framework, which relies on defining “test items” the code. The benefit of this is that the tests will integrate more directly with the VS Code interface and specific subgroups of tests can be run independently, on-demand. See TestItemRunner.jl for more.

Tip

If you want to have more control over your tests, you can try

23.4.1 Code Coverage

Code coverage refers to the fraction of lines in your source code that are covered by tests (described in more detail in Section 12.3.2). Codecov is a website that provides easy visualization of this coverage, and many Julia packages use it. It is available as a PkgTemplates.jl plugin. For public GitHub repositories using GitHub Actions, uploads are typically token-less. For private repositories or other CI providers, you’ll need to add the CODECOV_TOKEN secret as documented by Codecov.

23.5 Code Style

To make your code easy to read, it is recommended to follow a consistent set of guidelines. The official style guide is very short, so most people use third party style guides like BlueStyle or SciMLStyle.

23.5.1 Formatters

23.5.1.1 JuliaFormatter.jl

JuliaFormatter.jl is an automated formatter for Julia files which can help you enforce the style guide of your choice. Just add a file .JuliaFormatter.toml at the root of your repository, containing a single line like

style = "blue"

Then, the package directory will be formatted in the BlueStyle whenever you call

import JuliaFormatter
# run from the package root, and
# formats according to .JuliaFormatter.toml
JuliaFormatter.format(".")
Note

The default formatter uses JuliaFormatter.jl.

Tip

You can format code automatically in GitHub pull requests with the julia-format action, or add the formatting check directly to your test suite.

23.5.1.2 Runic.jl

Runic.jl is a popular choice as well. Like Python’s popular Black formatter, there is no configuration to the formatting options when using Runic. The benefit is increased consistency and no time wasted debating formatting decisions.

Runic, requires you to set up Runic using Pkg. The instructions are straightforward and available on the Runic.jl repository.

23.6 Code quality

Of course, there is more to code quality than just formatting. Aqua.jl (Auto QUality Assurance) provides a set of routines that examine other aspects of your package, from ensuring that there are no unused dependencies to catching ambiguous methods statically.

Include the following in your tests to have Aqua.jl run various checks each time your tests run:

using Aqua, MyAwesomePackage
Aqua.test_all(MyAwesomePackage)

JET.jl is tool that is similar to a static linter in other languages. This means that it can inspect your code and ‘understand’ it well enough to catch many types of errors before runtime. It does this by running type inference and figuring out how a given type will flow through the call stack of methods.

You can either use it in report mode (with a nice VSCode display) or in test mode as follows:

using JET, MyAwesomePackage
JET.report_package(MyAwesomePackage)
JET.test_package(MyAwesomePackage)

Note that both Aqua.jl and JET.jl might pick up false positives: refer to their respective documentations for ways to make them less sensitive.

Finally, ExplicitImports.jl can help you get rid of generic imports to specify where each of the variables in your package comes from. As a project gets more complex, using SomePackage can bring many, sometimes conflicting symbols into your current namespace. ExplicitImports forces you to either qualify the usage (e.g. SomePackage.somefunction(...)) or explicitly opt into importing certain variables.

23.7 Documentation

Refer to Section 12.4.2 for more detail on documentation and its importance. Here are some additional workflow tips for setting up documentation for your package.

DocStringExtensions.jl provides a few shortcuts that can speed up docstring creation by taking care of the obvious parts.

In addition to docstrings, Documenter.jl allows you to design a website for all of this, based on Markdown files contained in the docs subfolder of your package. Unsurprisingly, its own documentation is excellent and will teach you a lot. To build the documentation locally, just run

julia> using Pkg

julia> Pkg.activate("docs")

julia> include("docs/make.jl")

Then, use LiveServer.jl from your package folder to visualize and automatically update the website as the code changes (similar to Revise.jl, but for your docpages instead of your code):

julia> using LiveServer

julia> servedocs()

To host the documentation online easily, just select the Documenter plugin from PkgTemplates.jl during creation. Not only will this fill the docs subfolder with the appropriate starting files: it will also initialize a GitHub Actions workflow to build and deploy your website on GitHub pages. Lastly, in your repository’s Pages settings, configure deployment from the gh-pages branch (root).

23.8 Literate programming

Literate programming is so-called for combining written documents with the output of programs (literature + code = literate programming). These tools allow you to interleave code with texts, formulas, images and so on.

In addition to the Pluto.jl and Jupyter notebooks, take a look at Literate.jl to enrich your code with comments and translate it to various formats. Books.jl is relevant to draft long documents in a pure Julia way.

Quarto is an open-source scientific and technical publishing system that supports Python, R and Julia. Quarto can render markdown files (.md), Quarto markdown files (.qmd), and Jupyter Notebooks (.ipynb) into documents (Word, PDF, presentations), web pages, blog posts, books, and more. Additionally, Quarto makes it easy to share or publish rendered content to various online hosts.

PPTX.jl will create Microsoft PowerPoint files.

23.9 Versions and registration

23.9.1 Versions and Compatibility

The Julia community has adopted semantic versioning, which means every package must have a version, and the version numbering follows strict rules (the concept of versioning was covered in Section 12.6.2).

To comply with the versioning requirements in Pkg’s resolver, you need to specify compatibility bounds for your dependencies: this happens in the [compat] section of your Project.toml. To initialize these bounds with current dependency versions, use the ]compat command in the Pkg mode of the REPL, or the package PackageCompatUI.jl.

Over time, new versions of your dependencies will be released. The CompatHelper.jl GitHub Action will help you monitor upstream Julia dependencies and suggest changes to your Project.toml’s [compat] section accordingly. In addition, Dependabot can monitor the dependencies… of your GitHub actions themselves. Both of these are included in the default PkgTemplates setup.

Tip

It may also happen that you incorrectly promise compatibility with an old version of a package and not realize it (since Pkg prefers newer versions within the compatibility bounds, not all combinations get tested). To prevent that, the julia-downgrade-compat GitHub action tests your package with the oldest possible version of every dependency, and verifies that everything still works.

23.9.2 Registration

If your package is useful to others in the community, it may be a good idea to register it, that is, make it part of the pool of packages that can be installed with

pkg> add MyAwesomePackage  # made possible by registration

Note that unregistered packages can also be installed by anyone from the GitHub URL, but this is a less reproducible solution:

pkg> add https://github.com/myuser/MyAwesomePackage  # not ideal

To register your package, check out the general registry guidelines. The Registrator.jl bot can help you automate the process. Another handy bot, provided by default with PkgTemplates.jl, is TagBot: it automatically tags new versions of your package following each registry release. If you have performed the necessary SSH configuration, TagBot will also trigger documentation website builds following each release.

23.9.2.1 Local Registry

For distributing privately (or publicly if you make the repository public), LocalRegistry.jl provides convenience functions for creating a new registry, adding new packages, and updating versions for the packages. If you want to share packages internally, create and register packages to a repository that’s hosted somewhere you and your team can access. If you wanted to make the repository public, you can publish the registry repository somewhere publicly accessible (such as a public GitHub repository).

Once established, other users can add a repository as easily as entering package mode and running registry add. Say that we have already put a registry we called FinancePackages in a repository on the company intranet:

pkg> registry add http://company-intranet.com/git/FinancePackages.git

23.9.2.2 Hosted Registries

Alternatively to a self-hosted local registry, third party services such as JuliaHub provide managed registries well suited for corporate environments.

23.10 Reproducibility

Obtaining consistent and reproducible results is an essential part of model auditing and compliance. One tool to consider is DrWatson.jl. It is a general toolbox for running and re-running models in an orderly fashion.

Some specific issues come up in attempting to ensure reproducibility:

A first hurdle is random number generation, which is not guaranteed to remain stable across Julia versions. To ensure that the random streams remain exactly the same, you need to use StableRNGs.jl. The downside to this is that the random number generation will be considerably slower than the usual generator.

Another aspect is dataset download and management. The packages DataDeps.jl, DataToolkit.jl and ArtifactUtils.jl can help you bundle non-code elements with your package (some of these rely on artifacts - discussed in Section 12.6.3).

TipFinancial Modeling Pro Tip

Always version-control both Project.toml and Manifest.toml for regulated (auditable) workflows. Instantiating the environment on another machine ensures identical dependency versions, which is crucial for reproducible risk and valuation reports.

julia> using Pkg
julia> Pkg.activate(".")
julia> Pkg.instantiate()   # resolves to exact versions recorded in Manifest.toml

Note that for this to fully work, the replicating machine needs to be the same architecture (e.g. x64), OS (e.g. Windows), and Julia version (e.g. v1.10). If the versions differ, Julia may need to use a different set of dependencies for compatibility reasons. However, it’s still a good practice to store the Manifest.toml for important workflows.

And remember, that with package repositories, you generally do not want to check in the Manifest.toml. Instead, create scripts for the production workflows that do check in the Manifest.toml.

23.11 Interoperability

To ensure compatibility with earlier Julia versions, Compat.jl is your best ally.

Making packages play nice with one another is a key goal of the Julia ecosystem. Since Julia 1.9, this can be done with package extensions, which override specific behaviors based on the presence of a given package in the environment. For example, if you want to provide pre-configured plotting, but don’t in general need to include a plotting library as part of your package for all users and use cases. PackageExtensionTools.jl eases setting up extensions for your package.

Furthermore, the Julia ecosystem plays nice with other programming languages too. C and Fortran are natively supported. Python can be easily interfaced with the combination of CondaPkg.jl and PythonCall.jl. Other language compatibility packages can be found in the JuliaInterop organization, like RCall.jl.

23.12 Customization

Part of interoperability is also flexibility and customization: the Preferences.jl package gives a nice way to specify various options in TOML files. These customizable preferences persist across sessions and provide the preferences at both compile and runtime. For example, say different parts of a company had different preferred data sources but otherwise used the same code. This could be set in a way via Preferences.jl so that each team can share the logic while seamlessly defaulting to different data sources.

23.13 Collaboration

Once your package grows big enough, you might need to bring in some help. Working together on a software project has its own set of challenges, which are partially addressed by a good set of ground rules like SciML ColPrac. Of course, collaboration goes both ways: if you find a Julia package you really like, you are more than welcome to contribute as well, for example by opening issues or submitting pull requests.