Developing and publishing open-source software and my experience with an open peer-review process

Picture: Opening the door. Credit: Harrison Haines (Pexels).

by Navid Constantinou

I used to code very sloppily… I mean very sloppily. Just consider the file naming conventions I was typically using:

script_v2.m
script_v2_new.m
script_v2_new_today.m
script_v2_new_afterchatwithSarah.m

Perhaps this pattern looks familiar to some? 🙄 While I had some basic principles engraved in my head like “start simple”, “do an example first for something you know what to expect”, “build-up complexity slowly”, I was lacking the tools and structure that software developers have been using for a long time now, like, version control, continuous integration tests, and – believe it or not – documentation!

Back in 2017 a friend of mine started learning Julia, a free, open-source programming language that has recently been gaining in popularity. We were both at the Atmospheric and Oceanic Fluid Dynamics conference in Portland, Oregon, when on Monday he claimed “Dude, I’m going to rewrite all my code to reproduce my talk’s results in Julia for my presentation on Friday.” 

I then, painfully, watched him writing code for four days straight, while never actually running or testing anything. On Thursday afternoon he (supposedly) had everything done and was ready to hit “run”. Of course the code didn’t work. There are always bugs and mistakes. Personally, I make about one mistake per line of code.

Fortunately, the story didn’t end badly for my friend as he already had a Matlab code from which he got the results he needed for his presentation. But after the conference was over, we teamed up and started learning Julia together. 

The free evolution of the vorticity of two-dimensional turbulence. Same-sign vortices tend to merge into bigger vortices. If we run long enough we will end up with a so-called vortex condensate, a single pair of vortices, one positive and one negative.

To learn Julia, we thought we’d code up some pseudospectral partial differential equations solvers for some simple problems we needed for our research. But this time we were determined to do it properly: test everything, document all functions, write tutorials to help users (and also ourselves) to understand how to use the code. It turns out that only about 10%-15% of what you write is the actual code. The rest is split between tests for what the code is supposed to do, docstrings for all functions and types, elaborate documentation, examples, tutorials, etc.

All our code development was done in GitHub, which is open source and publicly available to anybody in the world. Everybody could see our mistakes, our hiccups, our fixes. At some point people started chiming into discussions about the issues we had. Some even started contributing to the code. Out of this journey emerged GeophysicalFlows.jl, a Julia package that provides solvers for geophysical fluid dynamics problems in periodic domains.

GeophysicalFlows.jl then started to gain popularity. People were actually using it for research. That inspired us to put together a small announcement paper for the package. The Journal of Open Source Software (JOSS) seemed the ideal place to make our submission. Papers in JOSS are usually short and contain just a summary, a statement of purpose for the package, and a brief summary of the current state of the field. The package’s code and its documentation are both reviewed, along with the paper itself. This review process is completely open: everything takes place in a GitHub issue and chats between the authors and reviewers are encouraged.

This open review process has been mind changing for me. For the first time I actually really enjoyed the review process. We were able to chat with the reviewers and the editor, ask for clarifications to their remarks, etc. We were even using emojis in the exchanges with reviewers 😉 – sure, that doesn’t sound important, but I like emojis! The reviewers’ remarks were addressed by changes to the paper and also by changes in the code, pull request, GitHub issues, etc. The reviewers often commented and reviewed the code pull requests.

As soon as the reviewers and editor were happy with the changes, the paper was accepted and after a day it was published with a working doi and what not! And all this for free! It makes you wonder what goes on in all these supposedly high-impact journals where when you get a paper accepted it usually takes about a month for the editorial staff to take the LaTeX-version of the submitted manuscript, convert it into something like Word format, introduce a bunch of typos, mess up some equations, and return a PDF document with the proofs back to the authors allowing them 48 hours to fix everything. And to think we pay them something close to 3k AUD for the privilege. It makes you wonder sometimes…

Good science requires good software – software development is crucial for science to move forward. My take from all of this: don’t hesitate to put effort into developing good software. Perhaps it’s true that the (academic) gains for the effort may be more longer-term than that of writing up a paper. But in the long run, good software will help you and others to pursue good quality research. 

Follow this link to read more about GeophysicalFlows.jl in the related Research Brief.

By Alvin Stone

Alvin worked as an editor with Fairfax Community News and then News Local for over a decade before moving across to media communications. As a media communicator he has worked for WWF-Australia and most recently Primary Communication, a boutique agency specialising in corporate clients in the energy, transport, IT and not-for-profit sectors.