Introduction

The art and LArSoft projects have provided binary distributions supporting various operating systems for experiments and projects to use. Although Linux distributions have been provided for a long time, in the last several years, various packages have supported macOS1 and Ubuntu platforms.

The general policy of the SciSoft team is to provide builds for the Linux, Ubuntu, and macOS platforms, limited to the current operating system (OS) and the one previous. The current list is:

  • Scientific Linux (Fermi) 6 and 7
  • Ubuntu LTS 16 and 18, builds generated on demand (LTS 18 builds not yet fully supported)
  • macOS 10.13 (High Sierra) and 10.14 (Mojave)

Producing builds for the Linux platforms will continue to be the primary focus as all grid systems support Linux. However, support for SLF6 has become more difficult as experiments request newer software technologies to use on a platform that has obsolete native software.2 Ubuntu builds are produced by request and generally are not difficult to support due to the similarity of its environment to Scientific Linux.

The platform that is the trickiest to develop for is macOS, the difficulties of which are partly explained in the next section.

macOS support complications

The following list enumerates some of the realities that the SciSoft team must cope with in order to support macOS builds.

  • Each fall, Apple releases a new OS, which must be vetted by Fermilab before the SciSoft team can start creating builds that support it. This yearly release cadence has been a struggle to keep up with.
  • Apple’s Xcode toolkit, which must be installed before a user can run art on a macOS platform, periodically releases updates with no warning. Any given Xcode update, if installed by a user, can break a currently functioning art environment if binary incompatibilities are introduced between the currently installed art packages and the Xcode update (see next bullet point).
  • Changes in macOS system headers have come into conflict with GCC, the primary compiler that has been used by the art and LArSoft projects. This conflict forced the SciSoft team to move to a different compiler for macOS support. The open-source Clang compiler was chosen and has been used on both macOS and Linux platforms. This change has been for the better, but the effort to get there was sizable.3
  • With Apple’s El Capitan OS (10.11), system integrity protection (SIP) was introduced, which is a security mechanism that conflicts with art’s execution model, which is Linux-based. The consequence is that users who wish to execute art jobs on their macOS laptops must disable SIP. Providing macOS builds that do not require SIP disablement is a significant issue that is being partly addressed by the Spack/SpackDev work done by SCD.
  • For macOS-specific development, the SciSoft team has had to use personal laptops and sometimes loaner laptops from the laboratory due to a lack of development infrastructure for macOS. In rare cases, select members have had access to macOS build machines for testing code.
  • External libraries’ requirements for macOS change, and such changes must be handled to ensure seamless interoperability with a UPS environment (e.g. NumPy requiring a BLAS package).
  • Rumors persist that macOS is planning to move from Intel hardware to ARM-based custom chips in the near future. If this move takes place, the SciSoft team will likely face additional complications in supporting macOS builds.

Due to these complications, it was necessary for us to poll the art- and LArSoft-using experiments as to their OS needs.

OS survey

On October 30, 2018, the SciSoft team leaders sent a survey to art and LArSoft users to determine OS needs and preferences. The message sent to users stated:

The SciSoft team, which supports the art and LArSoft projects, is re-evaluating the platform needs of its users. In particular, as the macOS environment continues to diverge from the Linux environment, it is important for the team to understand how much of the SciSoft community relies on macOS and whether that need can be met with available effort and infrastructure.

Survey questions

We requested the following information:

  1. Please give us your email in case we have followup questions.
  2. Please characterize your computing roles (select all that apply).
  3. If you are a developer, please select the experiments/projects to which you commit code (select all that apply).
  4. With what resources do you do your development?
    1. Of the resources you list in question 3, which resource is used for your primary development/analysis, and why?
  5. If you do development work on a separate platform than the one on which you do analysis, please explain.
  6. If you have a laptop or desktop, which OS are you running?
  7. If you’re not using your laptop/desktop for development or analysis, would you if the OS were supported?
  8. If we were to add support for one more OS, what would be your preference, and why?
  9. If the SciSoft team were to drop support for macOS, how would this affect your code development?
  10. If given an unconstrained choice, what type of machine would you use for development / analysis?
  11. Please provide any additional comments.

There were 46 respondents to the survey, the results of which are given below.4

0. Please give us your email in case we have followup questions.

Of the 46 respondents, 38 provided email addresses for us to contact them. We trust that no one who provided an email address also filled out the survey anonymously.

1. Please characterize your computing roles (select all that apply).

2. If you are a developer, please select the experiments/projects to which you commit code (select all that apply).

3. With what resources do you do your development?

3a. Of the resources you list in question 3, which resource is used for your primary development/analysis, and why?
Fermilab GPVMs because I began conducting my research there.
Fermilab GPVMs
Laptop because it does not require a network connection unlike the gpvm nodes
It depends how much access I need to the grid and central samples: if I need the grid I go on gpvms otherwise for smaller developments I prefer the laptop
Fermilab GPVMs – software already set up consistently with use instructions, available CPUs
Fermilab GPVMs
Fermilab GPVMs, they are the SLF6 computers I have accesses to
dunegpvm. It is a clean and consistent environment for software that all dunetpc users should have access to.
It is very efficient to have LArSoft on my mac and develop using Xcode. The Xcode debugger is easy to use.
desktop and GPVMs are pretty equivalent. I just use my laptop to connect to my desktop and the GPVM’s. I don’t compile framework code on it.
I do most analysis on my laptop – it goes with me
Fermilab GPVMs —> no LArSoft/uboonecode build for ubuntu
Fermilab GPVMs because then I know it will work on the grid
GPVMs
Fermilab GPVMs - occasional problems with working on Mac
Laptop, since I can work when the internet connection is not good
CERN lxplus, mostly because I am based there
Laptop, with me always, no network latency, better development tools (debugger, code profiling, etc)
GPVM, because the mu2e environment is always current, best build speed, best professional support, pnfs mounted
laptop, most convenient
Fermilab GPVMs
Easily configurable environment, packages already installed
Using GPVM’s because I still have to set up a decent container on my unsupported Linux desktop and laptop.
I use laptop mostly as it is always handy.
Laptop for SW development: convenience, and work on the go without needing internet access. GPVMs for analysis - access to data. Also GPVMs when expect a lot of large recompiles for a particular SW project - mu2ebuild compiles faster than my laptop.
Fermi GPVM - only place where I can run larsoft and dunetpc without too much hustle
BNL laptop and desktop. I’m more efficient on a low latency local connection than SSH’ing somewhere and likewise using an OS that provides the tools I need (Ubuntu). Having root access it critical. Other reasons include large amounts (large on the scale of personal use) of cheap and local disk is available. My workstation has 40 cores which helps compile times. My laptop has very fast NVMe which helps testing throughput for prototype DUNE FD DAQ code.
GPVMs because I don’t know how to use my laptop that well and I have cleaner access to running on the grid and it is easier to use UPS
laptop; accessibility (commuting, home, etc.)
Desktop as it’s primary resource I use. Test on MacOS as it’s convenient to be able to run our software in a small standalone environment (and my laptop is a Mac).
Fermilab GPVM
CERN lxplus since I am based at CERN
Local desktop - interactive latency
desktop; for compatibility and speed
Development & small analysis jobs: Desktop (for best performance). Large analysis jobs: Batch farm (for maximum # of cores)
Desktop: Most available and powerful resource to hand
desktop
Generally GPVMS/build machines for development, and a mix of those and laptop for analysis, but it goes back and forth. I’d like to do more of everything on my laptop, but it’s difficult to work out the python versions for OSX sometimes.
I use both depending on outage schedule etc. Latency is better when developing locally, bug GPVM is needed for grid access.
Fermilab desktop. It’s dedicated resource to myself, and kept up 24/7.
I use code editors on my laptop, but do all compiling and running on the GPVMs.
primary development on icarusgpvm, analysis of root ntuples locally. convenience of environment setup and ease of sharing code and data.
GPVMs. I want to use the same resources as most of my colleagues do so that I will see the same issues that they do.

4. If you do development work on a separate platform than the one on which you do analysis, please explain.
Same 3-a: it depends how much access I need to the grid and central samples
development of code, especially code for private use, can be more convenient on a laptop or desktop. This is particularly true if investigating behavior of a small piece of LArSoft or a detector sim, such as drift velocity calculation code or geometry files.
I check out code to run on my Fedora Linux laptop while traveling.
Do most art development on GPVM, b/c it is easier than running my own VM on my Mac
The vast majority of my work is done externally to art/LArSoft. Part of the reason is because I can iterate quickly with local compiled code (no LArSoft build available for my OS).
I do everything on GPVM
Can work with a desktop if I am in the office.
Development of Wire-Cell is primarily on Ubuntu while production processing is on SL7. To test for SL7, Singularity containers are used. No Mac is currently used for testing other than what gets done on FNAL CI. Reasons are same as above.
I don’t
Use multiple platforms for analysis
usually development/small tests on my laptop, higher statistics analysis on GPVM
A desktop/workstation offers significantly better build performance for me than central systems (at Jefferson Lab) with shared disks. I think this is largely due to better disk I/O.
Generally the build machine is a bit faster for doing compilation/pushing up code.
I use MacOS machines to analyze TTrees produced on Linux machines.
Fermilab GPVMs. It has NFS mount of dCache, where I can quickly grab a data/MC file to test and analyze.
too lazy to learn how to setup environment for art/LArSoft on my laptop

5. If you have a laptop or desktop, which OS are you running?

6. If you’re not using your laptop/desktop for development or analysis, would you if the OS were supported?

7. If we were to add support for one more OS, what would be your preference, and why?

There were 19 of 46 respondents to this question.

macOS

Two of the seven macOS respondents therefore preferred that macOS Mojave would be added, which was not yet supported at the time when the survey was distributed. However, the remaining five respondents have more cryptic responses, some suggesting an unawareness that macOS is supported by SciSoft:

  • “macOS because it is what most users have on their laptops.”
  • “macOS any (but Mojave if I have to choose, because it will have the longest support life).”
  • “macOS, most of my collaborators’ personal computers are macs”.
  • “mac – almost everyone uses them”
  • “I am solidly dependent on OS X”
Ubuntu

Of the seven Ubuntu respondents, five would like Ubuntu 18.04 LTS, which is not yet supported by the SciSoft team. One of those five wrote:

  • “Ubuntu 18.04 as it’s the new LTS. Until adopting Singularity the lack of Ubuntu 18.04 support in UPS products has held me back upgrading some systems. It would be nice to add equivalent support for Ubuntu UPS products in CVMFS particularly experiment code builds. However, I must admit that since adopting Singularity to provide an SL7 image the real need for such parity is reduced.”

The two other Ubuntu respondents wrote:

  • “Better ubuntu support”
  • “Ubuntu. It has a large user base. It comes with newer kernel and compiler by default than REHL, and many new tools are firstly released/available in its official repo.”

where the latter comment indicates a respondent who was not aware that Ubuntu builds are already provided on demand.

Other

The “Other” respondents provided the following comments:

  • “Anything that helps expand the bulk resources we can use, for example, on new grid sites or supercomputers - threading, KNL specific compiler, track fitting on GPU and TPU”
  • “Whatever has the largest installed base of resources we can use. Maybe ubuntu?”
  • “VAX-VMS”

8. If the SciSoft team were to drop support for macOS, how would this affect your code development?

We also plot the same results but split according to experiment. Note that as individuals were allowed to specify more than one experiment, the same individual’s response may appear in multiple experiments below.

9. If given an unconstrained choice, what type of machine would you use for development/analysis?

Of the 46 survey respondents, 15 did not respond to this question.

Number of respondents
Linux 2
macOS 11
macOS (analysis), Linux (development) 2
SL7 3
Ubuntu 6

Other responses included:
a desktop equivalent
Desktop with Ubuntu for development and containers on HPC for analysis.
Development/small analysis: iMac Pro. Major analysis: Batch farm
Fedora
lab-supported SL6 or 7, 100-core with TB of SSD disk
Laptop
Whatever matches the largest base of installed compute nodes that we use; currently SLF6.

10. Please provide any additional comments.
Positively: Mac is a hindrance to a clean, centralized environment.
People would be much more efficient if there was a lab-supported IDE.
It would be easier for me – fewer releases to support. Otherwise not at all.
Emulators are not really an option unless can still access development tools
As long as I have a reasonable network connection, which for me is always, I will always go to GPVM. I know some like to work offline, always syncing the code and environment to some other machine, but it would be more effective for the lab to buy them a fast uplink dongle. I know people always say “What if I’m on a plane?” But really how often is that, and can’t you find a few papers to read? Or these days, just pay for wifi..
I currently run a CentOS 6 virtual machine on my laptop. It is on this virtual machine that I develop and test my art based code.
Did I mention containers? An Ubuntu 18.04 LTS container allowing to contact host’s X11 server should be enough to solve the Linux support side. I know this survey is more about OSX, but a docker solution might ease the development for some of the OSX users too.
Lossing Mac: I would lose an option for possible future transition onto my laptop
Just to reiterate an above comment. A monoculture platform for development and use is bad for code quality. Supporting variety gives the beneficial side effect of shaking out bugs that would otherwise be hidden in the monoculture. This additional benefit that should be added in support of what might otherwise be seen as wasted effort to support a small user base. As much as I don’t care about Mac OS X personally, supporting just SL7 and Ubuntu brings us closer to a monoculture. Supporting just SL7 would be very problematic.
you guys are doing great and thank you very much for putting this together, I think it does a lot for the community to know that you guys are listening and trying to make development easier for everyone. Keep up the great work!
I have found over the years that development on several, not-too-similar platforms, like Linux and macOS, can help uncover subtle coding issues, unwarranted assumptions etc. and leads to overall better code quality, standards conformance, and thereby portability and upgrade resilience. For that reason alone, I would be loath to see macOS support go. Also, many members of our group, including myself, have personal Macs and seem to be noticably more productive if they do not have to switch to working on remote systems for development. I think that–especially for development and quick tests–people greatly appreciate being able to run /art/ natively on their personal machines.
If native macOS support is dropped, there must be an alternative that does not require an always on network connection (such as “ssh to a supported resource”), hence the preference for adding Docker support (I consider CVMFS semi-offline modulo careful caching). Official support for macOS is less a concern than retaining an ability to attempt build and installation on unsupported (or future) platforms. It’s fine if there are big warnings about “not supported, use at risk”, it’s fine if things then break, but it’s not fine if the system actively prevents experimentation and exploration here (as UPS/cetbuildtools do). Retaining the possibility of portability to unsupported systems does not imply support has to be added for those systems! Equally, SciSoft should remain open to accepting patches to fix issues on unsupported platforms (provided they don’t break the supported ones of course!).
Mu2e doesn’t support MacOS so art MacOS support is currently irrelevant.

Survey conclusions

In light of the large effort required, the small number of macOS users served, and the reduced effort available, the SciSoft team plans to shed support for macOS. No imminent actions will be taken regarding these findings. Discussions, however, are underway to determine a path forward for those users that rely on macOS.

  1. By “macOS” we refer to the set of operating systems that Apple has used for its Mac line of computers, including Mac OS X, OS X, and macOS.↩︎

  2. The end-of-life for SLF6 is late 2020.↩︎

  3. Informing the SCD-supported build system (cetbuildtools/mrb) of the location of the open-source Clang libraries is not yet resolved. Strictly speaking, the code still works, but only accidentally.↩︎

  4. A 47th respondent was a member of the SciSoft team, and their responses have been omitted from the results shown here.↩︎