Managing Subdirectories In Share: A Deep Dive

Hey guys! Let's talk about something a bit technical, but super important for keeping things running smoothly: managing subdirectories within the share directory, specifically in the context of ACCESS-NRI and access3-share. We'll be focusing on the src, stubs, and timing directories, figuring out where they came from, and how to best handle them to avoid any headaches down the road. This is a deep dive into the guts of how things are set up, so buckle up!

The share Directory: A Historical Overview

Okay, so the share directory is like a shared workspace, housing essential files and resources used across different parts of the system. Within this, we've got three subdirectories that have caused a bit of head-scratching over time: src, stubs, and timing. Understanding their origins is key to managing them effectively. Micael J. Oliveira, who initially set up these directories, gave us some insights, and we'll unpack those, along with some modern context.

Diving into stubs

The stubs directory is pretty straightforward. Micael created it to provide a stand-in for a specific MCT (Model Coupling Toolkit) routine that the CESM (Community Earth System Model) driver depends on. The idea was that this MCT routine wasn't actually being used, so a simple stub would suffice. This is a common practice in software development: if you have a dependency, but don't need all the functionality, you create a lightweight replacement. This keeps things lean and prevents unnecessary bloat.

Unpacking src

Now, src is a bit more involved. These files are automatically generated during the CESM build process, specifically within CESM_share. Micael took the generated files and copied them into our share directory. The idea was to avoid replicating the entire build machinery, which can be complex. However, this approach introduces a risk: the files in src might become outdated if the original template files (from which they're generated) in CESM_share are updated. This is a critical point that we need to keep in mind, and the core of the problem, so please remember.

The original files are generated from template .F90.in files located in the CESM_share/src directory on GitHub. The generation process leverages modern CESM tooling, specifically cime/CIME/non_py/src/CMake/genf90_utils.cmake. This CMake script calls cime/CIME/non_py/externals/genf90/genf90.pl, a Perl script that does the actual file generation. This means that any changes to the template files or the generation scripts will not be reflected in our local src directory unless we manually update it. This is a problem, because it requires continuous human intervention and a lot of focus, which is not ideal.

Exploring timing

The timing directory contains a copy of the GPTL (GNU Portable Timing Library) timing/profiling library. This was originally pulled from somewhere within CIME, although the exact location is a bit hazy. The core idea is to provide tools for measuring the performance of the code. This is very important for identifying performance bottlenecks and optimizing the code. Like src, the contents of timing might not be up-to-date with the latest version in the CIME repository, leading to potential issues.

Potential Risks and Challenges

The primary risk with the current setup lies in the potential for the src and timing directories to become out of sync with their upstream sources (the original locations where they were copied from). This can lead to subtle bugs, performance issues, or even outright failures if the copied files are not compatible with the rest of the system. For instance, there has already been at least one update to the template files in CESM_share since the src directory was copied. This highlights the importance of keeping these directories up-to-date. Otherwise you have to continuously be updating them manually, or fixing the bugs it can create.

Potential Solutions and Considerations

So, what can we do to mitigate these risks? Let's explore some options:

1. Manual Updates

The simplest approach is to periodically check for updates in the upstream sources and manually update the src and timing directories. This involves comparing the current files with their source counterparts and merging any changes. This method is straightforward, but it can be time-consuming and error-prone, especially if the changes are significant. It requires a developer to be consistently monitoring updates in the original repository, which isn't practical or useful long-term.

2. Automating the Update Process

A better solution could be to automate the update process. This might involve creating scripts that automatically compare the files in the share directory with their upstream sources and apply the necessary changes. This would reduce the manual effort and minimize the risk of human error. It will also reduce the time spent on the project by a developer. This could work by setting up a script that runs on a schedule.

3. Using Git Submodules (with Caution)

Another option is to use Git submodules to include the relevant parts of CIME directly in the share directory. This would ensure that the src and timing directories are always in sync with their upstream sources. However, the CIME repository is quite large, and we only need a small portion of it. Using a submodule would therefore add a lot of unnecessary bloat. Chris B., explored using git-sparse-checkout, which would allow us to only clone specific parts of the CIME repository. However, this didn't seem promising. Therefore, you should not take this approach.

4. Re-evaluating Dependencies

Perhaps we can re-evaluate the dependencies on the files in src and timing. Could we refactor the code to avoid relying on these files altogether? This might be a more radical solution, but it could eliminate the need to manage these directories entirely. You can try to avoid the source of the problem, so you do not have to update those files.

Conclusion

Managing subdirectories in the share directory requires a careful balance between convenience and maintainability. While the current setup might have been convenient initially, it introduces potential risks. By understanding the origins of these directories and considering the available solutions, we can develop a strategy that ensures our system remains stable, performant, and easy to maintain. It's a trade-off: do we want to make our life easy now, or do we want to build a system that is robust long-term? Manual updates are ok, but eventually they will need to be automated or removed entirely. This is a very important consideration.

It's crucial to regularly review and update these subdirectories, especially the src directory, to ensure consistency with the latest CESM_share updates. Consider the automation route to ensure you don't run into a lot of problems in the future. Thanks for reading guys!