Software Development Best Practices

Best practices for developers of CIG hosted codes are specified at three different levels:

Minimum Best Practices. Minimum expectation for all codes hosted by or distributed by CIG.
Standard Best Practices. All codes should be following this level of best practices. If not, codes should have a plan of active development to achieve this level.
Target Best Practices. Codes under active development should have a long-term plan to implement these practices.

More information on all CIG best practices can be found in our git repo [best practices].

Minimum Best Practices

Practices that codes must follow in order to be accepted by CIG.

1.		Licensing
		a. Open source license. See licensing.
2.		Version control
		a. Use of version control (e.g., subversion, git) to manage code changes
3.		Portability, configuration, and building
		a. Code builds on Unix-like machines (Linux, Darwin) with free tools (compilers)
		b. Well designed, portable build system (e.g. cmake, make, configure-unix only, setup.py, etc.)
4.		Testing
		a. Code includes tests to verify that it runs properly.
		b. Results of accuracy and/or performance benchmarks if established by the research community.
5.		Documentation
		a. Instructions for building and installing the code.
		b. Description of all parameters including units (if dimensionless, specify scaling used).
		c. Explanation of physics the code simulates.
		d. Illustration of how to use the code to solve scientific problems.
		i. Cookbook examples with sample, editable input files.
6.		Citable publication

Standard Best Practices

Practices in addition to the Minimum Best Practices that should be used by all codes developed within the CIG community. Codes not meeting all standards should be actively working to eliminate deficiencies.

1.		Version Control
		a. Differentiation between maintenance (bug fixes) and new development.
		b. Source tree limited to files necessary to build software and documentation and run small verification tests.
2.		Coding
		a. User-friendly specification of parameters (e.g., graphical user interfaces, human readable parameter files) outside of source code/specified at run time
		b. Development plan (updated yearly) with prioritization of new features and estimated timetable for their implementation
		c. Comments in code describing:
		i. Algorithms with appropriate references
		ii. Purpose of functions, objects, etc. and descriptions of arguments (inputs / outputs), groups of objects, and tutorials
		d. Modular design
		i. Balance use of external libraries (e.g., PETSc) to maximize reuse while minimizing dependencies and maintenance
		ii. Allow users to extend code (new features or alternative implementations) without destroying original functionality or modifying main branch.
		e. Error trapping
		i. User errors should result in a message that helps the user correct the problem. User errors should not result in a crash.
		ii. Internal errors are generally bugs (or unintended uses). Ideally consistency checks will catch internal errors and generate an error message that helps the developer fix the problem.
		f. Scalability
		i. Use of distributed/parallel data structures
		ii. Messages used to transfer information (e.g., MPI) instead of filesystem
3.		Portability, configuration, and building
		a. Verification that dependencies are available and usable.
		b. Automation and portability of configuration and building.
		c. Output all configuration and build options during runtime (e.g. commit id, compiler options, checksum) to facilitate reproducibility.
4.		Testing
		a. Code includes pass/fail tests that verify it runs properly.
5.		Documentation
		a. User documentation: workflow for research use.
		b. Developer documentation: description of how to extend code in anticipated ways.
6.		User workflow
		a. Running different simulations does not require rebuilding.
		b. User specified directories and filenames for input and output.
		c. Use of standard binary file formats (e.g., NetCDF, HDF5).
		d. Citation for code version.

Target Best Practices

Practices in addition to the Standard Best Practices. Desirable practices that developers should implement in defining long-term development priorities for codes developed within the CIG community.

1.		Version Control
		a. New features added in separate branches.
		b. Stable development (master) branches for rapid release of new features.
2.		Coding
		a. Functionality implemented as a library rather than an application
		i. Leverage alternative implementations via plugins.
		ii. Extend library features in applications without modifying original code.
		iii. Construct higher level applications using libraries as building blocks.
		b. Output of provenance information (parameters used).
		c. Scalability
		i. Parallel access to inputs and outputs (e.g., HDF5).
		d. Checkpointing.
3.		Portability, configuration, and building
		a. User can select compilers, optimization, additional build flags during configuration without modifying files under version control.
		b. Permit multiple builds using the same source tree.
		c. Software can be installed to a central location.
4.		Testing
		a. Pass/fail unit testing for code verification at a fine grain level.
		b. Method of Manufactured Solutions for code verification at a coarse grain level.
5.		Documentation
		a. Guidelines on parameter scales/combinations for which code is designed/tested.
		b. FAQs or knowledge base.
6.		User workflow
		a. Reproducibility via archiving of workflow