Merge pull request #122 from tlestang/compare_reference_tgv_output

Compare reference tgv output

.github/workflows/Build.yml

@@ -46,3 +46,16 @@ jobs:
       run: |
         export PATH=$(pwd)/openmpi-4.1.4/installed/bin/:$PATH
         make BUILD=debug
+
+    - name: Run TGV case
+      run: |
+        cp test/data/Taylor-Green-Vortex/reference_input.i3d input.i3d
+        export PATH=$(pwd)/openmpi-4.1.4/installed/bin/:$PATH
+        mpirun -np 2 xcompact3d
+
+    - name: Compare output time evolution to reference output
+      run: |
+        pip install numpy
+        python test/compare_TGV_time_evolution.py \
+            --input time_evol.dat \
+            --reference test/data/Taylor-Green-Vortex/reference_time_evol.dat

test/README.md

+# Testing
+
+## Semi-automated Taylor-Green vortex test case
+
+### Reference output data
+
+File `data/Taylor-Green-Vortex/reference_time_evol.dat` contains
+reference output values for the TGV test case. The values were
+generated base on input file `reference_input.3d` and running the `xcompact3d`
+executable as
+
+```
+mpirun -np 2 xcompact3d
+```
+
+See metadata.txt for the revision number the executable was compiled from, as well as
+`gfortran` and OpenMPI versions.
+
+#### Generating reference output data
+
+It is possible to update and push new test data manually. However, **for
+reliability and consistency reasons the recommended way is to generate
+output data through the dedicated GitHub action** *Generate reference
+test data*.
+
+On the "Actions" tab, select the *Generate reference test data* action
+on the left hand side panel:
+
+![](img/find_test_data_generation_workflow.png)
+
+Next, trigger the workflow on the `master` branch run by clicking on
+*Run workflow* as shown below.
+
+![](img/run_test_data_generation_workflow.png)
+
+Following successful completion, the workflow will open a new Pull
+Request titled /Update reference output TGV data/. 
+
+### Comparing output to reference data
+
+```
+cp test/data/Taylor-Green-Vortex/reference_input.i3d input.i3d
+./xcompact3d # Generates time_evol.dat
+python test/compare_TGV_time_evolution.py \
+    --input time_evol.dat \
+	--reference test/data/Taylor-Green-Vortex/reference_time_evol.dat
+```
+
+### Automated test on Pull Request
+
+The above test will run automatically each time new commits are pushed
+to a remote branch for which a Pull Request is open for integration in
+the `master` branch.
+
+This workflow is based on GitHub actions and described in the
+[.github/workflows/Build.yml](https://github.com/xcompact3d/Incompact3d/blob/master/.github/workflows/Build.yml) configuration file. This workflow
+
+1. Builds and caches OpenMPI.
+2. Builds Incompact3d.
+3. Runs the `xcompact3d` executable on reference input file `reference_input.i3d`.
+4. Runs the `compare_TGV_time_evolution.py` script to compare
+   generated output to reference output.
+
+
+
+

test/compare_TGV_time_evolution.py

+import sys
+import argparse
+
+from numpy import loadtxt, testing
+
+
+desc = "Compare last output to reference Taylor-Green Vortex output."
+parser = argparse.ArgumentParser(description=desc)
+parser.add_argument('-i', '--input', type=str)
+parser.add_argument('-r', '--reference', type=str)
+args = parser.parse_args()
+
+values = loadtxt(args.input)
+reference = loadtxt(args.reference)
+
+all_columns_match = True
+for column_idx in range(values.shape[1]):
+    print(f"Checking column no {column_idx + 1} of {values.shape[1]}... ", end="")
+    try:
+        testing.assert_allclose(
+            values[:, column_idx], reference[:, column_idx]
+        )
+        print("OK")
+    except AssertionError:
+        print("MISMATCH")
+        all_columns_match = False
+
+if all_columns_match:
+    print("SUCCESS -- output matches reference values.")
+else:
+    print("MISMATCH -- some columns do not match reference values, see above.")
+    sys.exit(1)

test/data/Taylor-Green-Vortex/reference_time_evol.dat

@@ -47,4 +47,4 @@
   0.234500000000E+01  0.123649965321E+00  0.822051427310E-03  0.822061505110E-03  0.657641141133E+00
   0.239500000000E+01  0.123608392154E+00  0.840963298341E-03  0.840977734706E-03  0.672770637596E+00
   0.244500000000E+01  0.123565856143E+00  0.860557414162E-03  0.860577791313E-03  0.688445929724E+00
-  0.249500000000E+01  0.123522322715E+00  0.880848661712E-03  0.880876992968E-03  0.704678927002E+00
+  0.249500000000E+01  0.123522322715E+00  0.880848661712E-03  0.880876992968E-03  0.704678927002E+00
\ No newline at end of file