MoeVelGenerator: the MOEA-to-VELMA Configuration Generator

Overview

This guide explains what the MoeVelGenerator utility tool is for, and how to use it.

Background

Running the MOEA Framework for VELMA produces a MOEA outputs .csv file with multiple rows of results, one row per VELMA simulation run. Via the values in its columns, a given row reports the objective scores and VELMA parameters settings (categorized into "variables", "followers", and "exclusives") unique to a particular VELMA run.

As an example, here is a screen-capture of part of the contents of a MOEA outputs.csv file:

With the contents of the MOEA output .csv file available, the next step in the user's workflow is usually an analysis of which VELMA simulation runs (i.e. rows of MOEA output) are the "best" -- based on the objectives columns scores. For one or more of these best runs, the user typically wishes to re-run VELMA outside of MOEA, to generate full VELMA results and analyze them in detail. But, in order to run VELMA with the same simulation configuration parameters that produced the "best" row of objectives scores the user must recreate the specific .xml configuration file that MOEA ran VELMA with, because MOEA does not report/save that .xml file anywhere.

The .xml file for a particular row can be created by duplicating the original ("source") VELMA simulation configuration .xml file (specified for use when MOEA ran), and then subsituting each value in the row from a column representing a VELMA parameter inot the duplcate .xml. When this process in completed, the duplicate .xml is the .xml file that MOEA ran to produce the results for the given output row.
(Although perhaps not exactly the same: the user will likely set VELMA's suppressAllOutputFiles false instead of true, and change the run_index and .xml file's name to make it and its full results directory easy to identify.)

Performing the above work manually can be a slow and mistake-prone process, and it is exactly this process that the MoeVelGenerator tool automates: generating a VELMA .xml configuration file from a row of a MOEA outputs .csv file, using a VELMA source .xml as a starting template.

Platform Requirements and Runtime Notes

Running MoeVelGenerator requires the following:

• Python 3.4 or later
• Windows 10 (although the tool is not Windows-specific, Windows it the only platform it has been tested on).
• Windows Powershell (other shells should work, but PowerShell is the command shell it has been tested on).

The Step-by-Step Instructions assume a Windows 10 and Powershell environment.

Step-by-Step Instructions

These instructions start assuming you have already run MOEA for VELMA, and the MOEA configuration and outputs .csv files are available and accessible.

1. Determine the VELMA simulation configuration .xml file used as the source for the MOEA output results.

The original ("source") VELMA simulation configuration .xml file that MOEA used to build the .xml files for each row of its output results is specified in MOEA's own configuration .csv file. Open that file and find the configuration record (row) containing the keyword "VELMA_SOURCE". The third column in the row should contain the fully-qualified path + name of the VELMA source .xml file. Make a note of this path + file name for use as the velma_xml_source command line argument in Step 3 below.

You can look for VELMA_SOURCE either by opening the MOEA configuration .csv file in Excel, or by using PowerShell's Select-String cmdlet. Here are examples of each:
In Excel:
Via Select-String:

2. Open a PowerShell command window

(Note: there are many ways of opening PowerShell -- this guide details only one of them.)

1. Click the Windows Start icon to open the start menu. Note: do not click into the menu, just open it.
2. Type "powershell": Windows will replace the start menu with its search results menu.
3. Click the "Windows PowerShell" desktop app: it should be the "Best match", at the top of the list of search results.

3. Run MoeVelGenerator

As mentioned, you must have Python 3 installed and available via the Powershell command window to run MoeVelmGenerator.

To run MoeVelGenerator, start Python 3 and pass it the fully-qualified location and name of the MoeVelGenerator .py file, followed by command line arguments for MoeVelGenerator.
(Typically, Python 3 is started from a PowerShell as py -3, however this may differ on your system; python and python3 are other possibilities.)

But wait -- what are the command line arguments for MoeVelGenerator?
You can pass MoeVelGenerator the single argument "-h" or "--help" to ask it list its required and optional arguments.
Here is an example, run immediately after starting a Windows Powershell as detailed above in Step 2:

As the help message indicates, there are 3 required and 1 optional command line arguments that must must appear in the order shown by the message. (The help message refers to these 4 arguments as positional because they must appear on the command line in the order specified)

1. moea_csv_filename
   (required) the path and filename of the MOEA outputs .csv file to extract parameterization data from.
2. moea_csv_row_index
   (required) the (integer) row number of the data row to extract parameter values from. As the message indicates, the index number is 1-based, and includes the header row -- so, the index of the first row of the file that contains parameters is "2".
   (Backstory note: the index numbering scheme was chosen to correspond to the row numbers of the .csv file when viewed in Excel.)
3. velma_xml_source
   (required) the path and filename of the VELMA simulation configuration .xml file used as the source for the MOEA/VELMA run that generated the MOEA outputs .csv file specified as argument #1 (and that you determined and noted in Step 1 of this guide).
4. velma_xml_target
   (optional -- but highly recommended) the name of the VELMA simulation configuration file that MoeVelGenerator will create.

What about when the VELMA target .xml file already exists?

MoeVelGenerator won't automatically overwrite a pre-existing file specified by velma_xml_target, however, you can force it to do so by adding the optional --overwrite argument to the end of the command line.
The following example illustrates how this works, by first generating a new .xml file, then attempting (and not) overwriting it, then successfully overwriting it by including the prescribed argument:

What happens when the VELMA target .xml file is not specified?

As noted, the velma_xml_target argument is not required, but it is highly recommended.
When it is left off the command line, MoeVelGenerator echoes the .xml file that it would have written to stdout (i.e. the PowerShell command window itself) instead. It is possible to still capture the xml by redirecting stdout to a file, but we recommend against this. For one thing, there is no check for whether the file you redirect stdout to exists or not, so capturing the output in this way circumvents the "file already exists?" check that MoeVelGenerator would have performed for you. For another, the automatic equating of the VELMA run_index parameter and the velma_xml_target file name doesn't happen.

What about the run_index and suppressAllOutputFiles parameters?

MoeVelGenerator sets the value of the VELMA run_index parameter to the file name specified by the velma_xml_target argument's values. Only the file name portion is used; the path and dot-extension portions are ignored. As noted above, if no value is specified for velma_xml_target, the value of run_index is left untouched.

MoeVelGenerator does not set suppressAllOutputFiles to true, even if (as is usually the case) it is a parameter in the specified MOEA outputs data row. This is because MoeVelGenerator is usually used to create a simuator configuration for a full-results run.