Rapyds Application Framework Example

In this example, we will couple a simple external code to the Rapyds environment, and then run the code via Rapyds at the end of the process.

In order to perform this work, we will in-turn assume each of the four described roles of External Developer (ED), Rapyds Developer (RD), Plugin Developer (RD) and Rapyds User (RU).

In working through this manual, it would be helpful that you have the associated fileset at hand to run the provided examples, and adapt the code as per the defined exercises. All files should be available in the set, with the relevant files described in accordance with the specific role which is intended to control and/or develop them.

External code Developer (ED) role

Let us begin by considering the ED role. Here we have to introduce the external code, and consider the input and output files applicable to its usage.

It is often not necessary to assume this role in the coupling process, as the external code is potentially provided as is, or in binary, and as such cannot be altered. However, if the code is under your control, features can be added as needed. Nevertheless, it is needed to have an understanding of the input and output file formats used by the external code.

For this example, let us use a simple python program as external code, with the code listing supplied below. The file set associated with this tutorial is hosted on GitLab. The external program used in this tutorial can be found in this file set, under the name run_me.py.

import os
import yaml
import datetime
import sys

if __name__ == '__main__':
    from dateutil.parser import parse

    # read the input file
    with open(sys.argv[1], 'r') as f:
        inp = yaml.safe_load(f)

    print('Hello', inp.get('name', 'Unknown'))

    dob = inp.get('dob', inp.get('date_of_birth'))

    if isinstance(dob, str):
        dob = parse(dob)

    inp['age'] = int((datetime.datetime.now() - dob).days/365.0)

    o_file = os.path.splitext(sys.argv[1])[0] + '.out'

    with open(o_file, 'w') as f:
        yaml.safe_dump(inp, f)

Can you deduce what this simple program does? In essence it reads an input file which contains some personal information on the user, such as name and date of birth. The program then calculates the age of the user, and writes that to the output file of the program.

The input and output files associated with this program are in yaml format, which is a useful text-based format for keyword based data. The content of the input file (demo.yaml) is listed below, and the file can be found in your plugin/DEMO/ folder, along with the external python code itself.

name: Rian
dob: 1978-11-14 00:00:00

You can run the code with the following command (assuming the inputfile name is demo.yaml):

python run_me.py demo.yaml

Such a run would result in the following output file being produced (named demo.out)

age: 44
dob: 1978-11-14 00:00:00
name: Rian

Rapyds Developer (RD) role

As a Rapyds developer, a code-independent application has to be created to allow the calculation of your age, so that codes which perform this function can be driven by Rapyds. Once this application exists, it can be used for all codes which perform the function driven by this application. Therefore this role need not be assumed if the application of interest already exists, and the process of coupling an external code can be continued from the Plugin Developer (PD) role.

However, in this case, we assume that no application for age calculation exists in Rapyds, and we have to develop it here.

To accomplish this, there are five generic steps for our Demo application:

Define the parameters for the application: Create a DemoParameters class, which derives from ProjectParameters and ApplicationParameters. You can then add any additional parameters needed by your application.
Define the project input structure: Create a DemoProject class, which derives from both InputTemplate and Project.
Define the output processing structure: Create a DemoRecipe class, as placeholder for the code-dependent output processing routines to fetch the output.
Define the results of the application: Create a DemoResults class, which collects all data to store in this application.
Define the application itself: Create a Demo class, which is the Rapyds application itself, and associate to it the defined DemoParameters, DemoInput and DemoResults

Lets proceed and apply each of these steps to develop our age calculator Rapyds application.

Hint

It should be clear that these classes will not all be used at the same stage of the Rapyds run. Note that Rapyds applications typically have at least three modes, namely pre (which just writes input files for the external code), execute (which runs the external code) and post (which post processes the output). For example, DemoProject is first used in the pre and execute modes, while DemoRecipe and DemoResults are instantiated only at post mode.

Creating the DemoParameters class

As stated, we assume that the Rapyds application is called demo, and thus we create a demo.py module, and place it in the rapyds/applications folder. You will also find the demo.py file in the top level of the provided file set. Copy it to your rapyds/applications folder. As described, the first step is to define the DemoParameters class. This class contains all associated Rapyds-level input parameters for our new demo application. Its structure is as follows:

class DemoParameters(core.project.ProjectParameters,
                     application.ApplicationParameters):

    name = core.parameter_pack.create_property('name',
                                               'Your name dummy')

    dob = core.parameter_pack.create_parameter_with_rule('dob',
                                                         'Your date of birth',
                                                         rule=core.io.IsDateTime())

    date_of_birth = dob

    def __init__(self, **kwargs):
        core.project.ProjectParameters.__init__(self)
        application.ApplicationParameters.__init__(self)

        self._name = 'Unknown'
        self._dob = None

        self.update_parameters(**kwargs)

As required, the DemoParameters class inherits from ProjectParameters and ApplicationParameters, and so automatically is already aware of all parameters needed to set up a generic Application and Project.

Note

Remember that ProjectParameters collects all parameters needed to create the input file to the external codes, and ApplicationParameters collects all parameters needed to execute the application. If the ProjectParameters or ApplicationParameters class definition are unclear, revisit the discussion in Selected element relationships in the Rapyds ApplicationFramework

In addition to the inherited parameters from these two classes, two additional parameters are declared, namely dob and name. These data elements are needed to store the relevant data to this application. In addition, an alias is created for dob as date_of_birth.

Hint

These data elements or class variables are created via the create_property function from the ParameterPack class, which allow the parameters to be decorated with descriptors, verification rules, get, and set routines. In the background, the property creator sets the value of the internal data members (self._dob and self._name).

The constructor (__init__) firstly calls the constructors of the parent classes, and then sets internal data members to default values. Finally, update_parameters routine is called, which inspects the kwargs element for additional data and sets them if they match data elements in the demo class. This functionality is available from ParameterPack, as our DemoApplication class is also a ParameterPack.

Creating the DemoProject class

The next step is to create the DemoProject class, depicted below.

class DemoProject(core.project.Project,
                   core.input_template.InputTemplate):

    def __init__(self, parameters, *args, **kwargs):
        core.project.Project.__init__(self, parameters=parameters, *args, **kwargs)
        core.input_template.InputTemplate.__init__(self, tmpl_file='demo.tmpl', project=self)

        self._set_results()

This class is intended to facilitate the creation of the input files and folder structures for the calculation. If the concept of Project is somewhat unclear revisit the discussion in Selected element relationships in the Rapyds ApplicationFramework. In this case the DemoProject class inherits from the Project class and the InputTemplate.

Tip

The Project class prepares links to the Cheetah-based template mechanism for input file generation. It would be useful to read about how Cheetah templates work, and how your class structures and Cheetah templates interact. For more information, see Cheetah Engine.

You will note that the constructor of the parent InputTemplate accepts a tmpl_file argument, which is set to, in this case, demo.tmpl. This refers to the name of the primary input template (Cheetah template) which will be searched for at input creation time. Such a template, being code-dependent, is only compiled and instantiated (templates are also classes) at execution time. We will develop this template together when we assume the role of Plugin Developer (PD) a little later. For now suffice to say that a code-specific version of the demo.tmpl file should be part of the plugin of every code connected to this application.

The only further specific action taken in the constructor of this class, is the call to _set_results. Recall that the project should connect the input and the output of the project, and assist Rapyds to find the input and output files. In this case we are making sure that when the constructor is later called, that the results are attached to the class. Note that this specific _set_results routine will be overriden later when the Plugin is developed. Lets defer the discussion to then.

Creating the DemoRecipe class

The DemoRecipe class inherits from the general Recipe class, and defines a set of abstract methods (which have to be implemented later at the plugin stage). These methods will be called to fetch the desired output quantities from the external code output files. Of course at this level, the implementation is code independent, and as such there is no implementation of the methods here, but only an declaration of them.

class DemoRecipe(core.result.Recipe):

   @abc.abstractmethod
   def get_age(self) -> int:
       """
       Extract the persons age in years
       """

The class therefore has very little to discuss in terms of implementation. We are interested to fetch the age from the output file, which we intend to do with the function get_age, which itself then has to return an integer.

Creating the DemoResults class

The next step is to identify which output parameters we would like to store after execution of the external code. This is done in the DemoResults class, which inherits from the general ResultsSummary class.

class DemoResults(core.result.ResultsSummary):

    def __init__(self, parameters: DemoParameters, recipe: DemoRecipe, *args, **kwargs):

        super().__init__(parameters, recipe)

        self.name = parameters.name
        self.dob = parameters.dob
        self.age = recipe.get_age()

    def report(self, *args, **kwargs):

        core.io.message(f'{self.name} born on {self.dob} is {self.age} years old of starsign {self.starsign}')

Here we declare, in the constructor, the set of data elements which would be used as output variables in the application - in other words what output should this application store. We would like to store some of the input parameters, from the parameter pack (DemoParameters is passed to the constructor), and some results from the Recipe (DemoRecipe is also part of the signature).

For self.age we therefore indicate that it will be fetched from recipe.get_age, which we have just defined in the step before.

Creating the DemoApplication class

We have now created all helper classes for this application, and we can therefore develop the actual application class DemoApplication, which inherits from Application.

Note

Recall from Selected element relationships in the Rapyds ApplicationFramework that there are three types of applications, with the Application class used for a single application which runs a single external code.

The DemoApplication class looks as follows:

class DemoApplication(application.Application):

    project_type = DemoProject
    result_type = DemoResults

    Parameters = DemoParameters

    def create_viz_mode(self, subparsers):
        pass

We can see that the application does very little, other then connecting all the helper classes we have developed thus far to the application. In other words, we are now done with the code-independent part of the implementation, and now proceed to the part Rapyds which interacts with a specific external code - or in other words, the external code Plugin.

Defining the application main method

Finally, we need the demo.py module to defines its own main method, so that the Rapyds User can later call this application.

def main(parameters, *args, **kwargs):
    application.main(DemoApplication, parameters=parameters, *args, **kwargs)

We are now done with the Demo Rapyds application. The application is now ready to be connected to external codes via development of appropriate plugins for each external code.

Plugin Developer (PD) role

So what is a Plugin? In the Rapyds environment, a plugin is the link between the code-independent application, and the external code. The plugin has primarily two main functions:

Provision of the code-dependent set of templates, which will write the input files for the specific external code
Provision of the code-dependent recipe, which will process the output files from the external code.

You will note that in the provided fileset, there is a folder named plugin, with a DEMO folder inside it. This DEMO folder is the top-level plugin folder for the external code.

Within this parent plugin/DEMO folder, we firstly find, in this case, the external code itself - run_me.py. It is not necessary for the external code to be in the Plugin folder, and it can be generally installed anywhere on your system - we will later specify the path to the external code when we discuss the step to Register the Plugin.

Secondly, we find a folder for the demo/templates, and another for the demo/recipes. The additional demo level is not critical, but is used here since we would like the recipe to be within a python package (and thus need a level to include the __init__.py file).

Let us start by taking a look at the templates folder, which contains, in this case, two files, namely demo.tmpl (the input file template), and secondly execute.tmpl (which is the template for executing the applcation).

Defining the code templates

The input file template: demo.tmpl

The templates are based on the Cheetah Template Engine technology. Within this system, the templates themselves are a hybrid of python code and plain text. This allows a powerful set of tools for creating a dynamic input file, which references the elements of the Rapyds UML model of Rapyds Data Model.

Looking at the demo.tmpl, we find:

Hint

In the Cheetah template, all lines are considered pure text and passed directly to the input file, unless they start with a “#”, in which case they are interpreted as python code. “##” implies a comment line, and “$” is used to identify a class member or variable.

#from applications.demo import DemoProject
#from demo.recipes.my_demo import MyDemoRecipe
#extends DemoProject
#def __init__(*args,**kwargs)
#pass
#end def
#def _set_version()
#set self.__version__ = '1.0.0'
#end def
#def _set_results()
#set self.input_file_patterns = ['*.yaml']
#set self.results.recipe = MyDemoRecipe()
#set self.results.required_output_files = ['*.out']
##set self.results.optional_output_files = ['*.whatever']
#end def
## Input starts
name: $project.parameters.name
dob: $str($project.parameters.dob)

Here there are a number of important aspects to note:

The template, being a python source also, imports the relevant classes from which it requires functionality and data. Here we import DemoProject and DemoRecipe. We will discuss why in a moment.
The Template is itself a class, and extends from DemoProject. Remember that DemoProject is an InputTemplate also, and therefore we are just completing its implementation here. Note that a constructor to the template class is provided, but has not explicit functionality in this case.
Two helper routines are implemented here(set_version and set_results). The most important for our discussion is set_results, which was called in the constructor of the DemoProject class, and we would like to over-ride it here, as we now know which external code we are dealing with. Recall that set_results is the link between the input and output files (as discussed in Creating the DemoProject class), and defines the input file patterns (to find all relevant input files), the output file patterns (to find all relevant output files) and the recipe (to be able to process the output file).
We can now understand why both DemoProject and DemoRecipe were important to import here. The former is needed since our template extends from it, and the latter is needed to instantiate the recipe class and connect it to the project.
Finally, the last part of the template, is the actual input file specification for the external code. This is the most important part of the template, as it combines pure text (which is directly passed to the input file by the Cheetah template engine), with relevant data elements. Compare this text with the structure of the input file we looked at before under External code Developer (ED) role. You would note it is identical, with the exception of the dynamic nature as it now refers to data elements in the DemoParameter class.

Note

It is important to understand when this class is actually created. As it is a code-dependent part of the system, it can only be defined at run-time once the user has selected the target code for deployment. Only at that time when input generation is performed, does this specific template class get compiled and instantiated. If another external code was selected for this application, this demo.tmpl file would not even be accessed, and the demo.tmpl in another plugin would be compiled in its place.

The execution template: execute.tmpl

The execution template is always required if the application has an execution mode. In this case the execution template is relatively simple, and just calls the run_me.py python script:

#from applications.run import ExecuteTemplate
#extends ExecuteTemplate
#def __init__(*args,**kwargs)
#pass
#end def
python $project_parameters.executable $input_file

With the background you already have, this file should now be easy to interpret. It extends from ExecuteTemplate (a Rapyds class), and the only line of consequence is the execution line python $project_parameters.executable $input_file, which tells Rapyds what the command line for running the external code is. We saw how this code is called manually under External code Developer (ED) role.

Note

Here we refer to the project parameter $project_parameters.executable, which you have not encountered and we have not yet set. This well be discussed shortly, when we look at Register the Plugin.

Defining the code recipes

We are now fast approaching the end of the coupling work, and have to implement the last part of the plugin, namely the recipe for processing the output file from the external code. This file is found in the plugin/DEMO/demo/recipes folder in your file set.

We see:

from applications.demo import DemoRecipe
import yaml


class MyDemoRecipe(DemoRecipe):
    def attach(self, parameters, project_data, *files):

        with open(files[0], 'r') as f:
            self.data = yaml.safe_load(f)

    def get_age(self) -> int:
        return int(self.data['age'])

Firstly, as required, we note that we are creating a new recipe class (MyDemoRecipe), which derives from our previously developed DemoRecipe class. If you recall, we did not fully complete the work in the DemoRecipe class, as we had defined abstract methods there which would fetch the required output data and pack them into the code-independent structure.

This is what we are then left to do here - implement the get_age function, which we had declared as abstract before. Recall that an abstract function/method has to be implemented in any class which further derives from the that parent. As we now know (as we are in the plugin) with which external code we are working, this job is relatively simple.

Firstly, we override the attach method, which reads the output file - in this case this is a straight forward read of the output yaml file. Effectively we are reading a file which looks like the output file we viewed in External code Developer (ED) role.

Next, we implement the get_age method, and effectively just return the age entry of the dictionary which was read from the yaml file. This functionality is going to be applied during the so-called post mode, which would read the output file and populate the code-independent data structures.

Register the Plugin

The last step in preparing the plugin for usage, is to register the plugin with the Rapyds system, which makes Rapyds aware of the plugin, and sets the path to the external code executable. This is done by adding the following block to your Rapyds CONFIG file (whichever one you use, as you could do this in the generic Rapyds level config file, or your local one which you pass on the command line via the –config-file argument).

[DEMO]
root : 'C:\\oscar5\\oscar5\\training\\rapyds_development\\plugin\\DEMO\\demo\\templates'
executable: 'C:\\oscar5\\oscar5\\training\\rapyds_development\\plugin\\DEMO\\run_me.py'

Here we provide a Rapyds name to the external code (we call it DEMO), define the path to the templates folder, and define the executable path. This is then where the executable path, which was needed in the execute.tmpl, is defined (recall the discussion under The execution template: execute.tmpl).

Rapyds User (RU) role

We are now done with the coupling of our external run_me.py program, and can test whether the work we have done functions as expected. Firstly, we would need a Rapyds input file, which runs the demo application we developed.

You can find this file under the plugin_test folder of your fileset - it is named example_using_demo.py. Here it is:

import applications.demo as app
from core import *
import sys

param = app.DemoParameters()

param.name = 'Rian'
param.date_of_birth = '1978-11-14'
param.target_mode = 'DEMO'
param.working_directory = utilities.path_relative_to(__file__, 'TEST')
param.project_name = 'my_test'

if __name__ == '__main__':
    sys.path.insert(0, 'C:\\oscar5\\oscar5\\training\\rapyds_development\\plugin\\DEMO')
    app.main(param)

If you have at all used the OSCAR-5 package before, this file would be very familiar to you, as it follows the structure of any Rapyds input file.

We import our DemoApplication (as app), and we setup the application by setting the parameters of the DemoApplication parameter set. Here we should note that the target_mode is set to DEMO, which is the Rapyds name for our external code.

Note

See that the main method calls the main method of our demo application, and passes the parameter pack we just populated as argument. Compare that to the signature of the main method in our demo.py rapyds application. Further note that we add the top level path (to just above demo in the plugin package) to the path. This is done so that the demo package can be found, and in particular that the demo.recipes import in the demo.tmpl works. You should of course update this folder to match the path to your rapyds_development test set.

All that remains is to run our application, using the input data provided in this input file. Do this from the plugin_test:

oscar5 example_using_demo execute --force

This will generate the input to the external code, and execute it.

Hereafter you can post-process the output via:

oscar5 example_using_demo post --force

If both these two commands execute successfully, then the coupling of the external code is complete.

Assignment

Perform the following set of extension to the implementation of the age calculator and its associated interface to Rapyds. You must decide yourself which roles have to be assumed, and hence which files have to be updated, for each of the sub-steps. You can perform this on your own, or break into teams with each team member assuming a different role in the implementation, and thus work in parallel.

Implement the following:

Expand the age calculator to also take an input value in the yaml file indicating the year the user started working, calculate the number of years of work experience he/she has, and include that in the output file
Expand the Rapyds application to include the capability of interfacing with an age calculator that has this additional functionality
Update the Rapyds plug-in to access this new functionality of the age calculator
Perform a test calculation with Rapyds to verify the correctness of the new coupling feature

Final remarks

We have now performed a basic coupling of an external code, and defined the steps involved. However, many interesting features can still be added to this coupling to enrich the user experience. This includes

Adding documentation to the Rapyds calculation with the external code, so that your run produces an output document
Supplementing the input parameters used in the code input with data elements from the Rapyds data model. This might be somewhat contrived for a age calculator, as the Rapyds data model is focussed on reactor modelling, but is used extensively for reactor applications.
Applying the unit testing framework to automatically test developed code as changes are made
Adding GUI elements to the Plugin, both for input creation and results analysis.

We will cover some of these additions in the next section, named Further Application Enhancements.