Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 60 additions & 48 deletions docs/source/how-to/configuration/create-configuration.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,34 @@
# Create and Load Configuration

This guide shows how to write a pyAML configuration as a text file and load it into an `Accelerator`. The structure and syntax of the configuration are explained in detail in [Configuration Structure and Syntax](../../explanation/configuration).
This guide shows how to write a pyAML configuration and load it into an `Accelerator`. It gives recommendations for how to write it and list tools that are available to help.

```{tip}
Read [Configuration Structure and Syntax](../../explanation/configuration) which explains the concepts and ideas behind the configuration before you start.
```
The configuration can be written as a text file in YAML or JSON or as a dictionary.

```{note}
Tools are available to help writing the configuration. See [Tools That Help Writing the Configuration](../configuration/create-configuration.md#tools-that-help-writing-the-configuration) for the options.
```
Comment on lines +10 to +12

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about a page to present these tools (with subpages if needed) and having just a link to this page here?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is a good idea. My feeling is that the page is a bit too much information overflow now. I was thinking about maybe also move the parts about how to load and validate to separate pages and make this page purely about how to create it? But I didn't want to make to many changes to your PR in one go.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That page is too loaded, it's better to split it. I will have a try from your version.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think having the tools on a separate page is also good because then that page can be more dynamic while the page about how to create the config changes less frequently. My feeling is that we have some tools now but not all of them are good, some have potential for improvement whereas other should maybe just be removed after user feedback. And then we also have the new ideas in progress with templates and a tool to automatically generate a simple config from a lattice file that should be added when they are ready for user tests.


## The Rule to Remember

The configuration consists of a set of items which tells pyAML which objects to build when loading the configuration.

```{important}
Each item of the configuration names a Python class in its `class` field. **Every other field is an argument of the constructor of that class**, with the same name. When an argument is an object, its value is a nested item with its own `class` field.
Each item of the configuration names a Python class in its `class` field. **Every other field is an argument of the constructor of that class** with the same name as the field and the value to pass to the constructor. When a constructor argument is an object, its value in the configuration is a nested item with its own `class` field.
```

Writing a configuration is therefore the same as writing the Python code that creates the objects, in YAML (or JSON) instead of Python.
Writing and loading the configuration is the equivalent of writing the Python code that creates the objects yourself. The configuration just allows pyAML to create the objects for you.

## Find the Fields of a Class
## Finding the Accepted Fields

Before writing an item, look up the constructor arguments of its class. Any of these works:
Before writing an item, look up the constructor parameters of the class you want pyAML to build an object of. This can be done in several ways:

- `help()` in Python, which shows the constructor signature and describes each argument:
- Read the [API documentation](./../reference/index.md) of the class
- Use `help()` in Python since this shows the signature of the constructor and description of each parameter

For example:

```python
from pyaml.bpm.bpm import BPM
Expand All @@ -33,31 +47,33 @@ Before writing an item, look up the constructor arguments of its class. Any of t
# | ...
```

- the [API documentation](https://pyaml.readthedocs.io/en/stable/) of the class,
- the `describe()` method of the schema in the [schema registry](./use-schema-registry.ipynb).
- Use the schema registry. The `describe()` method lists the fields of a registered class with their types. See [Use the Schema Registry](../how-to/configuration/use-schema-registry.ipynb).
- Use a JSON Schema in an external tool. See [Tools That Help Writing the Configuration](../configuration/create-configuration.md#tools-that-help-writing-the-configuration) for the options.

Parameters with a default value is optional and can be left out of the configuration if you wish.

Arguments without a default value (here `name`) are required fields; the others can be left out.
## Write Configuration as a Text File

## Write the Configuration File
Here an example is shown for how to create the configuration in a YAML file. The steps are similar if using JSON. The steps below build a small but complete configuration, using the names of the [test lattice](../../tutorials/functionality/01_create_accelerator).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see why a mention of the test lattice is needed. I would remove this part

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me. I just moved the existing section and thought the link was not ideal but you are right. It is better to remove it.


Create a file, for example `accelerator.yaml`, with any text editor. The steps below build a small but complete configuration, using the names of the [test lattice](../../tutorials/functionality/01_create_accelerator).
Create a file, for example `accelerator.yaml`, with any text editor. If you want the editor to suggest the fields, you can use VS Code together with a JSON Schema. See [Use JSON Schema in VS Code](../configuration/use-vscode-json-schema.md) for instructions.

### 1. The Accelerator

The root item is the `Accelerator`. Its required arguments are `facility`, `machine` and `energy`:

```yaml
class: pyaml.accelerator.Accelerator
facility: My facility
machine: sr
facility: my_facility
machine: storage_ring
energy: 1.0e9
```

### 2. The Control Modes

Add the control modes as lists in `simulators` and `controls`. Their `name` is also the name used to access them (`accelerator.design`, `accelerator.live`).
Add the control modes as lists in `simulators` and `controls`. Their `name` is also the name used to access them, for example `accelerator.design`, `accelerator.live` etc.

A simulator needs the path to a lattice file. Use `${path:...}` so that the path is resolved relative to the [configuration root](../../explanation/configuration.md#configuration-root) and the lattice is not loaded as a configuration file:
A simulator needs the path to a lattice file. All [formats that can be loaded by pyAT](https://atcollab.github.io/at/p/api/at.load.html#module-at.load) works. If you use the JSON format, you need to use the `${path:...}` [resolver](../../explanation/configuration.md#resolvers) to avoid the lattice being loaded as if it was a configuration file.

```yaml
simulators:
Expand All @@ -66,35 +82,24 @@ simulators:
lattice: ${path:lattice.json}
```

A control system is given by the class of the bindings you use. Its arguments depend on the bindings, for example for `pyaml-cs-oa`:
A control system is given by the class of the bindings you use. The arguments depend on the bindings and catalog type you decide to use. The catalog describes how the keys used by the devices map to control-system signals. See [Control System Catalogs](../../explanation/catalog.md) for the different types of catalogs.

For example for `pyaml-cs-oa` using a dynamic catalog for TANGO:

```yaml
controls:
- class: pyaml_cs_oa.controlsystem.OphydAsyncControlSystem
name: live
backend: tango
catalog: catalog.yaml
```

The catalog describes how the keys used by the devices map to control-system signals. It follows exactly the same rule. A static catalog for `pyaml-cs-oa` looks like this (one entry per key):

```yaml
class: pyaml_cs_oa.static_catalog.StaticCatalog
entries:
- class: pyaml_cs_oa.static_catalog_entry.StaticCatalogEntry
key: AN01-AR/EM-QP/QF.01/magnetic_strength
device:
class: pyaml_cs_oa.tangoAtt.TangoAtt
attribute: AN01-AR/EM-QP/QF.01/magnetic_strength
unit: 1/m
# ... one entry for each key used in the configuration
```

See [Control System Catalogs](../../explanation/catalog.md) for the different types of catalogs. If you only want to use the simulator, you can leave out `controls` entirely.
If you only want to use the simulator, you can leave out `controls` entirely.

### 3. The Devices

Add the elements of the machine in `devices`. For a magnet, the `model` argument is an object (the magnet model, which also handles the unit conversion), so it is written as a nested item:
Add the elements of the machine in `devices`. For a magnet, the `model` argument is an object (the magnet model, which also handles the unit conversion), so it is written as a nested item.

The strings given to `physics`, `x_pos` and `y_pos` are keys looked up in the catalog of the control system.

```yaml
devices:
Expand All @@ -116,7 +121,7 @@ devices:
y_pos: AN01-AR/DG-EPOS/BPM.01/y
```

By default, the `name` of an element is also the name of the element in the lattice of the simulator. Use `lattice_names` if they differ. The strings given to `physics`, `x_pos` and `y_pos` are keys looked up in the catalog of the control system.
By default, the `name` of an element is also the name of the element in the lattice of the simulator. If you want to use a different name in pyAML, use `lattice_names` to map between pyAML and the lattice.

### 4. The Arrays

Expand All @@ -141,8 +146,8 @@ Putting it all together:

```yaml
class: pyaml.accelerator.Accelerator
facility: My facility
machine: sr
facility: my_facility
machine: storage_ring
energy: 1.0e9
simulators:
- class: pyaml.lattice.simulator.Simulator
Expand All @@ -152,7 +157,6 @@ controls:
- class: pyaml_cs_oa.controlsystem.OphydAsyncControlSystem
name: live
backend: tango
catalog: catalog.yaml
devices:
- class: pyaml.magnet.quadrupole.Quadrupole
name: QF_001
Expand Down Expand Up @@ -184,7 +188,7 @@ arrays:

## Split the Configuration into Several Files

For a real machine the configuration becomes long. When the configuration is loaded from a file, any string value ending with `.yaml`, `.yml` or `.json` is replaced by the content of that file (this is how `catalog: catalog.yaml` above is loaded). Inside a list, if the file contains a list, its items are added to the parent list. For example, move the quadrupoles to `devices/quadrupoles.yaml`:
For a real machine the configuration becomes long. When the configuration is loaded from a file, any string value ending with `.yaml`, `.yml` or `.json` is replaced by the content of that file. Inside a list, if the file contains a list, its items are added to the parent list. For example, move the quadrupoles to `devices/quadrupoles.yaml`:

```yaml
# devices/quadrupoles.yaml
Expand Down Expand Up @@ -214,7 +218,7 @@ Values can also come from environment variables with `${env:NAME}`. See [Resolve

## Use Your Own Classes

The rule is not limited to pyAML classes. Any class that can be imported can be used in the configuration, for example a magnet model specific to your facility:
The configuration is not limited to pyAML classes. Any class that can be imported can be used in the configuration, for example a magnet model specific to your facility:

```python
# my_facility/models.py
Expand All @@ -234,7 +238,7 @@ model:

## Load the Configuration

Set the configuration root, which is the directory used to resolve relative paths, then load the file with `Accelerator.load()`:
Set the [configuration root](../../explanation/configuration.md#configuration-root), which is the directory used to resolve relative paths, then load the file with `Accelerator.load()`.

```python
from pyaml.configuration import ROOT
Expand All @@ -246,9 +250,9 @@ accelerator = Accelerator.load("accelerator.yaml")
accelerator.design.magnets.get("Quadrupoles").strengths.get()
```

If the control-system bindings are not installed or you only want to use the simulator, add `ignore_external=True`. The `controls` section is then skipped.
If the control-system bindings are not installed or you only want to use the simulator, add `ignore_external=True` and the `controls` section is skipped without having to remove it from the configuration.

The configuration can also be given as a nested dictionary with `Accelerator.from_dict()`, following the same rule:
The configuration can also be loaded as a nested dictionary with `Accelerator.from_dict()`. This also allows to write the configuration directly as a dictionary instead of a text file if you prefer.

```python
import yaml
Expand All @@ -263,7 +267,9 @@ See the API documentation for the [Accelerator](https://pyaml.readthedocs.io/en/

## Validate the Configuration

Each item is checked when it is created: a missing required field or an unknown field raises a `PyAMLConfigException` naming the class and the field. The whole configuration can also be validated before anything is created, which gives all errors at once:
If the classes you use have enabled validation during object creation (this is the default for all common pyAML classes), the configuration will be validated as part of creating the objects: a missing required field or an unknown field raises a `PyAMLConfigException` naming the class and the field.

The whole configuration can also be validated before anything is created.

```python
from pyaml.validation import SchemaRegistry
Expand All @@ -276,12 +282,18 @@ The configuration can also be validated without loading it, which is useful if y

## Tools That Help Writing the Configuration

Writing the file by hand is often the simplest way to start, but tools based on a [JSON Schema](../../explanation/schema_and_validation.md) can suggest the available fields and check their types while you write:
There are tools available to help to write and modify the configuration. They can suggest the fields and check their types while you write. The tools are under development and testing so new or other tools might be available in the future based on user feedback.

Some of the tools are based on a [JSON Schema](https://json-schema.org). For information about JSON Schemas and how to generate them, see [Configuration Schemas and Validation](../../explanation/schema_and_validation.md) and [Generate JSON Schemas](./generate-json-schema.ipynb).

Currently these tools are available:

- [Use ConfigurationSchema](./use-configuration-schema.ipynb) objects to create the configuration in Python and export it as a dictionary or text file. This allows to program the configuration.

- [Use a JSON Schema in VS Code](./use-vscode-json-schema.md)
- [Use the MetaConfigurator](./use-meta-configurator.md), a form-based editor in the browser
- [Use ConfigurationSchema](./use-configuration-schema.ipynb) objects to create the configuration in Python and export it as a dictionary or text file

AI coding assistants can also help: supply for example a lattice file, a description of the naming conventions of your control system, and the JSON Schema of the pyAML configuration.
- [Use the MetaConfigurator](./use-meta-configurator.md), a form-based editor in the browser

For information about JSON Schemas and how to generate them, see [Configuration Schemas and Validation](../../explanation/schema_and_validation.md) and [Generate JSON Schemas](./generate-json-schema.ipynb).
```{tip}
AI coding assistants can also help: supply for example a lattice file, a description of the naming conventions of your control system, and the JSON Schema of the pyAML configuration and ask it to write the configuration for you.
```
Loading