3. Test and validate your component library


As your team’s library of devices and circuits evolves and is validated, it is important to maintain the quality of the library. To that end, you can add regression tests to your library. With Luceda IP Manager, you can test the layout, netlist and simulation model of the devices and circuits in your team’s library for changes. The devices and circuits can be tested against known-good reference files. Other tests can be added as well, such as detailed tests of the circuit model.

Such tests should be run regularly. They should definitely be run whenever you make any change to the library. Luceda IP Manager uses the standard Python test framework pytest which provides an excellent platform to make this part of your team’s standard workflow.

Introduction to IP Manager

Changes can be desired or undesired:

  • A desired change can be, for instance, a parameter update of a device after testing reveals a value for that parameter that gives more optimal performance. After making the change, the tests will catch such a change, and subsequently, the test itself can be updated to reflect the new desired state.

  • An undesired change can, for instance, be an unintended change in an algorithm inside a PCell in the library. The tests will fail after such a change is made. The author of the algorithm can then go back and fix the algorithm, after which the tests should pass again.

Changes can also originate from elsewhere. In particular, the PDK on which your team’s library is based may have been changed with a difference in layout or models of devices in your library as a result. By applying and consistently running regression tests on your library, you can therefore significantly reduce the risk for errors and increase the quality of your team’s library.

In this tutorial, we show how to equip the library pteam_library_si_fab that was developed in Develop and distribute your component library with tests based on Luceda IP Manager.

Additional resources

3.1. Configuring the library for testing

In order to add tests to pteam_library_sifab, we first extend its folder structure with a tests folder with a few essential files:

pteam_library_si_fab
├── building
│   └── build_pteam_library_si_fab.py
├── ipkiss
│   └── pteam_library_si_fab
│       ├── components
│       ├── technology
│       ├── __init__.py
│       └── all.py
└── tests
    ├── conftest.py
    ├── generate_reffiles.py
    └── pytest.ini
  • conftest.py contains predefined data that can be used by each test.

  • generate_reffiles.py is a script to generate the known-good reference files for the tested components.

  • pytest.ini contains configuration settings for pytest.

In conftest.py, we specify the PDK technology to be loaded for testing. Our library is based on si_fab so we use the si_fab PDK technology:

luceda-academy/libraries/pteam_library_si_fab/tests/conftest.py
# pytest configuration file
import pytest


# import the technology used for testing this library for the full test session


@pytest.fixture(scope="session", autouse=True)
def TECH():
    from si_fab.all import TECH

    return TECH

In pytest.ini, we specify that we want to have an HTML report. The pytest-html plugin of pytest is used for that purpose. This all comes pre-installed with Luceda IPKISS.

luceda-academy/libraries/pteam_library_si_fab/tests/pytest.ini
[pytest]
addopts = '--html=report.html' '--self-contained-html'

Finally, generate_reffiles.py contains a single function call to generate the known-good reference files for the tests that we will add in the next step.

luceda-academy/libraries/pteam_library_si_fab/tests/generate_reffiles.py
# Regenerates the reference files for the tests in this directory.
# Be careful when using this function: it will overwrite all reference files if they already exist.
# When the reference files already exist, and tests are failing, make sure to compare the generated files
# against the reference files to spot for (un)expected changes.

from ip_manager.testing import generate_reference_files

generate_reference_files()

3.2. Adding component reference tests to the library

Now that the basic configuration has been made, we can start to equip our library with tests. We will add regression tests of the LayoutView, NetlistView and CircuitModelView of the Mux2Heated and SplitterTree components in pteam_library_si_fab.

Tests are defined in Python script files that have a name starting with test_. Pytest will automatically recognize those and scan them for tests that need to be run.

It is advised to organize the tests for different components or families of components in separate test files. In this case, we test the lattice filter and the splitter tree in separate files:

pteam_library_si_fab
├── building
│   └── build_pteam_library_si_fab.py
├── ipkiss
│   └── pteam_library_si_fab
│       ├── components
│       ├── technology
│       ├── __init__.py
│       └── all.py
└── tests
    ├── conftest.py
    ├── generate_reffiles.py
    ├── pytest.ini
    ├── test_mux2.py
    └── test_splittertree.py

In test_mux2.py, we define a test for the Mux2Heated PCell that will test several aspects of the cell against known-good reference files:

luceda-academy/libraries/pteam_library_si_fab/tests/test_mux2.py
from ip_manager.testing import Compare, ComponentReferenceTest
import pytest
import numpy


@pytest.mark.comparisons([Compare.LayoutToGds, Compare.LayoutToXML, Compare.NetlistToXML, Compare.SMatrix])
class Test_Mux2Heated(ComponentReferenceTest):
    @pytest.fixture
    def component(self):
        from pteam_library_si_fab.all import Mux2Heated

        return Mux2Heated(name="Mux2HeatedDefault")

    @pytest.fixture
    def wavelengths(self):
        return numpy.linspace(1.54, 1.56, 200)

Let’s analyze this script:

  • First, we import ComponentReferenceTest and Compare from the ip_manager.testing package. ComponentReferenceTest is a class that we will use to define a test based on known-good reference files for a component. Compare is a Python enumeration of possible comparisons.

  • We also import the pytest and numpy packages.

  • Then we define our test by means of a class that starts with Test_. Pytest will automatically recognize this as a test. By inheriting from ComponentReferenceTest, we get all the test functionality provided by Luceda IP Manager.

  • The class is marked with the tests that we want to run using @pytest.mark.comparisons which takes a list of Compare items:

    • Compare.LayoutToGds compares the LayoutView to a known-good GDSII file. The benefit of this is that GDSII is what you will use for tape-out.

    • Compare.LayoutToXML compares the LayoutView to a known-good XML file. The benefit of this is that it includes detailed information about the optical and electrical ports.

    • Compare.NetlistToXML compares the NetlistView to a known-good XML file. This way you can be sure that the logical interface of your component is correctly defined as well.

    • Compare.SMatrix compares the S-parameter matrix generated by the CircuitModelView to a known-good S-matrix data file.

  • Inside the class we define two methods. Both methods are decorated with @pytest.fixture which instructs pytest that this is data that needs to be generated before running the test.

    • component(self) defines the Luceda IPKISS PCell object which we want to test. In this case, we load Mux2Heated from pteam_library_si_fab.all and instantiate it. Note that we assign a specific name to the component. IP Manager tests should not rely on the automatic cell name generation of IPKISS, because pytest tests are not guaranteed to run in a specific order. When relying on automatically generated names, tests could fail because of differences in test execution order.

    • wavelengths(self) returns an array of wavelengths for which Compare.SMatrix will calculate and compare the S-matrix.

The test for SplitterTree in test_splittertree.py follows exactly the same pattern.

3.3. Generating and checking the known-good reference files

Before we can run these tests, we’ll need to obtain the known-good reference files first and check that they are correct.

From your IDE (PyCharm), simply execute the generate_reffiles.py script. The result is a new subfolder ref which contains 4 files for each of the components corresponding to the 4 comparisons defined above.

pteam_library_si_fab
├── building
├── ipkiss
└── tests
    └── ref
        ├── Test_Mux2Heated_caphe.smat
        ├── Test_Mux2Heated_layout.xml
        ├── Test_Mux2Heated_layout_nofilter.gds
        ├── Test_Mux2Heated_netlist.xml
        ├── Test_SplitterTree_caphe.smat
        ├── Test_SplitterTree_layout.xml
        ├── Test_SplitterTree_layout_nofilter.gds
        └── Test_SplitterTree_netlist.xml

If you open the _layout.xml file, you can see that it contains a full hierarchical representation of the layout in XML format and check that the properties of the ports are correct:

Excerpt of the known-good layout XML reference file for Mux2Heated.
<IPKISSLIBRARY name="MUX2HEATED_1">
<STRUCTURE name="MUX2HEATED_1">
<ELECTRICALPORT layer="PPLAYER M1-DRW" name="ht_in0" position="(14.520, 1.325)" />
<ELECTRICALPORT layer="PPLAYER M1-DRW" name="ht_out1" position="(96.576, 1.325)" />
<ELECTRICALPORT layer="PPLAYER M1-DRW" name="ht_out3" position="(207.419, -1.325)" />
<PORT angle="0.00" name="out1" position="(227.069, -1.325)" trace_template="SI_FAB_WIRE_WGTEMPLATE_1" />
<ELECTRICALPORT layer="PPLAYER M1-DRW" name="ht_in3" position="(177.419, -1.325)" />
<ELECTRICALPORT layer="PPLAYER M1-DRW" name="ht_in2" position="(118.379, -1.325)" />
<ELECTRICALPORT layer="PPLAYER M1-DRW" name="ht_out0" position="(44.520, 1.325)" />
<PORT angle="180.00" name="in1" position="(-9.520, -1.325)" trace_template="SI_FAB_WIRE_WGTEMPLATE_3" />

If you open the _layout_nofilter.gds file in KLayout, you can check in detail whether the component is correctly implemented:

../../../_images/ipmanager_mux2heated_gdsref.png

Known-good GDSII reference file for Mux2Heated.

3.4. Running the tests

Once the reference files are generated, we are ready to run the tests for the first time. In order to conveniently do that from PyCharm, first make sure that pytest is used as the default test runner. In the File>Settings menu, go to Tools and then Python Integrated Tools. In the Default test runner field, select pytest.

../../../_images/ipmanager_pytest_runner.png

Set pytest as the default test runner in PyCharm.

The tests can now be run from PyCharm by right-clicking on the test_mux2.py file and selecting Run `pytest in test_mux2..`. PyCharm shows the test results in a separate pane:

../../../_images/ipmanager_test_results.png

Test pane in PyCharm with test results.

You can also run tests for the whole library by selecting the tests folder and selecting `` Run pytest in tests`` from the context menu.

3.5. Updating tests after a desired change

Imagine that after experimental testing of the Mux2Heated component, it turns out that it works better with a slightly different first power coupling parameter:

First value of the power_couplings updated from 0.5 to 0.49.
 class Mux2Heated(MZILatticeFilterHeated):
 """Two-way wavelength demultiplexer with a passband of 50% of the FSR at the center_wavelength.
 """
 _name_prefix = "MUX2HEATED"

 power_couplings = i3.LockedProperty(default=[0.48, 0.13, 0.12, 0.5, 0.25])

 ...

When we run the tests after making this change, we see that 3 out of the 4 tests fail:

../../../_images/ipmanager_tests_failed.png

Tests fail after a known change.

The netlist test is still passing since we did not change that, but the layout and S-matrix have changed due to the parameter change. Hence, the new layout and model are not equal to the reference files anymore.

For each test, a temporary file was saved in the same ref folder, which reflects the new version. These temporary files have almost the same name but an extra .tmp extension is added:

../../../_images/ipmanager_test_tmpfiles.png

Temporary (new) files saved along the reference files.

We can now, for instance, overlay the two GDSII files in KLayout and check that the .tmp.gds file correctly reflects the new state:

../../../_images/ipmanager_compare_gds.png

Overlay of the reference and temporary GDSII file (coloring of layers changed to see the difference).

After you validate that the .tmp file is the new known-good file, just replace the old reference file with the .tmp file by removing the old file and renaming the .tmp file.

3.6. Where to go from here

The tests that we have added here test the components with their default parameters and only test the two high-level PCells in the library. Tests can be extended to cover also the other supporting code in pteam_library_si_fab or to test the PCells across a range of parameters. See Getting started: increasing test coverage and Advanced features and options for in-depth guidelines.