Python Client Presentatoin

aeonlib.png

codecompletion.png

frustrated.jpg

jsonhell.png

lco-aeonlib.md

title	sub_title	author
Pure-Python Client Libraries	Making Python more like Java	Austin Riba

Aeonlib

A suite of modules to enable TDA/MMA observations. Like a client library for every observatory.

Starting with LCO, we can design a client library that is intuitive to use.

LCO API

How do clients currently interact with the OCS/LCO?

1. Simply write a small JSON object to describe your observation request (or copy and paste it from the portal):

LCO API

2. Very carefully make modifications.

{
    "operator": "SINGLE",
    "observation_type": "NOMRAL",
    "requests": [
        {
            "acceptability_threshold": 90,
            "configuration_repeats": 1,
            "optimization_type": "TIME",

(Can you spot the mistake?)

LCO API

3. Call requests.post() to the observation portal's validation endpoint repeatedly until all mistakes have been eliminated.

LCO API

Pros:

• Language agnostic
• Flexible
• Everyone knows JSON

Cons:

• Not discoverable
• Nearly impossible to write offline, must either constantly reference docs or copy and paste
• Some things are tedius:
  • Units
  • Exact Strings
  • Lists
• No indication that your request is invalid until it is possibly too late.
  • Oops missed a TOO because I accidentlally mispelled an instrument name and was too rushed to check with validate()

Python Types

Don't do anything besides look pretty without tooling.

class RequestGroup:
    name: Annotated[str, StringConstraints(max_length=50)]
    proposal: str
    ipp_value: NonNegativeFloat

Pydantic

• Pydantic is the most widely used data validation library for Python.
• It plays well with both your IDE and your brain.
• We can use it to build a client library that alleviates some of the pain of plain JSON/python dictionaries.

class RequestGroup(BaseModel):
    name: Annotated[str, StringConstraints(max_length=50)]
    proposal: str
    ipp_value: NonNegativeFloat

assert RequestGroup(name="I'm Python", proposal="LCO123", ipp_value=100.0)

👍

Pydantic

assert RequestGroup(name="I'm Python", proposal="LCO123", ipp_value=-20)

👎

ValidationError: 1 validation error for RequestGroup
ipp_value
  Input should be greater than or equal to 0 [type=greater_than_equal, input_value=-20, input_type=int]
    For further information visit https://errors.pydantic.dev/2.11/v/greater_than_equal

Pydantic - JSON

RequestGroup(name="I'm Python", proposal="LCO123", ipp_value=100.0).model_dump_json()

{"name":"I'm Python","proposal":"LCO123","ipp_value":100.0}

Pydantic - JSON

RequestGroup.model_validate_json('{"name":"I\'m Python","proposal":"LCO123","ipp_value":100.0}')

RequestGroup(name="I'm Python", proposal='LCO123', ipp_value=100.0)

Pydantic + Aeonlib = ♥️

• Create Pydantic models to represent the request language!
• Write requests in Python instead of JSON!
• (Some) Validation without network requests!
• IDE Integration! Code completion! Serialize to and from data formats!

But there's a problem...

All the instruments have different parameters!!!

Out of Band information

The current solution is to accept all possible values for every field and then validate them using ConfigDB.

Deeply integrated clients (e.g. observation portal frontend) can leverage the /api/instruments OCS endpoint to build a UI that validates input a little more.

{
  "instrument_config": {
    "filter_options": ["OIII", "SII", "Astrodon-Exo", "I", "R", "U", "w", "Y"]}
  }
}

Works great for a webapp but not great if you are writing a script or program and want to interact with the LCO API.

Alternative: Mass Duplication

Instead of accepting all possible values for a field, create a "schema" per-instrument that only accepts it's own subset of values.

{
  "Lco0M4ScicamQhy600OConfig": {
    "filter_options": ["OIII", "SII", "Astrodon-Exo"]}
  }
}

Downsides:

• Code duplication - instruments have a lot in common
• Painful if not impossible to maintain.
• Constantly out of date.

The Solution: Code Generation

• Generate Pydantic schemas using the /api/instruments observation portal endpoint.
• 5 Pydantic objects per instrument - deeply nested :(
• Defined within a Jinja Template
• 600 + lines of highly repetitive, machine generated code for your pleasure.

curl https://observe.lco.global/api/instruments/ | codegen/lco/generator.py > src/aeonlib/ocs/lco/instruments.py

Rule of Generation: Avoid hand-hacking: write programs to write programs when you can.

-- Eric Raymond, Unix Philosophy, Rule #14

MUSCAT Example

class Lco2M0ScicamMuscatOpticalElements(BaseModel):
    narrowband_g_position: Literal["out", "in"]
    narrowband_r_position: Literal["out", "in"]
    narrowband_i_position: Literal["out", "in"]
    narrowband_z_position: Literal["out", "in"]


class Lco2M0ScicamMuscatGuidingConfig(BaseModel):
    mode: Literal["ON", "OFF"]
    optional: bool
    """Whether the guiding is optional or not"""
    exposure_time: Annotated[int, NonNegativeInt, Le(120)] | None = None
    """Guiding exposure time"""
    extra_params: dict[Any, Any] = {}


class Lco2M0ScicamMuscatAcquisitionConfig(BaseModel):
    mode: Literal["OFF"]
    exposure_time: Annotated[int, NonNegativeInt, Le(60)] | None = None
    """Acquisition exposure time"""
    extra_params: dict[Any, Any] = {}


class Lco2M0ScicamMuscatConfig(BaseModel):
    exposure_count: PositiveInt
    """The number of exposures to take. This field must be set to a value greater than 0"""
    exposure_time: NonNegativeInt
    """ Exposure time in seconds"""
    mode: Literal["MUSCAT_SLOW", "MUSCAT_FAST"]
    rois: list[Roi] | None = None
    extra_params: dict[Any, Any] = {}
    optical_elements: Lco2M0ScicamMuscatOpticalElements


class Lco2M0ScicamMuscat(BaseModel):
    type: Literal["EXPOSE", "REPEAT_EXPOSE", "BIAS", "DARK", "STANDARD", "SCRIPT", "AUTO_FOCUS", "ENGINEERING", "SKY_FLAT"]
    instrument_type: Literal["2M0-SCICAM-MUSCAT"] = "2M0-SCICAM-MUSCAT"
    repeat_duration: NonNegativeInt | None = None
    extra_params: dict[Any, Any] = {}
    instrument_configs: list[Lco2M0ScicamMuscatConfig] = []
    acquisition_config: Lco2M0ScicamMuscatAcquisitionConfig
    guiding_config: Lco2M0ScicamMuscatGuidingConfig
    target: SiderealTarget | NonSiderealTarget
    constraints: Constraints

    config_class = Lco2M0ScicamMuscatConfig
    guiding_config_class = Lco2M0ScicamMuscatGuidingConfig
    acquisition_config_class = Lco2M0ScicamMuscatAcquisitionConfig
    optical_elements_class = Lco2M0ScicamMuscatOpticalElements

A good compromise?

Good stuff:

• Prevents us from pretending instruments on the network are generic, because they aren't.
• Code-gen allows definition to stay relatively up to date.
• JSON escape hatch if you need it.

Bad stuff:

• Can be verbose.
• Lots of library updates.
• extra_data with cerberus validaiton schemas not handled (yet)

whyyy.jpg