mip: Better support for natmod installations

[jonnor]

Summary

It is possible to install .mpy files using mip, including those that have native code (native modules). But currently one must specify the exact file URL for such files - which for native modules means knowing the correct ABI version and architecture version - something users rarely know offhand. This makes it difficult to document how to install a natmod, and makes it difficult to have scripts that automatically install the correct version.

This change allows using a single URL to specify a native module package - regardless of the ABI/architecture. mip will then lookup the correct hardware architecture and MicroPython ABI version, and substitute this into the URL before doing the download/install.

The use of variables that are substituted allows some flexibility wrt different locations/layouts for .mpy files. This is useful because there is currently no standard (to my knowledge) for how to lay out. So this should be immediately useful to existing distributions of mpy natmods.

Note: for import foo to work the .mpy file, the file itself must be named foo.mpy. Thus the MPY_ARCH and MPY_VERSION markers can only meaningfully be in directory part of a path.

Testing

Tested on RP2 port using a Pico W. Example code follows.

Installs from https://github.com/emlearn/emlearn-micropython project - should work on any hardware supported by emlearn-micropython. By removing the network part, should also work on Unix port.

import network
import mip
import time

# NOTE: must fill this in
WIFI_SSID = None
WIFI_PASSWORD = None

print("Connecting to WiFi", end="")
sta_if = network.WLAN(network.STA_IF)
sta_if.active(True)
sta_if.connect(WIFI_SSID, WIFI_PASSWORD)
while not sta_if.isconnected():
  print(".", end="")
  time.sleep(0.1)
print(" Connected!")

u = 'https://emlearn.github.io/emlearn-micropython/builds/latest/{MPY_ARCH}_{MPY_VERSION}/emlearn_iir.mpy'
mip.install(u)

import emlearn_iir
print(dir(emlearn_iir))

Trade-offs and Alternatives

Small increase in code size of the mip package.
It adds nothing to the core runtime of MicroPython. An alternative implementation that would do versioned/architecture handling at import time would likely increase size of the code.

Generative AI

I did not use generative AI tools when creating this PR.

[Josverl]

I see the need for a solution to this problem, and i like the simplicity of the solution.

To me the MPY_ARCH and MPY_VERSION placeholders are a bit magical.

Apart from some bikeshedding on the prefix , MP / MPY ,
I would expect these to be imported from sys as a string, if only to allow a user to see and validate the actual fetch url.

Possibly that can also be solved by a verbose parameter to mip that makes it show the Uri mip is trying to fetch from.

[jonnor]

Yeah they are somewhat magical - I added a minimum of docs in micropython/micropython#19478 - that can also be expanded. There actually is example code in the docs already for getting this arch/version stuff out - https://docs.micropython.org/en/latest/reference/mpyfiles.html#versioning-and-compatibility-of-mpy-files

I have no opinion on the prefix - will adopt whatever the maintainers think appropriate. Currently there is a mix of MP and MPY in codebase - where the MPY is mostly what is used around .mpy files, like in mpy_ld.py etc.

Seeing the exact URL would be useful. There is print of "downloading XYZ" from mip.install now - but it actually happens before URL rewriting - so will show the variables. This affects not only the expansion of these variables, but also github: prefix expansion.
This print could be moved further down, or another print added with the expanded form. If this is a blocker/desired as part of this MR, I can make that change.