Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
b7bfc5f
Initial implementation
jholveck Feb 24, 2026
796d60c
Merge branch 'main' into feat-array-library-methods
jholveck Feb 24, 2026
8acdf96
Fix Python 3.9 compatibility
jholveck Feb 26, 2026
6e22c78
Merge branch 'main' into feat-array-library-methods
jholveck Jun 2, 2026
70f89ed
Update to 11.0 conventions
jholveck Jun 2, 2026
689c52b
Merge branch 'main' into feat-array-library-methods
jholveck Jun 24, 2026
cd2ce58
Fix an inadvertent change
jholveck Jun 24, 2026
37238be
Docstring cleanups, and widen TensorFlow dtype support a bit
jholveck Jun 25, 2026
1f7a78f
Merge branch 'main' into feat-array-library-methods
jholveck Jun 26, 2026
1aa9430
Doc improvements
jholveck Jun 26, 2026
5d99377
Merge branch 'main' into feat-array-library-methods
jholveck Jun 26, 2026
9afa3a2
Add some introductory documentation about grab
jholveck Jun 26, 2026
3a693f5
Small improvements
jholveck Jun 26, 2026
e2c3e67
Update cat-detector to use to_torch
jholveck Jun 27, 2026
5bbbcc0
Remove a TODO that I did
jholveck Jun 27, 2026
1c84624
Add more testing for the third-party array exports.
jholveck Jun 28, 2026
106b20b
Add better support for non-CPU devices in to_torch and to_tensorflow.
jholveck Jun 28, 2026
706ea74
You actually have to call functions for them to work. :-/
jholveck Jun 28, 2026
38c50b0
Minor fixes from AI review
jholveck Jun 28, 2026
672e0f0
Improve to_torch performance on the CPU
jholveck Jun 28, 2026
a914e4a
Don't use a non-blocking copy.
jholveck Jun 28, 2026
43d5b07
Speedups for to_tensorflow
jholveck Jun 30, 2026
f8566b3
Merge branch 'main' into feat-array-library-methods
jholveck Jun 30, 2026
13e1366
Add release notes
jholveck Jun 30, 2026
8f0060c
Document primary_monitor in the usage section
jholveck Jun 30, 2026
a2e1e8b
Add a 10-minute timeout to tests
jholveck Jun 30, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,13 @@ jobs:
matrix:
os:
- emoji: 🐧
name: linux
runs-on: [ubuntu-latest]
- emoji: 🍎
name: macos
runs-on: [macos-latest]
- emoji: 🪟
name: windows
runs-on: [windows-latest]
python:
- name: CPython 3.10
Expand All @@ -84,11 +87,32 @@ jobs:
run: |
python -m pip install -U pip
python -m pip install -e '.[dev,tests]'

# We don't include PyTorch or TensorFlow in the dependencies. This is partly because they are large and not
# needed for most users. But it's mostly because the right way to install them depends a lot on the user's
# needs (such as GPU support) and OS.

# The PyPI versions of PyTorch for Linux include CUDA for Linux, so we use an index with a CPU-only version there.
# For the other OSs, the PyPI versions are CPU-only.
- name: Install PyTorch (Linux CPU)
if: matrix.os.name == 'linux'
run: python -m pip install torch --index-url https://download.pytorch.org/whl/cpu
- name: Install PyTorch (macOS/Windows)
if: matrix.os.name != 'linux'
run: python -m pip install torch
# TensorFlow stable (as of this June 2026) only supports up to Python 3.12; see
# https://www.tensorflow.org/install/pip#software_requirements
- name: Install TensorFlow (Python 3.10-3.12)
if: contains(fromJSON('["3.10", "3.11", "3.12"]'), matrix.python.runs-on)
run: python -m pip install tensorflow

- name: Tests (GNU/Linux)
if: matrix.os.emoji == '🐧'
timeout-minutes: 10
if: matrix.os.name == 'linux'
run: xvfb-run python -m pytest
- name: Tests (macOS, Windows)
if: matrix.os.emoji != '🐧'
timeout-minutes: 10
if: matrix.os.name != 'linux'
run: python -m pytest

automerge:
Expand Down
43 changes: 2 additions & 41 deletions demos/cat-detector.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,26 +128,6 @@
MIN_AREA_FRAC = 0.001


# This function is here for illustrative purposes: the demo doesn't currently call it, but there's a commented-out
# line in the main loop that shows how you might use it.
def screenshot_to_tensor(sct_img: mss.ScreenShot, device: str | torch.device) -> torch.Tensor:
"""Convert an MSS ScreenShot to a CHW PyTorch tensor."""

# Get a 1d tensor of BGRA values. PyTorch will issue a warning at this step: the ScreenShot's bgra object is
# read-only, but PyTorch doesn't support read-only tensors. However, this is harmless in our case: we'll end up
# copying the data anyway.
img = torch.frombuffer(sct_img.bgra, dtype=torch.uint8)
# Bring everything to the desired device. This is still just a linear buffer of BGRA bytes.
img = img.to(device)
# The next two steps will all just create views of the original tensor, without copying the data.
img = img.view(sct_img.height, sct_img.width, 4) # Interpret as BGRA HWC
img = img.permute(2, 0, 1) # Permute the axes: BGRA CHW
# This final step will create a copy. Copying the data is required to reorder the channels. This also has the
# advantage of also making the tensor contiguous, for more efficient access.
img = img[[2, 1, 0], ...] # Reorder the channels: RGB CHW
return img


def top_unique_labels(labels: torch.Tensor, scores: torch.Tensor) -> torch.Tensor:
"""Return the unique labels, ordered by descending score.

Expand Down Expand Up @@ -278,27 +258,8 @@ def main() -> None:
# Grab the screenshot.
sct_img = sct.grab(monitor)

# We transfer the image from MSS to PyTorch via a Pillow Image. Faster approaches exist (see
# screenshot_to_tensor), but PIL is more readable. The bulk of the time in this program is spent doing
# the AI work, so we just use the most convenient mechanism.
img = Image.frombuffer("RGB", sct_img.size, sct_img.bgra, "raw", "BGRX", 0, 1)

# We explicitly convert it to a tensor here, even though Torchvision can also convert it in the preprocess
# step. This is so that we send it to the GPU before we do the preprocessing: PIL Images are always on
# the CPU, and doing the preprocessing on the GPU is much faster.
#
# Most image APIs, including MSS, use an array layout of [height, width, channels]. In MSS, the
# ScreenShot.bgra data follows this convention, even though it's exposed as a flat bytes object.
#
# In contrast, most AI frameworks expect images in [channels, height, width] order. The pil_to_tensor
# helper performs this rearrangement for us.
img_tensor = torchvision.transforms.v2.functional.pil_to_tensor(img).to(device)

# An alternative to using PIL is shown in screenshot_to_tensor. In one test, this saves about 20 ms per
# frame if using a GPU, and about 200 ms if using a CPU. This would replace the "img=" and "img_tensor="
# lines above.
#
#img_tensor = screenshot_to_tensor(sct_img, device)
# Transfer the image from MSS to PyTorch.
img_tensor = sct_img.to_torch(device=device)

# Do the preprocessing stages that the trained model expects; see the comment where we define preprocess.
# The traditional name for inputs to a neural net is "x", because AI programmers aren't terribly
Expand Down
2 changes: 1 addition & 1 deletion demos/tinytv-stream-simple.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ def main() -> None:
# The next step is to resize the image to fit the TinyTV's screen. There's a great image
# manipulation library called PIL, or Pillow, that can do that. Let's transfer the raw pixels in
# the ScreenShot object into a PIL Image.
original_image = Image.frombuffer("RGB", screenshot.size, screenshot.bgra, "raw", "BGRX", 0, 1)
original_image = screenshot.to_pil("RGB")

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's so much cleaner that way!


# Now, we can resize it. The resize method may stretch the image to make it match the TinyTV's
# screen; the advanced demo gives other options. Using a reducing gap is optional, but speeds up
Expand Down
2 changes: 1 addition & 1 deletion demos/tinytv-stream.py
Original file line number Diff line number Diff line change
Expand Up @@ -348,7 +348,7 @@ def capture_image(

while True:
sct_img = sct.grab(rect)
pil_img = Image.frombuffer("RGB", sct_img.size, sct_img.bgra, "raw", "BGRX", 0, 1)
pil_img = sct_img.to_pil("RGB")
yield pil_img


Expand Down
4 changes: 2 additions & 2 deletions demos/video-capture-simple.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ def main() -> None:
monitor = sct.monitors[1]

# Because of how H.264 video stores color information, libx264 requires the video size to be a multiple of
# two.
# two.
monitor["width"] = (monitor["width"] // 2) * 2
monitor["height"] = (monitor["height"] // 2) * 2

Expand Down Expand Up @@ -129,7 +129,7 @@ def main() -> None:
# use PIL: you can create an Image from the screenshot, and create a VideoFrame from that. That said,
# if you want to boost the fps rate by about 50%, check out the full demo, and search for
# from_numpy_buffer.
img = Image.frombuffer("RGB", screenshot.size, screenshot.bgra, "raw", "BGRX", 0, 1)
img = screenshot.to_pil("RGB")
frame = av.VideoFrame.from_image(img)

# When we encode frames, we get back a list of packets. Often, we'll get no packets at first: the
Expand Down
9 changes: 4 additions & 5 deletions demos/video-capture.py
Original file line number Diff line number Diff line change
Expand Up @@ -227,20 +227,19 @@ def video_process(
# Many Python objects expose their underlying memory via the "buffer protocol". A buffer is just a view of
# raw bytes that other libraries can interpret without copying.
#
# Common buffer objects include: `bytes`, `bytearray`, `memoryview`, and `array.array`. `screenshot.bgra` is
# also a buffer (currently it is a `bytes` object, though that detail may change in the future).
# Common buffer objects include: `bytes`, `bytearray`, `memoryview`, and `array.array`. A ScreenShot object
# stores its pixel data in a buffer.
#
# Minimum-copy path: ScreenShot -> NumPy -> VideoFrame
# ----------------------------------------------------
#
# `np.frombuffer()` creates an ndarray *view* of an existing buffer (no copy). Reshaping also stays as a
# view.
# view. `ScreenShot.to_numpy()` uses the same approach internally and keeps the zero-copy behavior.
#
# PyAV's `VideoFrame.from_ndarray()` always copies the data into a new frame-owned buffer. For this demo we
# use the undocumented `VideoFrame.from_numpy_buffer()`, which creates a `VideoFrame` that shares memory with
# the ndarray.
ndarray = np.frombuffer(screenshot.bgra, dtype=np.uint8)
ndarray = ndarray.reshape(screenshot.height, screenshot.width, 4)
ndarray = screenshot.to_numpy(channels="BGRA")
frame = av.VideoFrame.from_numpy_buffer(ndarray, format="bgra")

# Set the PTS and time base for the frame.
Expand Down
13 changes: 11 additions & 2 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,5 +92,14 @@

# ----------------------------------------------

# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {"python": ("https://docs.python.org/3", None)}
intersphinx_mapping = {
"numpy": ("https://numpy.org/doc/stable/", None),
"pil": ("https://pillow.readthedocs.io/en/stable/", None),
"python": ("https://docs.python.org/3", None),
# TensorFlow doesn't have an official InterSphinx mapping, but there is a community one.
"tensorflow": (
"https://www.tensorflow.org/api_docs/python",
"https://github.com/GPflow/tensorflow-intersphinx/raw/master/tf2_py_objects.inv",
),
"torch": ("https://docs.pytorch.org/docs/stable/", None),
}
13 changes: 9 additions & 4 deletions docs/source/examples.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _examples:

========
Examples
========
Expand Down Expand Up @@ -122,8 +124,8 @@ falling back to XGetImage:
PIL
===

You can use the Python Image Library (aka Pillow) to do whatever you want with raw pixels.
This is an example using `frombuffer() <http://pillow.readthedocs.io/en/latest/reference/Image.html#PIL.Image.frombuffer>`_:
You can use the Python Image Library (aka Pillow) to do whatever you want with raw pixels. This is an example using
:py:meth:`mss.ScreenShot.to_pil`:

.. literalinclude:: examples/pil.py
:lines: 7-
Expand All @@ -133,20 +135,23 @@ This is an example using `frombuffer() <http://pillow.readthedocs.io/en/latest/r
Playing with pixels
-------------------

This is an example using `putdata() <https://github.com/python-pillow/Pillow/blob/b9b5d39f2b32cec75b9cf96b882acb7a77a4ed4b/PIL/Image.py#L1523>`_:
This is an example using :py:meth:`mss.ScreenShot.to_pil` and direct pixel edits:

.. literalinclude:: examples/pil_pixels.py
:lines: 7-

.. versionadded:: 3.0.0

OpenCV/Numpy
OpenCV/NumPy
============

See how fast you can record the screen.
You can easily view a HD movie with VLC and see it too in the OpenCV window.
And with __no__ lag please.

When using :py:meth:`mss.ScreenShot.to_numpy`, pass ``channels="BGR"`` for OpenCV, and use ``channels="RGB"`` (the
default) for scikit-image and most other frameworks.

.. literalinclude:: examples/opencv_numpy.py
:lines: 7-

Expand Down
10 changes: 5 additions & 5 deletions docs/source/examples/opencv_numpy.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
import time

import cv2
import numpy as np

import mss

Expand All @@ -18,15 +17,16 @@
while "Screen capturing":
last_time = time.time()

# Get raw pixels from the screen, save it to a Numpy array
img = np.array(sct.grab(monitor))
# Get raw pixels from the screen, save it to a NumPy array.
# Note that OpenCV expects colors in BGR order.
img = sct.grab(monitor).to_numpy(channels="BGR")

# Display the picture
cv2.imshow("OpenCV/Numpy normal", img)
cv2.imshow("OpenCV/NumPy normal", img)

# Display the picture in grayscale
# cv2.imshow('OpenCV/Numpy grayscale',
# cv2.cvtColor(img, cv2.COLOR_BGRA2GRAY))
# cv2.cvtColor(img, cv2.COLOR_BGR2GRAY))

print(f"fps: {1 / (time.time() - last_time)}")

Expand Down
8 changes: 2 additions & 6 deletions docs/source/examples/pil.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
"""This is part of the MSS Python's module.
Source: https://github.com/BoboTiG/python-mss.

PIL example using frombytes().
PIL example using ScreenShot.to_pil().
"""

from PIL import Image

import mss

with mss.MSS() as sct:
Expand All @@ -15,9 +13,7 @@
sct_img = sct.grab(monitor)

# Create the Image
img = Image.frombuffer("RGB", sct_img.size, sct_img.bgra, "raw", "BGRX", 0, 1)
# The same, but less efficient:
# img = Image.frombuffer('RGB', sct_img.size, sct_img.rgb, 'raw', 'RGB', 0, 1)
img = sct_img.to_pil("RGB")

# And save it!
output = f"monitor-{num}.png"
Expand Down
4 changes: 1 addition & 3 deletions docs/source/examples/pil_pixels.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,14 @@
PIL examples to play with pixels.
"""

from PIL import Image

import mss

with mss.MSS() as sct:
# Get a screenshot of the 1st monitor
sct_img = sct.grab(sct.monitors[1])

# Create an Image
img = Image.new("RGB", sct_img.size)
img = sct_img.to_pil("RGB")

# Best solution: create a list(tuple(R, G, B), ...) for putdata()
pixels = zip(sct_img.bgra[2::4], sct_img.bgra[1::4], sct_img.bgra[::4], strict=False)
Expand Down
2 changes: 1 addition & 1 deletion docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ An ultra fast cross-platform multiple screenshots module in pure python using ct
- **Python 3.10+**, :pep:`8` compliant, no dependency, thread-safe;
- very basic, it will grab one screenshot by monitor or a screenshot of all monitors and save it to a PNG file;
- but you can use PIL and benefit from all its formats (or add yours directly);
- integrate well with Numpy and OpenCV;
- integrate well with NumPy, OpenCV, PyTorch, and TensorFlow;
- it could be easily embedded into games and other software which require fast and platform optimized methods to grab screenshots (like AI, Computer Vision);
- get the `source code on GitHub <https://github.com/BoboTiG/python-mss>`_;
- learn with a `bunch of examples <https://python-mss.readthedocs.io/examples.html>`_;
Expand Down
20 changes: 11 additions & 9 deletions docs/source/release-history/v11.0.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,8 @@ The {py:attr}`mss.ScreenShot.raw` attribute has been deprecated, and will soon b
{py:attr}`mss.ScreenShot.bgra` property instead.

The {py:attr}`mss.ScreenShot.bgra` and {py:attr}`mss.ScreenShot.rgb` properties now will return bytes-like
{py:type}`memoryview` objects, not necessarily {py:type}`bytes` or {py:type}`bytearray` objects. For practical use
cases, this should not be noticible. This change allows faster access to screenshot data, with fewer memory
copies.
{py:class}`memoryview` objects, rather than {py:class}`bytes` or {py:class}`bytearray` objects. For practical use
cases, this should not be noticeable. This change allows faster access to screenshot data, with fewer memory copies.

### Python 3.9 EOL

Expand All @@ -48,13 +47,16 @@ processing time decreased from 22.64 ms to 18.59 ms per frame (approximately 18%

Support for additional operating systems is planned.

### General Improvements
### Export Methods

The MSS context object will now always surface inner exceptions, even if `__exit__` may also generate an exception during tear-down.
There are now convenient, easy-to-use methods to export screenshots to several popular formats: the popular PIL image
library, the NumPy array format for scientific computing, and the PyTorch and TensorFlow deep learning frameworks.

### Documentation
Exporting to these formats was always possible from MSS, but the best way to do it hasn’t always been obvious. By
providing these methods, MSS gives you tested, high-speed ways to transfer the data to these popular formats.

The documentation has received numerous small improvements. A few highlights:
See the documentation section {ref}`accessing_pixel_data` for details.

* The Pillow examples and demos using {py:meth}`PIL.Image.frombuffer` now explicitly specify the decoder arguments,
as recommended by Pillow. (#535)
### General Improvements

The MSS context object will now always surface inner exceptions, even if `__exit__` may also generate an exception during tear-down.
Loading
Loading