Repository navigation
error from netCDF in cf.write #960
Description
I am processing some CMIP6 data on Jasmin (making time-means). I get the following error. I realise that you probably need more background to diagnose it, but perhaps it's obvious from this much information? As a result of the error, the file is not properly written, just the start of it.
Thanks. Jonathan
Traceback (most recent call last):
File "/home/users/jonathan/bin/grouptime", line 97, in <module>
cf.write(n,ofile)
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/write.py", line 826, in __new__
netcdf.write(
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/decorators.py", line 171, in verbose_override_wrapper
return method_with_verbose_kwarg(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 5232, in write
self._file_io_iteration(
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 5478, in _file_io_iteration
self._write_global_attributes(fields)
File "/apps/jasmin/jaspy/miniforge_envs/jaspy3.12/mf3-25.3.0-3/envs/jaspy3.12-mf3-25.3.0-3-v20250704/lib/python3.12/site-packages/cfdm/read_write/netcdf/netcdfwrite.py", line 4610, in _write_global_attributes
g["netcdf"].setncattr(
File "src/netCDF4/_netCDF4.pyx", line 3087, in netCDF4._netCDF4.Dataset.setncattr
File "src/netCDF4/_netCDF4.pyx", line 1858, in netCDF4._netCDF4._set_att
File "src/netCDF4/_netCDF4.pyx", line 6733, in netCDF4._netCDF4._strencode
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 61-63: surrogates not allowed
Activity
davidhassell commentedon Jul 23, 2026
Thanks for the report, Jonathan.
Could you let us know which version of cf-python you are using (the output from
cf.environment()is ideal), and if you could send the JASMIN path to the file I can investigate further..David
JonathanGregory commentedon Jul 23, 2026
Dear David
On jasmin in
/gws/ssde/j25a/ncas_climate/vol2/users/jonathan/cmip6I execute/home/users/jonathan/bin/grouptime -v prthetao/prthetao_Emon_MPI-ESM1-2-HR_faf-heat_r1i1p1f1_gn_191501-191912.nc /tmp, where the first argument is the input file, the second the output directory. The verbose (-v) terminal output is below, starting with the cf environment. Is this what you need?Best wishes
Jonathan
davidhassell commentedon Jul 31, 2026
Thanks, Jonathan.
I'm not sure what's going on, but it's surely a bug. I tried with v3.20.0, and it sill failed until I added
netcdf_backend='netCDF4'to thecf.writecall:Then it worked:
We can't fix
v3.18.0, so I recommend upgrading to3.20.0and using the newcf.writekeyword. I can then look into the still-present bug without holding you up any longer.JonathanGregory commentedon Aug 3, 2026
Thanks for investigating, @davidhassell. On Jasmin I use
module load jaspy, which I presume is the latest they support. How would I combine this with a later cf-python?davidhassell commentedon Aug 5, 2026
Dear Jonathan,
The easiest way might be to install a local copy of
cf-pythonat the newer version:$ pip install --user cf-python==3.20.0This puts that version of
cfin (something like)~/.local/lib/python3.12/site-packages/, and I think that location will automatically be at the front of yourPYTHONPATH:If it's still picking up 3.18.0, you can force it to find your newer version with:
$ export PYTHONPATH=~/.local/lib/python3.12/site-packages:$PYTHONPATH # (or whatever your local directory is)JonathanGregory commentedon Aug 5, 2026
Dear David
Thanks very much. That works exactly as you described and avoids the problem.
Should I be concerned about these warnings:
What is
p5dumpfor, which it warns me isn't in my PATH?Best wishes
Jonathan
davidhassell commentedon Aug 5, 2026
All red herrings, and safely ignored :) - you are using local data (hence no need to call on
fsspec), and p5dump is a command-line tool that ships withpyfive, which is nothing to do with cf-pythonJonathanGregory commentedon Aug 5, 2026
😄
davidhassell commentedon Aug 14, 2026
This will be fixed in the next release, meaning you shouldn't need to specify the
netCDF4backend: NCAS-CMS/cfdm#423JonathanGregory commentedon Aug 17, 2026