BUG: parsing ISO8601 string with `format=` and timezone name fails

This issue has been created since 2022-11-17.

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

In pandas 1.5.1:

import pandas as pd
from pandas import *
import sys
import datetime as dt

print(dt.datetime.strptime('2020-01-01 00:00:00UTC', '%Y-%m-%d %H:%M:%S%Z'))  # works!
to_datetime('2020-01-01 00:00:00UTC', format='%Y-%m-%d %H:%M:%S%Z')  # crashes!

Issue Description

The ISO8601 fastpath in

https://github.com/pandas-dev/pandas/blob/6ec3d8f9dfd5dc89a2e9695ee4e01d8c280ebe2c/pandas/_libs/tslibs/src/datetime/np_datetime_strings.c

has no way to deal with '%Z' formatted-strings. They shouldn't get there in the first place.

Hence, '%Z' should be excluded from

def format_is_iso(f: str) -> bint:
"""
Does format match the iso8601 set that can be handled by the C parser?
Generally of form YYYY-MM-DDTHH:MM:SS - date separator can be different
but must be consistent. Leading 0s in dates and times are optional.
"""
iso_template = '%Y{date_sep}%m{date_sep}%d{time_sep}%H:%M:%S{micro_or_tz}'.format
excluded_formats = ['%Y%m%d', '%Y%m', '%Y']
for date_sep in [' ', '/', '\\', '-', '.', '']:
for time_sep in [' ', 'T']:
for micro_or_tz in ['', '%z', '%Z', '.%f', '.%f%z', '.%f%Z']:
if (iso_template(date_sep=date_sep,
time_sep=time_sep,
micro_or_tz=micro_or_tz,
).startswith(f) and f not in excluded_formats):
return True
return False

Expected Behavior

In [1]: to_datetime('2020-01-01 00:00:00UTC', format='%Y-%m-%d %H:%M:%S%Z')
Out[1]: Timestamp('2020-01-01 00:00:00+0000', tz='UTC')

Installed Versions

INSTALLED VERSIONS

commit : 91111fd
python : 3.10.6.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.102.1-microsoft-standard-WSL2
Version : #1 SMP Wed Mar 2 00:30:59 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 1.5.1
numpy : 1.23.3
pytz : 2022.2.1
dateutil : 2.8.1
setuptools : 65.3.0
pip : 22.2.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.5.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : None
fastparquet : None
fsspec : 2022.10.0
gcsfs : None
matplotlib : 3.6.1
numba : None
numexpr : None
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : 2022.10.0
scipy : 1.9.1
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None
None

MarcoGorelli wrote this answer on 2022-11-17
More Details About Repo
Owner Name pandas-dev
Repo Name pandas
Full Name pandas-dev/pandas
Language Python
Created Date 2010-08-24
Updated Date 2022-12-07
Star Count 36164
Watcher Count 1118
Fork Count 15472
Issue Count 3683

YOU MAY BE INTERESTED

Issue Title Created Date Comment Count Updated Date
[Feature]: JavaScript Framework-agnostic Frontend 1 2022-11-18 2022-11-27
An invalid value was specified for display_name 1 2021-04-27 2022-11-29
color of startup background does not match 8 2022-01-15 2022-03-31
LUA output & Injector 8 conflict cause no output to VR_THRESH 6 2022-03-08 2022-11-18
New logo for your clownery 0 2022-10-17 2022-11-17
Unsnapped behaviour impossible in Loop mode? 6 2022-09-03 2022-10-29
ImageField not populated width/height for child model 2 2021-08-19 2022-10-29
[Release] Generate v1.0 1 2020-04-30 2022-10-01
[Tests] Implement Integration Test 7 2020-04-30 2022-11-30
[Binary] Release inside ROS main repo 20 2020-04-30 2022-12-07
"no prune" option to keep generated Secret 2 2021-10-06 2022-12-04
[Bug] Incorrect CoreUI CDN link provided in config/backpack/base.php 2 2022-06-30 2022-10-02
Can I update the vertex of the polygon? 2 2022-04-29 2022-12-04
Promoting Zappa To the Next Generation of Developers 3 2021-04-24 2022-10-06
Add performance testing to CI 0 2020-07-23 2022-04-05
Replace indexing by function name with int 1 2020-07-23 2022-09-19
Error recovery behavior is not clear 5 2022-02-14 2022-09-24
tests.unit.test_config.TestConfig: test_list_variables_empty failed 4 2021-09-16 2022-12-07
Build artifacts not being produced 2 2021-10-31 2022-11-20
Autocomplete isn't working properly 0 2022-11-04 2022-11-17
Cypress test fail: should create new channel. TC: SALEOR_0701 0 2022-11-03 2022-11-17
Heroku Server Error (500) 4 2022-02-20 2022-10-08
Can't start using docker-compose: 'Are you trying to mount a directory onto a file (or vice-versa)?' 4 2022-01-15 2022-10-14
[BUG] Cannot login into webmail from any account due to error: Invalid token 1 2022-10-09 2022-11-26
Upgrade zstd version for Linux packaging 0 2022-04-12 2022-10-12
OVM support 0 2021-03-23 2022-01-01
Have I Been Pwned API Key 1 2021-04-15 2022-11-27
Headful / headed runs failing on GitHub Actions (python-playwright / Chromium / Linux) 7 2021-06-04 2022-11-26
[Bug] Missing JSON element "microformat" (BrokenTubeException) 2 2022-09-17 2022-11-13
Video is not buffering during network interruption. 0 2021-11-06 2022-08-31
Tracking: ServiceWorker interaction 5 2019-05-20 2022-12-04
bug: `date_time`'s end is not inclusive 0 2021-12-23 2022-11-24
[AGW] Add UE security-mode events 4 2021-09-16 2022-11-10
TypeError("Params must be a param map but got %s." % type(params)) 6 2022-04-19 2022-11-29
Can't install pact_verifier_cli 4 2022-03-24 2022-11-30
egyshare.cc 1 2021-10-17 2022-01-17
Update acli and acquia provider integration 2 2022-01-11 2022-10-11
Data downloaded but command not returning 0 2020-03-07 2022-11-28
Converter timestamp for data 2 2022-02-18 2022-10-22
`Router::with_state_arc` might be misleading 12 2022-08-24 2022-09-21
测试官方nanodet-m-0.5x的map指标发现很低 0 2021-08-30 2022-09-29
IllegalStateException: Fragment IngredientsWithTagDialogFragment{b909bdf} (b06e681d-6053-4f9b-ad89-dce395eda729) has not... 0 2021-08-25 2022-05-11
panic: runtime error: index out of range [-9223372036852667758] 18 2022-01-24 2022-11-30
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]. 1 2021-05-02 2022-11-23
org.jacoco.core/src/org/jacoco/core/internal/instr/ProbeInserter.javar# insertProbe coverage is not complete 1 2022-04-14 2022-11-25
Follow system theme 4 2021-05-28 2022-01-15
Extension bundling with ESBuild documentation does not work 2 2021-11-10 2022-11-25
Grid View: Details Panel does not reopen (once hidden) when a task is clicked. Have to manually unhide the details panel. 3 2022-05-05 2022-11-13
Doesn't provide an example of how to fix this error 2 2022-07-19 2022-11-11
nvmrc support 3 2018-10-30 2022-01-23