BUG: Assignment to a .loc view of a naive datetime column changes its dtype to object

This issue has been created since 2022-11-22.

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

from datetime import datetime

import pandas as pd


def main():
    df = pd.DataFrame({
        'field': pd.to_datetime([datetime(2022, 1, 20), datetime(2022, 1, 22)]),
        'update': [True, False],
    })
    print("Before:", df.info())
    print(df.head(2))
    print()
    df_to_update = df[df['update']]
    df.loc[df['update'], ['field']] = df_to_update['field']
    print("After:", df.info())
    print(df.head(2))


if __name__ == "__main__":
    main()

Issue Description

When doing a partial assignment to a view created by .loc[predicate, [column]] in a column with dtype datetime64[ns] (naive datetime) the column dtype changes to object and the datetimes assigned are represented as floats.

With non-naive datetimes it works as expected, maintaining the dtype as datetime64[ns, timezone].

The reproducible example below prints the following:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   field   2 non-null      datetime64[ns]
 1   update  2 non-null      bool
dtypes: bool(1), datetime64[ns](1)
memory usage: 146.0 bytes
Before: None
       field  update
0 2022-01-20    True
1 2022-01-22   False

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   field   2 non-null      object
 1   update  2 non-null      bool
dtypes: bool(1), object(1)
memory usage: 146.0+ bytes
After: None
                 field  update
0  1642636800000000000    True
1  2022-01-22 00:00:00   False

Expected Behavior

The assignment shouldn't change the values nor the dtype of the column.

As an example, see what's shown by the reproducible example when we add tzinfo=timezone.utc to the values:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   field   2 non-null      datetime64[ns, UTC]
 1   update  2 non-null      bool
dtypes: bool(1), datetime64[ns, UTC](1)
memory usage: 146.0 bytes
Before: None
                      field  update
0 2022-01-20 00:00:00+00:00    True
1 2022-01-22 00:00:00+00:00   False

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   field   2 non-null      datetime64[ns, UTC]
 1   update  2 non-null      bool
dtypes: bool(1), datetime64[ns, UTC](1)
memory usage: 146.0 bytes
After: None
                      field  update
0 2022-01-20 00:00:00+00:00    True
1 2022-01-22 00:00:00+00:00   False

As you can see, with a timezone the column doesn't change the dtype and the values are interpreted as datetimes, not floats.

Installed Versions

❯ python
Python 3.8.14 (default, Oct 10 2022, 16:44:50)
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import pandas as pd
pd>>> pd.show_versions()

INSTALLED VERSIONS

commit : 91111fd
python : 3.8.14.final.0
python-bits : 64
OS : Linux
OS-release : 5.19.13-arch1-1
Version : #1 SMP PREEMPT_DYNAMIC Tue, 04 Oct 2022 14:36:58 +0000
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : es_ES.UTF-8
LOCALE : es_ES.UTF-8

pandas : 1.5.1
numpy : 1.23.5
pytz : 2022.6
dateutil : 2.8.2
setuptools : 56.0.0
pip : 22.0.4
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None

phofl wrote this answer on 2022-11-22

Hi, thanks for your report. This works on main, could use a test

More Details About Repo
Owner Name pandas-dev
Repo Name pandas
Full Name pandas-dev/pandas
Language Python
Created Date 2010-08-24
Updated Date 2022-12-07
Star Count 36164
Watcher Count 1118
Fork Count 15472
Issue Count 3683

YOU MAY BE INTERESTED

Issue Title Created Date Comment Count Updated Date
[Bug] No Games - Exception 1 2021-08-14 2022-10-26
Enhancement request: format multiple blocks at once 2 2021-01-07 2022-03-19
Enhancement request: tooltip to preview outcome of drag/drop operations 2 2021-01-07 2022-03-19
Shortcut not working 2 2021-01-02 2022-03-19
Latex compatibility of the converter 1 2021-01-03 2022-03-19
Enhancement request: sidebar expand/collapse 5 2021-01-07 2022-03-19
[Enhancement]: Auth proxy 1 2022-09-07 2022-11-24
Wrong asset-ref schema 0 2021-02-26 2022-10-24
Wrong decoding/encoding for 24bit unsigned integer 3 2021-01-11 2022-05-13
Relayer: Packet relay issue 2 2021-05-14 2022-05-11
Feature Request: bestsimilar.com builder 1 2021-12-17 2022-10-26
zef nuke site should re-install zef itself again 2 2021-10-23 2022-11-09
Test against more recent Ubuntu distribution 2 2021-02-08 2022-07-27
get_artist_albums and get_user_playlists restructured 0 2021-01-27 2022-11-01
Editor inserts "Control" and "Alt" strings when you press those keys on Firefox 4 2021-03-18 2022-12-01
Something broke with the optimizer since last Penrose week 6 2022-05-17 2022-11-28
URL values in custom columns 1 2021-04-26 2022-11-17
Can't connect to DB using `latest` image: SessionExpired: WebSocket connection failure. Due to security constraints in your web browser, the reason for the failure is not available to this Neo4j Driver 12 2020-03-20 2022-12-07
Urutan jabatan tidak sesuai 2 2022-09-14 2022-10-03
Cannot get data and report an error:EOF v1.5.0 2 2021-04-19 2022-11-23
add documentation for the indexing statistics terms on the repository settings page 1 2022-05-04 2022-09-27
Companion heroku deployment broken 0 2021-12-11 2022-11-04
CVE-2022-30123 : rack (2.2.3) 0 2022-09-03 2022-10-06
Rename addExtraField argument from $list to $visibility 0 2022-11-15 2022-11-17
Wordpress Zero Spam blocked user message gets cached even though no-cache headers are set 1 2021-11-25 2022-07-30
Workflow Pause / Unpause 1 2022-06-17 2022-11-30
Feature Idea: Being able to download images 1 2022-03-30 2022-11-13
release: cut v6 release 1 2021-12-08 2022-11-13
Graph Creation for Different Tasks 0 2021-04-18 2022-11-26
--install (which implies --system) breaks under a regular user 0 2021-04-01 2022-10-03
html() function always starts on first page 6 2021-07-08 2022-10-16
Problem with java 17 and tycho surefire plugin 4 2021-11-23 2022-11-22
Quicly in non-http3 applications 0 2022-01-08 2022-11-12
`groupby.transform` failing on dataframe with duplicates in column 3 2022-03-11 2022-10-09
GI/Pango/Objects/Font.hs:670:9: error: parse error on input ‘HarfBuzz.FeatureT.feature_t’ 10 2020-06-23 2022-01-21
What should the ULP power consumption be in deep sleep? (IDFGH-6208) 12 2021-11-11 2022-11-06
Refactor BoolUnification and SetUnification to use `Formula` 5 2022-09-06 2022-11-13
Workshop findings by Impact vs Effort 0 2019-12-04 2022-11-13
The swift package generate-xcodeproj command throws an error on Packages that use plugins 0 2022-10-12 2022-10-29
Data preprocessing pattern 1 2020-11-16 2022-11-13
Stack overflow in quickcheck case shrinking 3 2021-04-30 2022-11-26
Reindex command calls reset instead of reindex. 1 2020-04-30 2022-11-28
System-Insights not working with Custom Perf Counters 1 2020-05-16 2022-11-04
[Feature Request] Optional leading zeros for hydra variable "${hydra.job.num}" 2 2021-08-26 2022-12-01
[DataCap Application] Wel Vape 17 2021-12-16 2022-07-26
[DataCap Application] NHASH 17 2021-12-17 2022-07-29
Page http://graph.facebook.com/GoPivotal is invalid 1 2014-07-16 2022-07-30
node evacuation 0 2021-12-09 2022-01-01
Broken rendering of Fastify main branch 3 2021-12-13 2022-11-28
Add support for Reference-style Links 4 2021-12-06 2022-11-28