PERF: assigning in a for loop

This issue has been created since 2022-11-16.

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this issue exists on the latest version of pandas.

  • I have confirmed this issue exists on the main branch of pandas.

Reproducible Example

I'm experiencing a slowdown compared to prior versions when assigning a DataFrame in a for loop :

import pandas as pd

def foo(df):
    for idx in df.index:
        df.at[idx, "bar"] = 3

df = pd.DataFrame(range(10000))
df["bar"] = 0

%timeit foo(df)
# pandas 1.5.1 : 506 ms ± 20.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

My use case is more complex and the loop can't be vectorized, that's why I'm using a for loop instead.

Installed Versions

INSTALLED VERSIONS

commit : 91111fd
python : 3.9.6.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.0-19-amd64
Version : #1 SMP Debian 5.10.149-2 (2022-10-21)
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : fr_FR.UTF-8
LOCALE : fr_FR.UTF-8

pandas : 1.5.1
numpy : 1.23.3
pytz : 2022.2.1
dateutil : 2.8.2
setuptools : 65.4.0
pip : 22.2.2
Cython : 0.29.32
pytest : 6.2.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.0.3
lxml.etree : 4.9.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.5.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.6.0
numba : None
numexpr : None
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.9.1
snappy : None
sqlalchemy : 1.4.41
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None

Prior Performance

%timeit foo(df)
# pandas 1.4.4 : 67 ms ± 8.92 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
phofl wrote this answer on 2022-11-16

Hi, thanks for your report. Caused by #47074 cc @jorisvandenbossche

labelling as regression for now

jorisvandenbossche wrote this answer on 2022-11-16

Taking a look.

We might not be able to get the performance fully down to what it was before, it was maybe also a bit "too fast". To illustrate, using __setitem__ on a Series is also slower than doing that directly on the underlying array:

In [10]: s = pd.Series(range(10000))

In [11]: %timeit s[0] = 2
7.67 µs ± 92.9 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

In [12]: %timeit s._values[0] = 2
269 ns ± 1.29 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

This is expected (we do more than what numpy does, plus we have some overhead), while of course we want to limit this slowdown compared to numpy as much as possible.

But so previously, at was using SingleBlockManager.setitem_inplace, which was directly setting the values on the underlying numpy array. While now we go through the more generic Block.setitem, just like we do for Series.__setitem__.

This feels more correct do do (since we also do that for .loc etc), but from a quick look at a perf profile, there is certainly some overhead that can be reduced.
There was also some discussion on the original issue about setitem_inplace vs setitem: https://github.com/pandas-dev/pandas/pull/47074/files#r878902106. Given the performance impact, we could also look in keeping some "inplace" fastpath to test first as we did before. I actually had a commit in that PR that did this (453eaba), but removed in the end again arguing that it was not worth it (but at the time only considered behaviour, not performance). I can check if that commit would improve the reproducer here.

Saelyos wrote this answer on 2022-11-17

Hi, thanks for the investigation and the explanations.

After a bit of digging I found a slowdown even more important in some cases. I had a bit of trouble to make a reproducible example because from what I've understood it has do with the internal blocks so I needed a large DataFrame.

Reproducible Example

import pandas as pd

def foo(df):
    for idx in df.index:
        df.at[idx, "bar"] = ""

df = pd.DataFrame(range(100000))
df["bar"] = ""
df["a"] = "aaaaaaaaa"
df["b"] = "aaaaaaaaa"
df["c"] = "aaaaaaaaa"
df["d"] = "aaaaaaaaa"
df["e"] = "aaaaaaaaa"
df["f"] = "aaaaaaaaa"

%timeit foo(df)
# pandas 1.5.1: 4.73 s ± 238 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

df = df.copy()
%timeit foo(df)
# pandas 1.5.1: 36.1 s ± 620 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Prior Performance

%timeit foo(df)
# pandas 1.4.4: 519 ms ± 13.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

df = df.copy()
%timeit foo(df)
# pandas 1.4.4: 516 ms ± 18.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
jorisvandenbossche wrote this answer on 2022-11-18

I opened two PRs that each partly address this performance regression: #49771, #49772

Also your second example is considerably better with those two PRs:

In [5]: %timeit foo(df)
800 ms ± 44.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [6]: df = df.copy()

In [7]: %timeit foo(df)
822 ms ± 44.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
More Details About Repo
Owner Name pandas-dev
Repo Name pandas
Full Name pandas-dev/pandas
Language Python
Created Date 2010-08-24
Updated Date 2022-12-07
Star Count 36164
Watcher Count 1118
Fork Count 15472
Issue Count 3683

YOU MAY BE INTERESTED

Issue Title Created Date Comment Count Updated Date
add cli flags for rpc urls 0 2022-09-12 2022-11-24
After adding some custom user rules I can't access the options. 17 2022-11-19 2022-11-24
Media Buttons not are shown 1 2022-07-19 2022-10-12
Remote host and port should be configurable 8 2022-06-01 2022-11-09
Bugs on adding an image 0 2022-11-09 2022-11-15
ORSSerial/ORSSerialPacketDescriptor.h file not found 2 2021-01-16 2022-11-26
ORSSerialPort open -1 3 2021-01-21 2022-11-26
ORSSerial and SwiftUI/Combine 3 2020-10-01 2022-11-26
Add Pixelfed support 0 2021-10-04 2022-05-18
Support word ''' (tick) from CORE 0 2018-06-19 2022-11-19
add support for mermaid.js diagrams 3 2021-11-13 2022-11-24
Search for retractions 5 2022-03-04 2022-11-22
Hidden Fails To hide file on Windows SP1 x64 13 2018-07-17 2022-09-16
Set default resolutions for Wifi/mobile 4 2022-03-15 2022-11-24
support cloudflare page 5 2022-02-01 2022-11-17
Error in preprocessing (proc) in Rstudio 2 2021-03-04 2022-10-28
install with vim-plug 1 2021-04-21 2022-10-29
Cannot move to splits/panes 6 2021-04-06 2022-11-12
Runtime "dynamic_cast error" crash when pod is present both as a direct and as a transitive dependency 13 2022-01-25 2022-10-23
Intelligence Trainer: Universal custom "Titles and Phrases" 1 2021-10-06 2022-12-04
Exec not working 3 2021-03-13 2022-01-15
When I send a file single file use sample code, ,i got a page reload 0 2021-06-05 2022-11-18
Issue with Scenario-21 0 2021-01-21 2022-11-29
Issue in POST url generated for an API image to be pushed 2 2021-01-21 2022-11-29
Displays wrong stats after crossing 100 days 6 2021-11-25 2022-12-03
Entity doesn't have array for some reason 1 2021-10-28 2022-11-19
[2122fc] openrct2.exe: public: CD3DBase::UpdateTextures 0 2022-10-14 2022-10-07
Future Release Branches Frozen For Merging | branch:release-4.7 0 2020-06-27 2022-12-06
Release usethis 2.1.6 0 2022-05-20 2022-12-05
[Dashboard][Doc] Views not clickable 1 2022-05-25 2022-09-21
Popover content overflows when browser window is too small 0 2022-08-18 2022-09-18
README.md "Deploying a HA Kubernetes cluster" leads to 404 page not found link 0 2022-03-11 2022-11-12
Keystone UI (9) Make “Watch-only” consistent in capitalization throughout the interface 1 2022-04-22 2022-11-26
add Index and LastIndex property to $matches variable 7 2020-02-02 2022-10-05
Support Descriptions/BlockString from June 2018 GraphQL spec 16 2019-07-06 2022-11-12
Document the customization of the minikube.yml and openshift.yaml 1 2022-01-18 2022-10-27
No IME-related packages found 1 2022-11-21 2022-11-28
tls client certificate has expired 2 2022-05-16 2022-12-03
`VaSideBar` `position` prop 0 2022-06-17 2022-09-24
What is the difference between Caching and Snapshot? 0 2021-05-13 2022-11-02
Linter fails to parse public class fields 9 2021-07-23 2022-11-11
📦 pack-n-play test: TypeScript code failed 2 2022-01-02 2022-12-07
Integratind virtio drivers into curren Win10 iso fails. 12 2021-06-27 2022-11-19
Homekit qr code nowhere to be seen after initial installation 8 2021-12-24 2022-09-30
[Netatmo] [Thermostat] Unable to 'off' netatmo thermostat 5 2022-10-21 2022-10-16
使用ss-local,使用curl测试,无法连到本地的服务 2 2019-10-05 2022-12-02
web3.js: Hook up new blockheight-based transaction confirmation strategy all the way through sendAndConfirmTransaction 1 2022-05-14 2022-12-05
[Vectorized][Bug] Some query in SSB slower than not vectorized query engine 0 2022-01-25 2022-09-16
Add support for FLYWAY_CONFIG_FILES option in build.sbt 1 2021-08-13 2022-11-27
Exception caught by services library (! used on null) when pressing backspace in TextFormField 5 2021-09-25 2022-11-21