REGR: Setting values with 'loc' and boolean mask mixes up values (all-True mask)

This issue has been created since 2022-09-22.

According to documentation, using loc should work with Boolean arrays. However, the following does not seem to work:

import pandas as pd

n = 17

data = pd.DataFrame({
    "name": n * ["a"], 
    "x": range(n), 
    "y": range(n)
})

copy = data.copy()

idx = n * [True]
data.loc[idx, ["x", "y"]] = data[["x", "y"]]
assert data.equals(copy)   # Raises assertion error

The weird thing is that if n is smaller then the code works. This has been tested on Pandas 1.5.0

martinfleis wrote this answer on 2022-09-22

I believe that this is a regression in pandas 1.5.0 as this works correctly on 1.4.4. We ran into the same bug in downstream yesterday geopandas/geopandas#2558.

jorisvandenbossche wrote this answer on 2022-09-22

The reproducer from that issue:

In [1]: df = pd.DataFrame({'idx': range(20), 'col': np.arange(20, dtype="float")})

In [2]: df.loc[np.ones(len(df), dtype=bool), 'col'] = df.col.values

In [3]: df
Out[3]: 
    idx   col
0     0   0.0
1     1  17.0
2     2  16.0
3     3  15.0
4     4  14.0
5     5  13.0
6     6  12.0
7     7  11.0
8     8  10.0
9     9   9.0
10   10   8.0
11   11   7.0
12   12   6.0
13   13   5.0
14   14   4.0
15   15   3.0
16   16   2.0
17   17   1.0
18   18  18.0
19   19  19.0

The weird thing is that if n is smaller then the code works.

It indeed needs some minimum values, because of np.argsort only starting to return non-sorted values after a minimum number of True values:

In [4]: np.argsort([True]*10)
Out[4]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

In [5]: np.argsort([True]*17)
Out[5]: array([ 0, 14, 13, 12, 11, 10,  9, 15,  8,  6,  5,  4,  3,  2,  1,  7, 16])

This argsort is used here:

value = value[np.argsort(pi)]

That code assumes that indexer is already an integer indexer, not a boolean one. Before, we converted a boolean indexer to integers in _locIndexer._convert_to_indexer, but this was removed in #45501 (location in diff).

I haven't verified to be 100% sure, but so given the code path and the change in #45501, I assume that's the cause.

MarcoGorelli wrote this answer on 2022-09-24

I haven't verified to be 100% sure, but so given the code path and the change in #45501, I assume that's the cause.

yup, git bisect confirms

More Details About Repo
Owner Name pandas-dev
Repo Name pandas
Full Name pandas-dev/pandas
Language Python
Created Date 2010-08-24
Updated Date 2022-09-29
Star Count 35374
Watcher Count 1122
Fork Count 15034
Issue Count 3579

YOU MAY BE INTERESTED

Issue Title Created Date Comment Count Updated Date
AttributeError: module transformers has no attribute FeatureExtractionMixin 2 2022-08-24 2022-08-27
Automate the management of k8s namespaces 0 2021-06-25 2022-08-19
Got Overwriting cache with 304 response 55 2018-03-21 2022-08-19
visceral fat different from the one computed by xiaomi app 8 2021-08-17 2022-04-03
data cannot be used in stats graphs 5 2021-08-20 2022-08-07
expose the computed attributes as sensors 3 2021-08-17 2022-08-01
body score missing from attributes 2 2021-08-17 2022-08-02
Invite not working when self-hosted 0 2022-06-07 2022-09-22
Add 3 bookstore chain for India 1 2022-07-18 2022-09-27
Can you share the complete C++ project, the compilation file is currently missing 0 2021-05-07 2022-09-10
Decode raw h264 UDP byte[]? 0 2021-10-10 2022-09-04
Process 'command '/Users/userA/swift-android-5.0-r13/build-tools/1.9.6-swift5/swift-build'' finished with non-zero exit value 134 0 2021-07-04 2022-07-07
Freshly installed simple-bar gets me a 'simple-bar-spaces.jsx: Something went wrong' 3 2021-09-12 2022-09-28
Obtaining RSSI value from server while connected 6 2022-02-08 2022-08-22
unable to update RSSI values 0 2022-02-09 2022-09-23
repr(a) == repr(b) should imply a == b 2 2022-08-31 2022-09-22
Suggestion: Download only nonexistant/old-format covers 2 2020-03-24 2022-09-01
Crashes consistently at around 2.5k games and 1 GB ram usage. 5 2018-06-18 2022-08-15
git configをgoghで管理できたら便利なのでは? 2 2020-01-20 2022-07-15
alias(別名)機能が欲しい 6 2020-11-05 2021-12-30
gogh repos ignores the query 1 2021-03-07 2022-07-15
false positive SyntaxError: functions using *args before keyword arguments 3 2021-03-28 2022-08-18
Uptobox plugin & package name 47 2021-12-10 2022-08-25
Error when using the cli 0 2021-02-11 2022-09-06
Topic actions dropdown menu items list is broken 1 2022-07-25 2022-09-22
Getting Started Link broken 1 2021-11-29 2022-08-26
Parca UI search should be case-insensitive 0 2022-07-11 2022-08-05
Replace getNetwork() calls with getChainId() 1 2022-05-01 2022-08-04
Added catalan to dynamic programming 3 2021-11-22 2022-03-01
Running onto2vec given this error? 0 2021-10-06 2021-12-11
Java heap space out of memory error 2 2021-12-07 2021-12-31
Chrome window is shifted down and right on maximize 4 2021-10-24 2022-07-26
Issue deleting #scope.sync.pod using 'Delete Node' command 2 2022-05-05 2022-07-13
Re-add keycard functionality using status-keycard-go 1 2022-02-08 2022-07-01
List of suggestions is not showing up when pasting @ to mention someone in chat 3 2022-02-07 2022-02-18
RFE: add name parameters to list_* methods 2 2021-01-11 2022-09-21
OSS-Fuzz: Integer-overflow in spvtools::opt::PerformIntegerOperation 0 2021-09-10 2022-09-26
1.8.7版本服务器配置里保存不了账号密码 1 2021-08-17 2022-07-25
Support for ReadWriteMany without using NFS 10 2020-09-21 2022-07-21
Feature request : support @argumentDefinitions directive 2 2018-07-27 2022-07-20
Weird Display in RN V0.68 New Architecture Enabled 10 2022-02-26 2022-08-31
Can I debate it like it debated itself at the Oxford Union? 0 2022-06-22 2022-09-15
[Bug] [Standalone Server] DolphinPluginManagerConfig get maven sttings does not check file exists 1 2021-11-04 2022-07-20
Support for Gradle's configuration cache 0 2020-10-26 2022-08-17
Unused import is obsoleted 1 2021-05-29 2022-09-13
[FP]: mariadb-java-client-3.0.3.jar 1 2022-03-21 2022-09-07
Please, update `master` to `index-state: >=2021-12-18T00:00:08Z` 3 2021-12-22 2022-08-21
xo forcing lower case for models now? 2 2021-07-15 2022-09-16
EasyCLA Feedback & Issues 53 2021-10-28 2022-09-14
Renaming ZipkinUrl option into httpReporterUrl 3 2020-01-15 2022-09-25