BUG: groupby rolling apply giving strange index

This issue has been created since 2022-11-23.

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import datetime

d = pd.DataFrame([
    (datetime.date(2022, 1, 1), 'A', 10),
    (datetime.date(2022, 1, 2), 'B', 20),
    (datetime.date(2022, 1, 3), 'A', 30),
    (datetime.date(2022, 1, 4), 'B', 40),
],
columns=['date','group','value'])

d['date'] = pd.to_datetime(d['date'])

def func(s):
    print(s)
    return s.index.max().day

d.set_index(d.date).groupby('group').rolling('7d').value.apply(func)

Issue Description

I'm trying to apply a custom function using rolling apply and using the date index of the window, but the index of the data passed to the apply function seems incorrect

output of above rolling apply:

group  date      
A      2022-01-01    1.0
       2022-01-03    2.0
B      2022-01-02    3.0
       2022-01-04    4.0
Name: value, dtype: float64

Expected Behavior

group  date      
A      2022-01-01    1.0
       2022-01-03    3.0
B      2022-01-02    2.0
       2022-01-04    4.0
Name: value, dtype: float64

Installed Versions

INSTALLED VERSIONS

commit : 8dab54d
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.0-52-generic
Version : #58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : pt_BR.UTF-8

pandas : 1.5.2
numpy : 1.22.3
pytz : 2021.3
dateutil : 2.8.2
setuptools : 45.2.0
pip : 22.3.1
Cython : None
pytest : 4.6.9
hypothesis : None
sphinx : 4.4.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.1
html5lib : 1.0.1
pymysql : 1.0.2
psycopg2 : None
jinja2 : 3.1.2
IPython : 7.13.0
pandas_datareader: None
bs4 : 4.8.2
bottleneck : None
brotli : 1.0.9
fastparquet : None
fsspec : 2022.11.0
gcsfs : None
matplotlib : 3.6.0
numba : 0.56.4
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : 7.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.3.3
snappy : None
sqlalchemy : None
tables : 3.6.1
tabulate : 0.8.9
xarray : None
xlrd : 1.1.0
xlwt : 1.3.0
zstandard : None
tzdata : None

fakio wrote this answer on 2022-11-24

maybe related to #48056

fakio wrote this answer on 2022-11-29

seems related to #47494 and #45925 too @mroeschke

mroeschke wrote this answer on 2022-11-29

I believe the resulting index is correct here but the values are ordered incorrectly. Investigation and pull requests welcome!

fakio wrote this answer on 2022-12-01

from what I could debug until now, in https://github.com/pandas-dev/pandas/blob/main/pandas/core/window/rolling.py#L1417, "values" is the original values reordered by groups and self._on is the original index, so the resulting Series has mismatched index/values.

More Details About Repo
Owner Name pandas-dev
Repo Name pandas
Full Name pandas-dev/pandas
Language Python
Created Date 2010-08-24
Updated Date 2022-12-07
Star Count 36164
Watcher Count 1118
Fork Count 15472
Issue Count 3683

YOU MAY BE INTERESTED

Issue Title Created Date Comment Count Updated Date
dynamic import with Typescript doesn't work. 10 2021-03-03 2022-11-09
`fclones dedupe` results in updates to mtimes on directories 2 2021-11-24 2022-11-30
ignore size and report duplicates 1 2021-11-18 2022-11-30
The Shippable service will no longer be available as of May 3rd, 2021 1 2021-05-07 2022-10-11
[Feature] Allow for mocking file size 2 2021-05-25 2022-09-09
Resource to add: https://quizzes.madza.dev/ 1 2021-10-12 2022-07-26
Resource to add: https://getcssscan.com/css-buttons-examples 0 2021-10-09 2022-03-05
当从店铺退会后重新入会还能有机会得到奖励吗 1 2021-05-19 2022-10-09
Accessing android folder 0 2022-08-11 2022-11-05
Use more than `primary` and `secondary` colors when using typescript 3 2022-06-20 2022-10-15
Descriptor Loss and SuperPoint training 6 2020-07-16 2022-10-05
After processing Yarn asset imports Dialogue Runner should reload current programs 1 2021-03-03 2022-10-10
[PORTER]gcr.io/knative-releases/knative.dev/eventing/cmd/broker/filter:latest 2 2022-04-18 2022-04-14
[PORTER]docker.io/calico/pod2daemon-flexvol:v3.22.2 4 2022-04-18 2022-04-14
[PORTER]gcr.io/knative-releases/knative.dev/eventing/cmd/broker/ingress:latest 2 2022-04-18 2022-08-15
[PORTER]gcr.io/knative-releases/knative.dev/eventing/cmd/mtchannel_broker:latest 2 2022-04-18 2022-04-14
Cannot see the actual error messages . 1 2021-05-29 2022-11-06
`ModuleNotFoundError` Raised when using py3.11 0 2022-11-24 2022-12-04
Model doesn't predict all groups 0 2022-03-28 2022-11-20
clang on Windows defines _MSC_VER, renderer_d3d12.cpp fails to compile 1 2022-11-12 2022-11-29
Need help adding support for Spartan-7 board to arch-defs 5 2022-01-29 2022-10-22
Test for CORE-1440 [QA667] 0 2021-01-11 2022-12-04
[CT-743] [Feature] remove werzog script from core as it is moved to snowflake and redundant now 0 2022-06-10 2022-10-09
job_templates module does not have effective user settings 0 2022-03-25 2022-10-03
Hosts API to Reports API, why? 5 2022-03-24 2022-10-03
provide full path to module separately from name 2 2014-11-27 2022-09-23
Update Documentation KDTreeFlann.search* functions 0 2022-11-11 2022-11-29
Finish EIP-2681 support in VM 1 2021-11-30 2022-11-03
[Question] Pojavlauncher keeps deleting all the game version I install at midnight everyday is this a bug? How do I stop it from deleting the versions 3 2022-09-24 2022-11-26
[email protected]: download via aria2 failed 1 2022-01-20 2022-11-25
Register flags from provided config struct 4 2021-03-09 2022-08-30
There is miss reference code when calling weekday method of eager series. 0 2021-09-14 2022-10-17
Does LinksAndHandlebarsTemplate work on on prem 2019 1 2021-06-15 2022-11-28
[BUG] S3 Reports keep overwriting themselves 3 2022-11-07 2022-11-22
[Component] [loading] namespace 页面初始化ElLoading失效 1 2022-07-21 2022-10-07
Zen Thermostat Cooling 2 2021-11-11 2022-11-02
Feature Idea: Change workspace name 1 2021-10-14 2022-11-26
MPI run results in crash in metis_partitioner.C 1 2021-12-22 2022-01-02
Screen display divided into two parts 11 2022-02-04 2022-10-31
add font: Anka/Coder 3 2018-10-22 2022-11-19
\Cake\ORM\Table infinite loop bug 1 2022-11-23 2022-12-01
Possible to skip transcoding? 8 2019-12-04 2022-11-15
update dependencies please 1 2021-11-02 2022-11-29
Migrating to null safety 2 2021-04-27 2022-11-29
Supports loading of word files with images added on OneDrive 1 2022-06-20 2022-11-25
App crashes from `java.lang.IllegalArgumentException` on `CastPlayer.setMediaItems()` when passing large playlist 1 2022-07-04 2022-10-07
kafka-connect not starting on docker Swarm 0 2021-10-20 2022-12-05
Define builtin only license properties in constants rather than the model 4 2021-11-18 2022-11-26
[Questions] about the pods checking in the kube-system 5 2021-01-01 2022-12-05
Grid with auto row and column not rendering child Frame with correct height on iOS when child has a margin 2 2021-11-20 2022-11-26