Pandas groupby-transform() Function

Pandas 常用函数 Pandas Common Functions

groupby.transform()It is a function in Pandas for transforming data after grouping. Unlike aggregate functions,aggregation returns a summary result for each group, while transformation returns a result of the same length as the original data.。

In simple terms,transform()it broadcasts the grouped calculation result back to each row of the original data. This is very useful in scenarios such as calculating the relative position of each member within a group, within-group standardization, and differences from the group mean.

Basic syntax and parameters

transform()It is a member function of the GroupBy object. You need to first usegroupby()grouping before calling it.

Syntax format

GroupBy.transform(func, axis=0, *args, engine=None, engine_kwargs=None, **kwargs)

Parameter description

Parameter	Type	Description	Default value
func	str, list, dict or callable	The transformation function. It can be a built-in function name (such as 'mean', 'sum', 'std'), a list of functions, a dictionary specifying different functions for each column, or a custom function.	None
axis	int	The axis direction for applying the transformation. 0 means row-wise (grouping dimension), 1 means column-wise.	0
args	tuple	Additional positional arguments passed to the transformation function.	()
engine	str	Specifies the computation engine, which can be 'cython' or 'numba'. None means Pandas automatically selects it.	None
engine_kwargs	dict	A dictionary of additional parameters passed to the underlying engine.	None

Return value

Return type：SeriesorDataFrame
Description: Returns a result with the same number of rows as the original data. Each original data row gets the calculation result of its corresponding group.

Examples

Let's thoroughly master, through a series of examples from simple to complex,groupby.transform()the usage.

Example 1: Basic usage - Calculate group mean

transform()The most basic usage is to calculate the mean of each group and then broadcast the result back to each row.

Example

import pandas as pd

# Create student score data
data = {
'Name': ['Zhang San', 'Li Si', 'Wang Wu', 'Zhao Liu', Sun Qi, Zhou Ba, Wu Jiu, Zheng Shi],
Class: ['A', 'A', 'A', 'B', 'B', 'B', 'B', 'A'],
Chinese: [85, 92, 78, 88, 95, 82, 90, 87],
Mathematics: [90, 85, 92, 78, 88, 91, 85, 89]
}

# Create DataFrame
df = pd.DataFrame(data)

print("Student Grades Data:")
print(df)
print()

# Calculate the math average for each class, then broadcast to each row
df['Class Math Average'] = df.groupby(Class)[Mathematics].transform('mean')

# Calculate the difference between each student's score and the class average
df['Difference from average score'] = df[Mathematics] - df['Class Math Average']

print("Data after adding class average and difference:")
print(df)
print()

# Calculate the class average for multiple columns simultaneously
df[['Class Chinese Average', 'Class Math Average 2']] = df.groupby(Class)[[Chinese, Mathematics]].transform('mean')

print(Data after adding the multi-column averages:)
print(df)

Run result:

学生成绩数据：
   姓名  班级  语文  数学
0  张三   A   85   90
1  李四   A   92   85
2  王五   A   78   92
3  赵六   B   88   78
4  孙七   B   95   88
5  周八   B   82   91
6  吴九   B   90   85
7  郑十   A   87   89

添加班级平均分和差值后的数据：
   姓名  班级   语文   数学   班级数学平均分   与平均分差值
0  张三   A   85   90  89.000000   1.000000
1  李四   A   92   85  89.000000  -4.000000
2  王五   A   78   92  89.000000   3.000000
3  赵六   B   88   78  85.666667   -7.666667
4  孙七   B   95   88  85.666667   2.333333
5  周八   B   82   91  85.666667   5.333333
6  吴九   B   90   85  85.666667   -0.666667
7  郑十   A   87   89  89.000000   0.000000

添加多列平均值后的数据：
   姓名  班级   语文   数学  班级语文平均分  班级数学平均分2
0  张三   A   85   90  85.500000         89.0
1  李四   A   92   85  85.500000         89.0
2  王五   A   78   92  85.500000         89.0
3  赵六   B   88   78  88.750000         85.5
4  孙七   B   95   88  88.750000         85.5
5  周八   B   82   91  88.750000         85.5
6  吴九   B   90   85  88.750000         85.5
7  郑十   A   87   89  85.500000         89.0

Code explanation:

transform('mean')Calculate the math mean for each class.
Since Zhang San, Li Si, Wang Wu, and Zheng Shi are all in Class A, the "Class Math Average" they get is the mean of Class A's math scores.
The length of the return value is the same as the number of rows in the original data; this is the key difference between transform and agg.

Example 2: Within-group standardization (Z-Score)

transform()It is often used to calculate within-group standardized values, which is very useful when comparing the relative position of individuals within a group.

Example

import pandas as pd

# Create employee salary data
data = {
'Name': ['Zhang San', 'Li Si', 'Wang Wu', 'Zhao Liu', Sun Qi, Zhou Ba, Wu Jiu, Zheng Shi],
'Department': [Sales, Sales, Sales, 'Technology', 'Technology', 'Technology', Administration, Administration],
'salary': [5000, 8000, 6000, 9000, 12000, 10000, 4500, 7000]
}
df = pd.DataFrame(data)

print("Employee Salary Data:")
print(df)
print()

# Calculate the Z-Score (standardized score) of each employee's salary within the department
# Z-Score = (original value - within-group mean) / within-group standard deviation
df['Department average salary'] = df.groupby('Department')['salary'].transform('mean')
df['Department Salary Standard Deviation'] = df.groupby('Department')['salary'].transform('std')
df['Z-Score'] = (df['salary'] - df['Department average salary']) / df['Department Salary Standard Deviation']

print(Data after adding department statistics and Z-Score:)
print(df)
print()

# Calculate Z-Score at once using a custom function
def z_score(x):
"""Calculate Z-Score"""
return (x - x.mean()) / x.std()

df['Z-Score2'] = df.groupby('Department')['salary'].transform(z_score)

print(Z-Score calculated using a custom function:)
print(df[['Name', 'Department', 'salary', 'Z-Score2']])

Run result:

员工工资数据：
  姓名  部门    工资
0  张三   销售   5000
1  李四   销售   8000
2  王五   销售   6000
3  赵六   技术   9000
4  孙七   技术  12000
5  周八   技术  10000
6  吴九   行政   4500
7  郑十   行政   7000

添加部门统计和Z-Score后的数据：
   姓名   部门    工资    部门平均工资   部门工资标准差    Z-Score
0  张三   销售  5000  6333.333333  1525.755   -0.872871
1  李四   销售  8000  6333.333333  1525.755    1.092091
2  王五   销售  6000  6333.333333  1525.755   -0.218218
3  赵六   技术  9000 10333.333333  1525.755   -0.872871
4  孙七   技术 12000 10333.333333  1525.755    1.092091
5  周八   技术 10000 10333.333333  1525.755   -0.218218
6  吴九   行政   4500  5750.000000  1767.767   -0.707107
7  郑十   行政   7000  5750.000000  1767.767    0.707107

使用自定义函数计算的 Z-Score：
   姓名  部门    工资   Z-Score2
0  张三   销售  5000  -0.872871
1  李四   销售  8000   1.092091
2  王五   销售  6000  -0.218218
3  赵六   技术  9000  -0.872871
4  孙七   技术 12000   1.092091
5  周八   技术 10000  -0.218218
6  吴九   行政  4500  -0.707107
7  郑十   行政  7000   0.707107

Code explanation:

Z-Score indicates the number of standard deviations the original value is from the mean. A positive value means above the mean, and a negative value means below the mean.
Through Z-Score, the influence of salary level differences among departments can be eliminated, allowing comparison of employees' relative positions within their departments.
Zhang San's Z-Score is -0.87, indicating that his salary is about 0.87 standard deviations below the average level of the sales department.

Example 3: Within-group ranking and percentile

transform()It can also be used to calculate within-group rankings, which is very useful in sorting and competitive analysis.

Example

import pandas as pd
import numpy as np

# Create student score data
data = {
'Name': ['Zhang San', 'Li Si', 'Wang Wu', 'Zhao Liu', Sun Qi, Zhou Ba, Wu Jiu, Zheng Shi],
Class: ['A', 'A', 'A', 'B', 'B', 'B', 'B', 'A'],
Mathematics: [90, 85, 92, 78, 88, 91, 85, 89]
}
df = pd.DataFrame(data)

print("Student Grades Data:")
print(df)
print()

# Calculate each student's class ranking (from highest to lowest)
df['班levelinside排name'] = df.groupby(Class)[Mathematics].transform(lambda x: x.rank(ascending=False))

# Calculate each student's percentile (percentage ranking) within the class
df['Percentile within class'] = df.groupby(Class)[Mathematics].transform(lambda x: x.rank(pct=True))

print("Data after adding in-class ranking and percentile:")
print(df)
print()

# Another way to fill missing values: fill using the group mean
data_with_na = {
'Name': ['Zhang San', 'Li Si', 'Wang Wu', 'Zhao Liu', Sun Qi, Zhou Ba],
Class: ['A', 'A', 'A', 'B', 'B', 'B'],
'Score': [85, np.nan, 78, 88, np.nan, 82]
}
df_na = pd.DataFrame(data_with_na)

print(Data containing missing values:)
print(df_na)
print()

# Fill missing values with group mean
df_na['Filled score'] = df_na.groupby(Class)['Score'].transform(lambda x: x.fillna(x.mean()))

print(Result after filling with group mean:)
print(df_na)

Run result:

学生成绩数据：
   姓名  班级  数学
0  张三   A   90
1  李四   A   85
2  王五   A   92
3  赵六   B   78
4  孙七   B   88
5  周八   B   91
6  吴九   B   85
7  郑十   A   89

添加班级内排名和百分位后的数据：
   姓名   班级   数学   班级内排名   班级内百分位
0  张三   A   90   2.0   0.666667
1  李四   A   85   3.0   1.000000
2  王五   A   92   1.0   1.000000
3  赵六   B   78   4.0   0.250000
4  孙七   B   88   2.0   0.500000
5  周八   B   91   1.0   0.750000
6  吴九   B   85   3.0   0.500000
7  郑十   A   89   4.0   0.333333

包含缺失值的数据：
   姓名  班级   成绩
0  张三   A   85.0
1  李四   A    NaN
2  王五   A   78.0
3  赵六   B   88.0
4  孙七   B    NaN
5  周八   B   82.0

使用组内均值填充后的结果：
   姓名  班级   成绩  填充后的成绩
0  张三   A   85.0         85.0
1  李四   A    NaN         81.5
2  王五   A   78.0         78.0
3  赵六   B   88.0         88.0
4  孙七   B    NaN         85.0
5  周八   B   82.0         82.0

Code explanation:

rank(ascending=False)Calculate the ranking, with higher values ranked first.
rank(pct=True)Calculate the percentile ranking, with results between 0 and 1.
transform(lambda x: x.fillna(x.mean()))It is a common pattern for handling missing values, using the mean of each group to fill the missing values within that group.

Example 4: Using a custom transformation function

transform()Custom functions are supported, enabling more complex within-group data transformation logic.

Example

import pandas as pd
import numpy as np

# Create stock price data
data = {
'Stock Code': ['AAPL', 'AAPL', 'AAPL', 'AAPL', 'GOOG', 'GOOG', 'GOOG', 'GOOG'],
'Date': ['2024-01-01', '2024-01-02', '2024-01-03', '2024-01-04'] * 2,
'Close Price': [100, 105, 102, 110, 80, 85, 82, 90]
}
df = pd.DataFrame(data)

print("Stock price data:")
print(df)
print()

# Custom function: calculate the daily return rate for each stock
def calc_daily_return(prices):
"""Calculate daily return rate: (today's price - yesterday's price) / yesterday's price"""
return prices.pct_change()

# Calculate the daily return rate for each stock
df['Daily Return'] = df.groupby('Stock Code')['Close Price'].transform(calc_daily_return)

print("Data after adding daily return:")
print(df)
print()

# Custom function: calculate the normalized value based on the max and min within the group
def normalize(x):
"""Normalization: (x - min) / (max - min)"""
return (x - x.min()) / (x.max() - x.min())

# Calculate the normalized price for each stock
df['Normalized Price'] = df.groupby('Stock Code')['Close Price'].transform(normalize)

print("Data after adding normalized price:")
print(df)

Running result:

股票价格数据：
  股票代码         日期   收盘价
0  AAPL  2024-01-01   100
1  AAPL  2024-01-02   105
2  AAPL  2024-01-03   102
3  AAPL  2024-01-04   110
4  GOOG  2024-01-01    80
5  GOOG  2024-01-02    85
6  GOOG  2024-01-03    82
7  GOOG  2024-01-04    90

添加日收益率后的数据：
   股票代码         日期  收盘价     日收益率
0  AAPL  2024-01-01   100         NaN
1  AAPL  2024-01-02   105    0.050000
2  AAPL  2024-01-03   102   -0.028571
3  AAPL  2024-01-04   110    0.078431
4  GOOG  2024-01-01    80         NaN
5  GOOG  2024-01-02    85    0.062500
6  GOOG  2024-01-03    82   -0.035294
7  GOOG  2024-01-04    90    0.097561

添加归一化价格后的数据：
   股票代码         日期  收盘价     日收益率   归一化价格
0  AAPL  2024-01-01   100         NaN    0.000000
1  AAPL  2024-01-02   105    0.050000    0.500000
2  AAPL  2024-01-03   102   -0.028571    0.200000
3  AAPL  2024-01-04   110    0.078431    1.000000
4  GOOG  2024-01-01    80         NaN    0.000000
5  GOOG  2024-01-02    85    0.062500    0.500000
6  GOOG  2024-01-03    82   -0.035294    0.200000
7  GOOG  2024-01-04    90    0.097561    1.000000

Code explanation:

The transformation function receives a Series (the data within each group).
The length of the returned Series must match the length of the input Series.
The first value is NaN becausepct_change()it calculates a relative change, and the first day has no previous day's data.

Key difference:agg()andtransform()The difference is:agg()returns the aggregated result for each group (number of rows equals the number of groups), whiletransform()returns a result with the same length as the original data. Simple memory: agg compresses data, transform preserves the data shape.

Pandas 常用函数 Pandas common functions

Other extensions

Data Reading and Writing

Data Processing Core

Reference Manual

Statistics and Cases

Pandas groupby-transform() Function

Basic syntax and parameters

Syntax format

Parameter description

Return value

Examples

Example 1: Basic usage - Calculate group mean

Example

Example 2: Within-group standardization (Z-Score)

Example

Example 3: Within-group ranking and percentile

Example

Example 4: Using a custom transformation function

Example