Pandas df.filter() Function

Pandas Common Functions

filter()is a function in Pandas used to filter columns (rather than rows). It can select specific columns based on column names, label patterns, or regular expressions. Different fromloc[]andiloc[]other functions,filter()it is specifically designed for column selection, which is very useful when dealing with datasets with many columns.

In data analysis, sometimes we only need to focus on specific columns, such as selecting only numeric columns, only columns starting with a specific prefix, or only columns containing a specific keyword.filter()It is designed for these scenarios.

Basic syntax and parameters

filter()is a method of DataFrame and Series (for Series, only the items parameter is supported), used to filter columns based on conditions.

Syntax format

DataFrame.filter(items=None, like=None, regex=None, axis=None)

Parameter description

Parameter Type Required Description Default value
items list Optional Directly specify a list of column names to select matching columns. None
like str Optional Column names containing the specified string; supports fuzzy matching. None
regex str Optional Regular expression pattern to match column names. None
axis int or str Optional The axis to filter. 0 or 'index' means rows, 1 or 'columns' means columns. 1 or 'columns'

Return value description

  • Return type: Returns a new DataFrame containing the filtered columns.
  • Column filtering: By default,filter()it acts on columns (axis=1).
  • All rows are retained: Filtering only affects columns; row data remains unchanged.

Examples

Let's fully master the usage offilter()it through rich examples.

Example 1: Basic usage - using the items parameter

Directly specify a list of column names to select specific columns.

Example

import pandas as pd

# Create a sample DataFrame
data = {
    'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
    'age': [18, 19, 17, 18, 20],
    'score': [85, 92, 78, 90, 88],
    'grade': ['A', 'A', 'B', 'A', 'B'],
    'city': ['Beijing', 'Shanghai', 'Beijing', 'Guangzhou', 'Shanghai']
}
df = pd.DataFrame(data)

print("Original DataFrame:")
print(df)
print()

# Use the items parameter to select specific columns
print("Select the name and age columns:")
print(df.filter(items=['name', 'age']))
print()

# Select a single column (returns a DataFrame)
print("Select only the score column:")
print(df.filter(items=['score']))

Output:

原始 DataFrame:
      name  age  score grade     city
0    Alice   18     85     A   Beijing
1      Bob   19     92     A  Shanghai
2  Charlie   17     78     B   Beijing
3    David   18     90     A  Guangzhou
4      Eve   20     88     B  Shanghai

选择 name 和 age 列:
      name  age
0    Alice   18
1      Bob   19
2  Charlie   17
3    David   18
4      Eve   20

只选择 score 列:
   score
0     85
1     92
2     78
3     90
4     88

Code explanation:

  1. filter(items=['name', 'age'])Selects the specified columns and returns a new DataFrame containing these columns.
  2. All rows are retained; only specific columns are selected.
  3. If a specified column name does not exist, a KeyError is raised.

Example 2: Using the like parameter - fuzzy matching

likeThe `like` parameter allows us to filter columns whose names contain a specified string. This is very suitable for datasets with regular column names.

Example

import pandas as pd

# Create a DataFrame with regular column names
data = {
    'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
    'age_2020': [18, 19, 17, 18, 20],
    'age_2021': [19, 20, 18, 19, 21],
    'age_2022': [20, 21, 19, 20, 22],
    'score_2020': [85, 92, 78, 90, 88],
    'score_2021': [87, 94, 80, 92, 90],
    'score_2022': [89, 96, 82, 94, 92],
    'city': ['Beijing', 'Shanghai', 'Beijing', 'Guangzhou', 'Shanghai']
}
df = pd.DataFrame(data)

print("Original DataFrame:")
print(df)
print()

# Select all columns containing "age"
print("Columns containing 'age':")
print(df.filter(like='age'))
print()

# Select all columns containing "score"
print("Columns containing 'score':")
print(df.filter(like='score'))
print()

# Select all columns containing "2021"
print("Columns containing '2021':")
print(df.filter(like='2021'))

Output:

原始 DataFrame:
      name  age_2020  age_2021  age_2022  score_2020  score_2021  score_2022     city
0    Alice       18       19       20         85         87         89   Beijing
1      Bob       19       20       21         92         94         96  Shanghai
2  Charlie       17       18       19         78         80         82   Beijing
3    David       18       19       20         90         92         94  Guangzhou
4      Eve       20       21       22         88         90         92  Shanghai

包含 'age' 的列:
   age_2020  age_2021  age_2022
0       18       19       20
1       19       20       21
2       17       18       19
3       18       19       20
4       20       21       22

包含 'score' 的列:
   score_2020  score_2021  score_2022
0         85         87         89
1         92         94         96
2         78         80         82
3         90         92         94
4         88         90         92

包含 '2021' 的列:
   age_2021  score_2021
0       19       87
1       20       94
2       18       80
3       19       92
4       21       90

Code explanation:

  • filter(like='age')Selects all columns whose names contain "age".
  • Fuzzy matching does not require knowing the full column name; just provide a partial matching string.
  • This is especially useful when dealing with columns that follow similar naming conventions (e.g., columns organized by year).

Example 3: Using the regex parameter - regular expression matching

For more complex column name filtering needs, regular expressions can be used.

Example

import pandas as pd

# Create complex column names
data = {
    'id': [1, 2, 3, 4, 5],
    'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
    'age_2020': [18, 19, 17, 18, 20],
    'age_2021': [19, 20, 18, 19, 21],
    'score_math': [85, 92, 78, 90, 88],
    'score_english': [87, 94, 80, 92, 90],
    'score_science': [89, 96, 82, 94, 92],
    'city': ['Beijing', 'Shanghai', 'Beijing', 'Guangzhou', 'Shanghai']
}
df = pd.DataFrame(data)

print("Columns of the original DataFrame:")
print(df.columns.tolist())
print()

# Select columns starting with "score_"
print("Columns starting with 'score_':")
print(df.filter(regex='^score_'))
print()

# Select column names containing digits
print("Column names containing digits:")
print(df.filter(regex='\d'))  # \d matches digits
print()

# Select columns containing both "age" and "202"
print("Columns containing both 'age' and '202':")
print(df.filter(regex='age.*202|202.*age'))
print()

# Select columns starting with a letter and not containing underscores
print("Columns starting with a letter and without underscores:")
print(df.filter(regex='^[a-zA-Z]+$'))

Output:

原始 DataFrame 的列:
['id', 'name', 'age_2020', 'age_2021', 'score_math', 'score_english', 'score_science', 'city']

以 'score_' 开头的列:
   score_math  score_english  score_science
0         85         87         89
1         92         94         96
2         78         80         82
3         90         92         94
4         88         90         92

包含数字的列名:
   age_2020  age_2021  score_math  score_english  score_science
0       18       19         85         87         89
1       19       20         92         94         96
2       17       18         78         80         82
3       18       19         90         92         94
4       20       21         88         90         92

包含 'age' 和 '202' 的列:
   age_2020  age_2021
0       18       19
1       19       20
2       17       18
3       18       19
4       20       21

以字母开头且不含下划线的列:
      name   city
0    Alice  Beijing
1      Bob  Shanghai
2  Charlie  Beijing
3    David  Guangzhou
4    Eve  Shanghai

Code explanation:

  1. filter(regex='^score_')Use^to match columns starting with "score_".
  2. filter(regex='\d')Use\dto match column names containing digits.
  3. Regular expressions provide the greatest flexibility and can handle various complex matching needs.

Example 4: Using the axis parameter to filter rows

Althoughfilter()it is mainly used for columns, through theaxisparameter it can also be used to filter rows.

Example

import pandas as pd

# Create a DataFrame with a named index
data = {
    'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
    'age': [18, 19, 17, 18, 20],
    'score': [85, 92, 78, 90, 88]
}
df = pd.DataFrame(data, index=['row1', 'row2', 'row3', 'row4', 'row5'])

print("DataFrame with named index:")
print(df)
print()

# Use axis='index' to filter rows (using like)
print("Rows whose index contains 'row':")
print(df.filter(like='row', axis='index'))
print()

# Use items to filter rows with specific indices
print("Rows with index 'row1' and 'row3':")
print(df.filter(items=['row1', 'row3'], axis='index'))
print()

# Use regex to filter rows
print("Rows whose index matches the regex (row)
print(df.filter(regex='row[135]', axis='index'))

Output:

带命名索引的 DataFrame:
      name   age  score
row1  Alice   18     85
row2    Bob   19     92
row3  Charlie   17     78
row4  David   18     90
row5    Eve   20     88

索引包含 'row' 的行:
      name   age  score
row1  Alice   18     85
row2    Bob   19     92
row3  Charlie   17     78
row4  David   18     90
row5    Eve   20     88

索引为 'row1' 和 'row3' 的行:
      name   age  score
row1  Alice   18     85
row3  Charlie   17     78

索引匹配正则表达式（row[135]）的行:
      name   age  score
row1  Alice   18     85
row3  Charlie   17     78
row5    Eve   20     88

Code explanation:

  • axis='index'oraxis=0Tellsfilter()df.filter() to filter row indices.
  • like、items、regexThe parameters have the same usage in row filtering as in column filtering.
  • This is very useful when you need to filter specific rows based on index names.

Example 5: Combining with other functions

filter()It can be combined with other Pandas functions to achieve powerful data selection capabilities.

Example

import pandas as pd
import numpy as np

# Create a large DataFrame
np.random.seed(42)
df = pd.DataFrame(np.random.randn(5, 10), columns=[
    'A', 'B', 'C', 'D', 'E',
    'AA', 'BB', 'CC', 'DD', 'EE'
])

print("Columns of the original DataFrame:")
print(df.columns.tolist())
print()

# Filter single-letter columns and perform a calculation
single_letter_cols = df.filter(regex='^[A-E]$')
print("Mean of single-letter columns (A-E):")
print(single_letter_cols.mean())
print()

# Filter double-letter columns and perform a calculation
double_letter_cols = df.filter(regex='^[A-E]{2}$')
print("Mean of double-letter columns (AA-EE):")
print(double_letter_cols.mean())
print()

# First filter columns, then select rows
print("First 3 rows of double-letter columns:")
print(df.filter(regex='^[A-E]{2}$').head(3))

Output:

原始 DataFrame 的列:
['A', 'B', 'C', 'D', 'E', 'AA', 'BB', 'CC', 'DD', 'EE']

单字母列（A-E）的均值:
A    0.336744
B    0.128223
C   -0.234007
D   -0.347540
E   -0.197939
dtype: float64

双字母列（AA-EE）的均值:
AA    0.530256
BB   -0.671336
CC    0.506853
DD   -0.443588
EE   -0.456486
dtype: float64

双字母列的前 3 行:
         AA        BB        CC        DD        EE
0 -0.013497 -1.174139  0.214864  1.550698  0.375698
1 -1.368593  0.746580  0.669383 -0.717552 -1.159950
2  0.602619 -1.700736 -0.201647 -0.605166 -0.012750

Code explanation:

  • filter()It can be chained with other DataFrame methods.
  • First filter specific columns, then perform statistical calculations, improving code readability.
  • This is especially useful when processing wide tables.

Notes

  • items`items` is used to precisely specify column names, `like` is used for fuzzy matching, and `regex` is used for regular expression matching. Choose the appropriate method according to actual needs.likeWhen processing large datasets, first useregexto filter the needed columns, which can reduce the amount of data for subsequent calculations and improve performance.
  • By default, it acts on columns (axis=1), and can be used tofilter()change the filtering direction.
  • filter()Noteaxis='index'oraxis=0These three parameters are mutually exclusive and cannot be used simultaneously.
  • Tip:items、like、regexIt is a powerful tool for processing wide tables (datasets with many columns). Especially during the data exploration stage, if you only want to focus on a certain type of information (such as all numeric columns, all columns starting with a certain prefix), using

it can quickly complete column filtering.filter()Summaryfilter()Columns can be quickly filtered.

Summary

filter()It is a function in Pandas specifically used for filtering columns (and rows). It provides three filtering methods: exact specification (items), fuzzy matching (like), and regular expression matching (regex).

The main application scenarios of this function include: filtering specific columns, handling columns with regular naming conventions, and dynamically selecting columns for analysis. When used in combination with other Pandas functions, it can significantly improve data processing efficiency and code readability.

Pandas Common Functions

Other Extensions