Data cleaning by using Pandas methods

Data we get from different sources may not be in the form or format for direct use in our applications. We need to correct these data by using various methods available in Pandas.

thresh	Using thresh option of dropna() removing rows and columns
dropna	Delete NaN rows or columns
df.drop_duplicates	Delete Duplicate rows from DataFrame
duplicated	duplicate rows from DataFrame
Series.duplicated	duplicate value from Series
Series.drop_duplicates	Delete Duplicate data from Series
replace	Replace data
notnull	Check for Not Null and NaN data
fillna	Fill NA/NaN values
Email	Validate Email address

dtypes	Pandas Data types
select_dtypes	Subset of DataFrame based on data type

In Pandas missing data is represented by two value: None and NaN : NaN (an acronym for Not a Number)
It is generally recommended to use NaN (np.nan) instead of None for missing or undefined values in numerical columns of a Pandas DataFrame. This allows Pandas to retain the original data type (such as int), while treating the NaN values as missing data. This ensures consistent data types and enables proper handling of missing values using Pandas' functions and methods designed for numerical data.

Removing comma from a string column

df['p_view']=df['p_view'].apply(lambda x: x.replace(',',''))

Converting to integer data type

df['p_view'] = df['p_view'].astype('int')

A regex which selects only characters in UTF-8, removing the rest, for each field in the dataframe.

df.replace({r'[^\x00-\x7F]+':''}, regex=True, inplace=True)

Pandas columns() add_prefix() add_suffix()
Pandas Data Analysis

Numpy arrays Python & MySQL Python- Tutorials

Subhendu Mohapatra

Author & Instructor at plus2net

I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.

About the author Watch on YouTube

Connect:

Subscribe to our YouTube Channel here

Cleaning Data

Subhendu Mohapatra