Python pickle: Object Serialization, Protocols & Security

The pickle module serializes arbitrary Python object hierarchies into binary byte streams (pickling) and reconstructs them into identical in-memory objects (unpickling).

Binary Protocols vs. Portable JSON: While json provides human-readable text exchange across multiple languages, pickle serializes Python-specific data types like sets, complex numbers, tuples, and custom class instances natively. Always open target files using binary mode ('wb' and 'rb') via open(), and inspect reconstructed attributes with type() or len().

1. Syntax & Core Method Specifications 🔝

The standard pickle module revolves around four primary functions operating on files and byte streams:

# Core Methods in Python's standard pickle module
import pickle

pickle.dump(obj, file, protocol=None)  # Write object to open binary file ('wb')
obj = pickle.load(file)               # Read object from open binary file ('rb')
byte_data = pickle.dumps(obj)         # Serialize object to bytes in memory
obj = pickle.loads(byte_data)         # Reconstruct object from bytes in memory

2. File-Based Serialization: dump() and load() 🔝

When pickling to disk, files must be opened in binary mode ('wb' for writing, 'rb' for reading). Using text mode ('w' or 'r') causes a TypeError:

# 1. File-Based Serialization with dump() and load()
import pickle

records = {
    'names': ['Ravi', 'Raju', 'Alex'],
    'ids': [101, 102, 103],
    'scores': {'math': [88, 92, 79], 'english': [85, 78, 90]}
}

# Writing binary stream to disk ('wb' required)
with open('student_cache.pkl', 'wb') as f_out:
    pickle.dump(records, f_out, protocol=pickle.HIGHEST_PROTOCOL)

# Reading binary stream from disk ('rb' required)
with open('student_cache.pkl', 'rb') as f_in:
    loaded = pickle.load(f_in)

print("Reconstructed type:", type(loaded))
print("Student names:     ", loaded['names'])
print("Equal to original? ", loaded == records)

Terminal Output:

Reconstructed type: <class 'dict'>
Student names:      ['Ravi', 'Raju', 'Alex']
Equal to original?  True

3. In-Memory Byte Serialization: dumps() and loads() 🔝

When communicating across network sockets, distributed task queues, or database BLOB columns, dumps() and loads() operate directly on byte streams without touching disk:

# 2. In-Memory Byte Serialization with dumps() and loads()
import pickle

session = {
    'user_id': 9481,
    'roles': {'admin', 'editor'},
    'active': True
}

# dumps() converts Python objects to bytes without touching disk
raw_bytes = pickle.dumps(session)
print("Byte stream length:  ", len(raw_bytes), "bytes")

# loads() restores the in-memory Python object
restored = pickle.loads(raw_bytes)
print("Restored user ID:    ", restored['user_id'])
print("Restored roles set:  ", restored['roles'])

Terminal Output:

Byte stream length:   68 bytes
Restored user ID:     9481
Restored roles set:   {'admin', 'editor'}

4. Pickling Custom Classes & Managing State (__getstate__) 🔝

By default, pickle serializes an instance's __dict__. You can customize which attributes are pickled by implementing __getstate__() and __setstate__():

# 3. Serializing Custom Classes and Managing State
import pickle

class ModelCheckpoint:
    def __init__(self, name: str, weights: list):
        self.name = name
        self.weights = weights
        self.scratchpad = "Ephemeral" # Exclude from disk

    def __getstate__(self):
        """Customize dictionary state before pickling."""
        state = self.__dict__.copy()
        state.pop('scratchpad', None)
        return state

    def __setstate__(self, state):
        """Restore state upon unpickling."""
        self.__dict__.update(state)
        self.scratchpad = "Re-initialized"

model = ModelCheckpoint("ResNet_V2", [0.45, -0.12, 0.88])
restored = pickle.loads(pickle.dumps(model))

print("Model name: ", restored.name)
print("Weights:    ", restored.weights)
print("Scratchpad: ", restored.scratchpad)

Terminal Output:

Model name:  ResNet_V2
Weights:     [0.45, -0.12, 0.88]
Scratchpad:  Re-initialized

5. Security Vulnerabilities & Arbitrary Code Execution Exploit 🔝

An attacker can weaponize a pickled payload by implementing the __reduce__() method. When unpickled via pickle.loads(), Python executes the specified callable on the server immediately:

# 4. Security Risks: Arbitrary Code Execution via __reduce__
import pickle

class ExploitPayload:
    def __reduce__(self):
        return (eval, ("print('ALERT: Arbitrary code executed on host!')",))

exploit_bytes = pickle.dumps(ExploitPayload())
pickle.loads(exploit_bytes)

Terminal Output:

ALERT: Arbitrary code executed on host!

6. Pickle vs. JSON vs. Marshal: Architectural Comparison 🔝

7. Production Recipes: Safe Unpickler Whitelist 🔝

Enforcing a secure deserialization whitelist using pickle.Unpickler subclassing:

# 5. Production Recipe: Safe Unpickler Whitelist
import pickle
import io

class SafeUnpickler(pickle.Unpickler):
    ALLOWED_TYPES = {'dict', 'list', 'set', 'tuple', 'int', 'float', 'str', 'bool'}

    def find_class(self, module, name):
        if module == "builtins" and name in self.ALLOWED_TYPES:
            return getattr(__builtins__, name) if isinstance(__builtins__, dict) else getattr(__builtins__, name)
        raise pickle.UnpicklingError(f"Blocked unauthorized class: '{module}.{name}'")

# Safe data deserializes cleanly
safe_data = pickle.dumps({"user": "Alex", "points": 100})
result = SafeUnpickler(io.BytesIO(safe_data)).load()
print("Safe load result:", result)

Terminal Output:

Safe load result: {'user': 'Alex', 'points': 100}

8. Common Developer Pitfalls & Traps 🔝

Be aware of unpicklable types and text-mode file exceptions:

# 6. Common Pitfalls: Lambdas and Text Mode File Handles
import pickle

# Trap 1: Lambda functions cannot be pickled
try:
    pickle.dumps(lambda x: x * x)
except Exception as err:
    print("Lambda error:   ", type(err).__name__)

# Trap 2: Opening file in text mode instead of binary
try:
    with open('test.txt', 'w') as f: # Wrong: must be 'wb'
        pickle.dump([1, 2, 3], f)
except Exception as err:
    print("Text mode error:", type(err).__name__)

Terminal Output:

Lambda error:    PicklingError
Text mode error: TypeError

9. Interactive Practice & Companion Jupyter Notebook 🔝

Practice serialization protocols, custom class state management, and restricted unpickling in our companion Jupyter Notebook:

Self-Verifying Assertion Suite

Run these unit checks directly in Python or your local environment to verify pickle mechanics:

# 7. Self-verifying unit test suite for pickle
import pickle

# Test 1: Complex data structure round-trip equality
data = {"list": [1, 2, 3], "set": {10, 20}, "tuple": ("a", True, None)}
encoded = pickle.dumps(data)
decoded = pickle.loads(encoded)
assert decoded == data
assert isinstance(decoded["set"], set)

# Test 2: Protocol 5 compatibility
proto5 = pickle.dumps([1, 2, 3], protocol=pickle.HIGHEST_PROTOCOL)
assert pickle.loads(proto5) == [1, 2, 3]

print("All pickle unit assertions successfully verified!")

Terminal Output:

All pickle unit assertions successfully verified!

10. Frequently Asked Questions (FAQ) 🔝

Further Reading & Related Tutorials 🔝

• Python json Module — Standard cross-language serialization for web services and APIs.
• Python open() Function — Opening files in binary read ('rb') and write ('wb') modes.
• Python bytes Object — Working with raw binary byte sequences and memory buffers.
• Python Pickle with MySQL — Storing serialized Python binary objects in database BLOB columns.
• Python Dictionaries — Complete reference for nested mapping structures.
• Python Modules Overview — Managing imports and modular code organization.

Subhendu Mohapatra

Author & Instructor at plus2net

I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.

About the author   Watch on YouTube

Connect: