Performance and Scalability

This guide provides best practices for optimizing py3plex performance and scaling to large networks.

Memory Management

Sparse vs Dense Matrices

Always use sparse matrices for large networks:

# Good: Sparse representation (default)
supra_adj = network.get_supra_adjacency_matrix(sparse=True)

# Bad: Dense representation for large networks
# supra_adj = network.get_supra_adjacency_matrix(sparse=False)

Memory requirements:

Sparse: \(O(m)\) where \(m\) is number of edges
Dense: \(O(n^2)\) where \(n\) is total nodes across layers

Rule of thumb: Use sparse for networks with >1,000 nodes or density <10%

Matrix Construction Optimization

Avoid unnecessary matrix constructions:

# Bad: Repeated construction
for iteration in range(100):
    adj = network.get_supra_adjacency_matrix()
    # ... use adj ...

# Good: Construct once
adj = network.get_supra_adjacency_matrix()
for iteration in range(100):
    # ... use adj ...

Efficient Network Construction

Batch Operations

Add nodes and edges in batches rather than one at a time:

# Bad: Individual additions
for edge in edge_list:
    network.add_edges([[edge[0], edge[1], edge[2], edge[3], edge[4]]], input_type='list')

# Good: Batch addition
network.add_edges(edge_list, input_type='list')

Pre-allocate Data Structures

When possible, provide all data at construction time:

# Efficient: Provide all edges at once
network = multinet.multi_layer_network()
network.add_edges(all_edges, input_type='list')

# Less efficient: Incremental addition
network = multinet.multi_layer_network()
for edge_batch in batches:
    network.add_edges(edge_batch, input_type='list')

Algorithm Selection

Time Complexity Guidelines

Choose algorithms based on network size:

Network Size	Community Detection	Centrality	Embeddings
Small (<1K nodes)	Any	Any (including betweenness)	Node2Vec (full)
Medium (1K-10K)	Louvain	Degree, PageRank	Node2Vec (reduced walks)
Large (10K-100K)	Louvain, Label Prop	Degree only	DeepWalk (sampling)
Very Large (>100K)	Label Prop	Degree	Skip or sample

Avoid Expensive Operations

O(n²) or worse complexity:

Full betweenness centrality
All-pairs shortest paths
Closeness centrality on large networks
Dense matrix operations

Alternative approaches:

# Instead of betweenness on full network
# betweenness = calc.multilayer_betweenness_centrality()  # O(nm)

# Use degree as proxy or sample
degree = calc.overlapping_degree_centrality()  # O(n+m)

# Or sample nodes
import random
sample_nodes = random.sample(list(network.get_nodes()), k=100)
# Compute betweenness on induced subgraph

Parallel Processing

Using Joblib for Parallelization

py3plex operations can often be parallelized:

from joblib import Parallel, delayed
from py3plex.algorithms.statistics import multilayer_statistics as mls

# Parallel layer statistics
layers = network.get_layers()
densities = Parallel(n_jobs=-1)(
    delayed(mls.layer_density)(network, layer)
    for layer in layers
)

NumPy Vectorization

Use vectorized operations instead of loops:

import numpy as np

# Bad: Python loop
degrees = {}
for node in nodes:
    degrees[node] = sum(1 for neighbor in G.neighbors(node))

# Good: Vectorized
degrees = dict(G.degree())  # NetworkX optimized

# Even better for matrices
adj_matrix = network.get_adjacency_matrix(layer='layer1')
degrees = np.asarray(adj_matrix.sum(axis=1)).flatten()

Visualization Optimization

Layout Computation

Layout algorithms have varying complexity:

Layout	Complexity	Best For
Random	O(n)	Testing, very large networks
Circular	O(n)	Small networks with clear order
Spring/Force	O(n² × iterations)	<1,000 nodes
ForceAtlas2	O(n log n × iterations)	1,000-10,000 nodes

Optimization strategies:

# 1. Reduce iterations for approximate layout
pos = dm.compute_layout(G, algorithm='force', iterations=50)  # Default is often 500

# 2. Precompute and save layouts
import pickle
pos = dm.compute_layout(G, algorithm='force')
with open('layout.pkl', 'wb') as f:
    pickle.dump(pos, f)

# Load saved layout
with open('layout.pkl', 'rb') as f:
    pos = pickle.load(f)

Reduce Visual Elements

For large networks, reduce the number of rendered elements:

# 1. Sample edges
import random
all_edges = list(network.get_edges())
sampled_edges = random.sample(all_edges, k=min(1000, len(all_edges)))

# 2. Hide labels
draw_network(network, show_labels=False)

# 3. Use smaller node/edge sizes
draw_network(network, node_size=1, edge_width=0.1)

Use Appropriate Visualization Type

Match visualization to network size:

num_nodes = len(network.get_nodes())

if num_nodes < 100:
    # Full network with labels
    draw_multilayer_default([network], display=True, show_labels=True)

elif num_nodes < 1000:
    # Network without labels, force layout
    hairball_plot(network.core_network, layout_algorithm='force')

elif num_nodes < 10000:
    # Diagonal projection
    draw_multilayer_default([network], layout_style='diagonal')

else:
    # Matrix visualization
    adj = network.get_supra_adjacency_matrix(sparse=True)
    plt.spy(adj, markersize=0.1)
    plt.show()

I/O Optimization

Efficient File Formats

Choose appropriate file formats:

Format	Load Speed	File Size	Best For
gpickle	Fast	Small	Temporary storage, Python only
GraphML	Medium	Large (XML)	Interchange, attributes
Edge list	Fast	Small	Simple networks, no attributes
GML	Slow	Medium	Legacy compatibility

Recommendation:

# For temporary storage (fastest)
network.save_network("cache.gpickle", output_type="gpickle")
network = multinet.multi_layer_network().load_network("cache.gpickle", input_type="gpickle")

# For interchange (portable)
network.save_network("data.graphml", output_type="graphml")

Streaming Large Files

For very large networks, process incrementally:

# Read edge list in chunks
network = multinet.multi_layer_network()

chunk_size = 10000
with open('large_edgelist.txt', 'r') as f:
    chunk = []
    for line in f:
        chunk.append(line.strip().split())
        if len(chunk) >= chunk_size:
            network.add_edges(chunk, input_type='list')
            chunk = []

    # Add remaining edges
    if chunk:
        network.add_edges(chunk, input_type='list')

Benchmarking and Profiling

Time Your Code

Use timing utilities to identify bottlenecks:

import time

start = time.time()
communities = community_louvain.best_partition(network.core_network)
elapsed = time.time() - start
print(f"Community detection took {elapsed:.2f} seconds")

Memory Profiling

Monitor memory usage:

import tracemalloc

tracemalloc.start()

# Your code here
adj = network.get_supra_adjacency_matrix(sparse=True)

current, peak = tracemalloc.get_traced_memory()
print(f"Current memory: {current / 1024**2:.2f} MB")
print(f"Peak memory: {peak / 1024**2:.2f} MB")
tracemalloc.stop()

Line Profiling

For detailed profiling:

pip install line_profiler

# Add @profile decorator to functions
# Run with:
kernprof -l -v script.py

Caching Strategies

Function-Level Caching

Cache expensive computations:

from functools import lru_cache

@lru_cache(maxsize=128)
def expensive_centrality(network_hash, layer):
    """Cached centrality computation."""
    # Compute centrality
    return centrality_values

Network-Level Caching

Save intermediate results:

import os
import pickle

cache_file = 'communities_cache.pkl'

if os.path.exists(cache_file):
    with open(cache_file, 'rb') as f:
        communities = pickle.load(f)
else:
    communities = community_louvain.best_partition(network.core_network)
    with open(cache_file, 'wb') as f:
        pickle.dump(communities, f)

Best Practices Summary

Network Size Guidelines

< 1,000 nodes:

Any algorithm works
Full visualization with labels
Path-based centrality is fine
Dense matrices acceptable

1,000 - 10,000 nodes:

Use Louvain for communities
Degree/PageRank for centrality
Sparse matrices essential
Diagonal projection visualization
No labels in visualization

10,000 - 100,000 nodes:

Label propagation for communities
Degree centrality only
Batch operations critical
Matrix visualization
Sample for exploration

> 100,000 nodes:

Degree-based measures only
Label propagation
Incremental processing
Sampling required
Consider distributed tools for some tasks

Memory Checklist

✅ Use sparse matrices (sparse=True)

✅ Batch add edges (not one at a time)

✅ Avoid repeated matrix construction

✅ Use generators instead of lists where possible

✅ Clear unused variables (del variable)

✅ Monitor memory with tracemalloc

Speed Checklist

✅ Choose appropriate algorithms for network size

✅ Avoid O(n²) operations on large networks

✅ Use NumPy vectorization

✅ Cache expensive computations

✅ Precompute layouts

✅ Sample for exploration

✅ Profile to find bottlenecks

Benchmarking Examples

Run the built-in benchmarks:

cd benchmarks
python config_benchmark.py

Create your own benchmarks:

from py3plex.core import multinet
import time

# Create networks of different sizes
sizes = [100, 500, 1000, 5000]
times = []

for size in sizes:
    network = multinet.multi_layer_network()

    # Generate random edges
    import random
    edges = []
    for i in range(size * 5):  # ~5 edges per node
        edges.append([
            f'n{random.randint(0, size)}', 'L1',
            f'n{random.randint(0, size)}', 'L1', 1
        ])

    start = time.time()
    network.add_edges(edges, input_type='list')
    elapsed = time.time() - start

    times.append((size, elapsed))
    print(f"Size {size}: {elapsed:.3f}s")

Next Steps

Algorithm Selection Guide - Algorithm selection guide
Quick Start Guide - Usage examples
Multilayer Network Centrality Measures - Centrality tutorial
Benchmark scripts in benchmarks/ directory

Getting Help

For performance issues:

Check this guide first
Run built-in benchmarks
Profile your code
Open an issue with profiling results