Python Logging Best Practices: How to Write Clean, Useful Logs

Logging is one of the most important parts of maintaining a Python application, especially once a project moves beyond local development. When an application fails, behaves unexpectedly, or produces incorrect results, logs can provide the information needed to understand what happened without reproducing the problem manually.

Python provides a built-in logging module that supports different log levels, handlers, formatters, filters, and logging destinations.

However, simply adding logging.info() throughout a codebase does not automatically produce useful logs. Poorly designed logging can create noisy output, expose sensitive information, make debugging harder, or generate large amounts of unnecessary data. Good logging is therefore less about producing more messages and more about producing the right information at the right level and in the right format.

In this article, we will look at practical Python logging best practices, including how to configure the logging module, choose appropriate log levels, structure messages, handle exceptions, avoid common mistakes, and design logs that remain useful as an application grows.

Understanding Python Logging

Python's standard logging module provides a flexible framework for recording application events. Instead of relying on print() statements, developers can create log records containing information about what happened, how important the event is, and where it occurred.

A basic logging setup can be created with logging.basicConfig(). The following example configures the application to display informational messages and above while including the timestamp, log level, logger name, and message in each record.

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)

logging.info("Application started")
logging.warning("Configuration file is missing")

The configuration determines which messages are displayed and how they are formatted. With INFO as the logging level, informational messages are recorded along with higher-severity WARNING, ERROR, and CRITICAL messages. The timestamp and severity make the output considerably more useful than an unstructured print() statement. Python provides five commonly used logging levels:

• DEBUG — detailed information primarily useful during development and troubleshooting.

• INFO — confirmation that an expected application event occurred.

• WARNING — an unexpected situation that does not necessarily prevent the application from continuing.

• ERROR — a failure that prevents a particular operation from completing.

• CRITICAL — a serious failure that may prevent the application or a major component from continuing.

Choosing the correct level is important. A successful database connection might be an INFO event, while a failed request could be an ERROR. A missing optional configuration value might be a WARNING, while detailed information about an internal calculation may belong at DEBUG.

One common mistake is treating logging levels simply as different versions of the same message. They should instead communicate the significance of an event. This allows developers and monitoring systems to filter logs based on severity.

Use Module-Level Loggers

For applications containing multiple Python modules, using the root logger everywhere can make logs difficult to trace. A better approach is to create a logger for each module using logging.getLogger(__name__).

This automatically gives the logger the module's name, making it easier to determine where a log record originated.

import logging

logger = logging.getLogger(__name__)

def load_user(user_id):
    logger.info("Loading user %s", user_id)
    return {"id": user_id}

When this module generates a log message, the logger name can identify the source module. This becomes particularly useful in larger applications containing services, database modules, API clients, background workers, and other components.

Using name also allows the application's logging configuration to remain centralized while individual modules simply obtain their own logger.

Configure Logging Centrally

Libraries and application modules should generally avoid configuring global logging behavior themselves. Instead, the application's entry point should configure logging, while individual modules create loggers and emit records.

For example, a larger application might have a structure similar to:

project/
├── app.py
├── database.py
├── services.py
└── api.py

Each module can define its own logger:

import logging

logger = logging.getLogger(__name__)

The application's main entry point can then configure the logging system.

This separation keeps logging responsibilities clear. Individual modules decide what should be logged, while the application decides where those logs go and how they should be formatted. It also makes it easier to change the logging configuration without modifying every module.

Writing Clean and Useful Log Messages

The usefulness of a log depends heavily on the information contained in the message. A message such as Something went wrong provides very little information to someone investigating a production problem.

A useful log should provide enough context to understand the event without requiring the developer to inspect the source code immediately.

Compare these two messages:

logging.error("Request failed")

and:

logger.error(
    "Failed to fetch order %s for customer %s",
    order_id,
    customer_id
)

The second message communicates what operation failed and identifies the relevant entities. This additional context can significantly reduce the time required to investigate a problem. At the same time, logs should not become unnecessarily verbose. A message should explain the event rather than reproduce the entire state of the application.

Use Logging Arguments Instead of String Interpolation

Python's logging API supports arguments that are formatted when the log message is actually emitted. This is preferable to constructing the final string before passing it to the logger.

For example:

logger.debug("Processing file %s", filename)

rather than:

logger.debug(f"Processing file {filename}")

The first approach allows the logging system to avoid formatting the message when the corresponding log level is disabled. This can be useful when debug logging contains expensive values or occurs frequently. It also follows the intended interface of Python's logging module.

Avoid Logging Sensitive Information

Logs frequently end up in centralized systems, cloud storage, monitoring platforms, or long-term archives. Developers should therefore treat logs as potentially persistent data.

Passwords, authentication tokens, API keys, credit card information, private credentials, and other sensitive values should never be written directly to logs.

For example, this should be avoided:

logger.info("User login: username=%s password=%s", username, password)

Instead, log the event without exposing the credential:

logger.info("Login attempt for user %s", username)

Even usernames, email addresses, IP addresses, or other identifiers may require careful handling depending on the application's requirements and applicable privacy rules.

A useful rule is to ask whether the information is genuinely required to diagnose the event. If it is not necessary, it generally does not belong in the log.

Give Logs Enough Context

Context is especially important in applications handling multiple requests simultaneously. A message such as Payment failed may be almost useless when hundreds of requests are being processed every second.

Adding an identifier can make the event traceable:

logger.error(
    "Payment failed for order_id=%s transaction_id=%s",
    order_id,
    transaction_id
)

Request IDs, transaction IDs, job IDs, or other non-sensitive correlation identifiers can help developers connect related events across different parts of an application.

The goal is to make a log understandable when viewed independently, without requiring the reader to guess which request or operation produced it.

Exception Logging and Structured Logging

Exception handling is one of the areas where logging provides the most value. When an unexpected exception occurs, the log should normally preserve the traceback rather than recording only the exception message.

Python's logger.exception() method is designed specifically for this situation. It should be called inside an exception handler.

try:
    result = process_file("data.csv")
except Exception:
    logger.exception("Failed to process data.csv")

The resulting log contains the error message together with the traceback. This provides information about the exception type and the sequence of function calls that led to the failure. Simply logging str(exception) loses valuable diagnostic information:

try:
    result = process_file("data.csv")
except Exception as exc:
    logger.error("Processing failed: %s", exc)

The second approach records the exception's message but does not automatically provide the complete traceback. For unexpected failures, logger.exception() is usually more useful.

However, logging every exception with a broad except Exception block is not a substitute for proper exception handling. Exceptions should still be caught at an appropriate boundary and handled according to the application's requirements.

Do Not Log and Re-Raise Without Purpose

A common pattern is to catch an exception, log it, and immediately raise it again:

try:
    process_payment()
except Exception:
    logger.exception("Payment processing failed")
    raise

This can result in the same exception being logged multiple times as it propagates through different layers.

In some architectures, logging at the point where the exception is finally handled is cleaner. If lower-level code has useful context that will be lost later, logging there can still be justified.

The important principle is to avoid creating duplicate error records that make one failure appear to be several independent failures.

Use Handlers for Different Destinations

Python logging separates the creation of log records from their destination through handlers. A handler determines where a record should be sent.

For example, an application can write logs to both the console and a file.

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

console_handler = logging.StreamHandler()
file_handler = logging.FileHandler("application.log")

formatter = logging.Formatter(
    "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)

console_handler.setFormatter(formatter)
file_handler.setFormatter(formatter)

logger.addHandler(console_handler)
logger.addHandler(file_handler)

logger.info("Application started")

Handlers make logging flexible because the same log record can be directed to different destinations. A development environment might primarily use console output, while a production application might send records to files, a centralized logging service, or another monitoring system.

For production systems, file rotation is also important. Writing indefinitely to a single log file can eventually consume significant disk space. Python provides handlers such as RotatingFileHandler and TimedRotatingFileHandler for managing log files over time.

Prefer Structured Logs for Larger Systems

Plain-text logs are easy for humans to read, but structured logging becomes increasingly useful when logs are processed by machines.

A structured record might conceptually contain fields such as:

{
    "level": "ERROR",
    "event": "payment_failed",
    "order_id": "12345",
    "transaction_id": "abc789"
}

Instead of searching through arbitrary text, logging platforms can filter and aggregate individual fields. This becomes particularly valuable for applications running across multiple servers or containers. Python's standard logging module can support structured formats through custom formatters and additional libraries can provide more advanced structured logging functionality.

The important idea is to keep important information represented consistently. For example, using order_id in one message and order in another makes automated searching and analysis unnecessarily difficult.

Common Python Logging Mistakes to Avoid

Several logging problems appear repeatedly in Python applications. The first is using print() for application diagnostics. print() is perfectly appropriate for simple scripts and direct command-line output, but it does not provide logging levels, handlers, formatters, filtering, or integration with the rest of the logging system.

The second is logging everything at INFO. If every minor internal operation produces an informational message, important events can become buried in noise. Detailed diagnostic information usually belongs at DEBUG.

The third is producing vague messages such as:

Error occurred
Something failed
Invalid request
Operation complete

These messages lack context. A better message identifies the operation and, when appropriate, a relevant identifier. The fourth is creating a new logging configuration inside every module. This can result in duplicate handlers, inconsistent formats, repeated messages, and difficult-to-maintain configuration.

Another problem is excessive logging inside high-frequency operations. A loop processing millions of records should not necessarily produce one log record per iteration. Logging itself has overhead, and enormous logs can become more difficult to search than no logs at all. Instead, applications can log meaningful milestones or aggregate information:

logger.info("Processed %d records", processed_count)

This communicates progress without generating an enormous number of records.

Finally, developers should avoid treating logs as the application's primary data store. Logs are designed to record events and provide diagnostic information. Business data, audit records, metrics, and application state may require separate storage mechanisms.

Building a Practical Logging Strategy

A good logging strategy begins with deciding what information developers will need when something goes wrong. For most Python applications, a practical baseline includes module-specific loggers, centralized configuration, consistent formatting, appropriate severity levels, contextual identifiers, and traceback information for unexpected exceptions. A typical application might follow a pattern such as:

import logging

logger = logging.getLogger(__name__)

def create_user(user_id):
    logger.info("Creating user user_id=%s", user_id)

    try:
        save_user(user_id)
    except DatabaseError:
        logger.exception("Failed to save user user_id=%s", user_id)
        raise

This example keeps the logging focused on meaningful events. The successful operation is recorded at INFO, while the database failure is recorded at ERROR with the traceback preserved.

As applications become more distributed, the same principles continue to apply. Logs should remain consistent, searchable, contextual, and safe to store. Correlation IDs can connect events belonging to the same request, structured fields can make logs easier to query, and centralized collection can provide visibility across multiple application instances.

Logging should also complement metrics and tracing rather than replace them. Metrics are useful for understanding numerical trends such as request rates or error rates, while traces help follow an individual request through multiple services. Logs provide detailed event-level context.

The strongest observability setup therefore uses each mechanism for what it does best.

Good Python logging is ultimately about signal rather than volume. A well-designed log tells us what happened, where it happened, how significant it was, and enough context to investigate it. Python's logging module provides the core tools needed to build that system, but the quality of the final result depends on how those tools are used.

By choosing appropriate log levels, creating module-level loggers, centralizing configuration, writing contextual messages, preserving exception tracebacks, protecting sensitive information, and using structured data when appropriate, we can create logs that remain useful from local development through production systems.