Machine Learning

Handle Surroundings Variables with Pydantic

February 13, 2025
roosho

Introduction

Builders work on purposes which can be presupposed to be deployed on some server with a purpose to enable anybody to make use of these. Sometimes within the machine the place these apps stay, builders arrange setting variables that enable the app to run. These variables will be API keys of exterior providers, URL of your database and way more.

For native improvement although, it’s actually inconvenient to declare these variables on the machine as a result of it’s a sluggish and messy course of. So I’d wish to share on this quick tutorial the best way to use Pydantic to deal with setting variables in a safe method.

.env file

What you generally do in a Python venture is to retailer all of your setting variables in a file named .env. This can be a textual content file containing all of the variables in a key : worth format. You should use additionally the worth of one of many variables to declare one of many different variables by leveraging the {} syntax.

The next is an instance:

#.env file

OPENAI_API_KEY="sk-your personal key"
OPENAI_MODEL_ID="gpt-4o-mini"

# Growth settings
DOMAIN=instance.org
ADMIN_EMAIL=admin@${DOMAIN}

WANDB_API_KEY="your-private-key"
WANDB_PROJECT="myproject"
WANDB_ENTITY="my-entity"

SERPAPI_KEY= "your-api-key"
PERPLEXITY_TOKEN = "your-api-token"

Bear in mind the .env file ought to stay personal, so it will be important that this file is talked about in your .gitignore file, to ensure that you by no means push it on GitHub, in any other case, different builders might steal your keys and use the instruments you’ve paid for.

env.instance file

To ease the lifetime of builders who will clone your repository, you would embrace an env.instance file in your venture. This can be a file containing solely the keys of what’s supposed to enter the .env file. On this method, different individuals know what APIs, tokens, or secrets and techniques on the whole they should set to make the scripts work.

#env.instance

OPENAI_API_KEY=""
OPENAI_MODEL_ID=""

DOMAIN=""
ADMIN_EMAIL=""

WANDB_API_KEY=""
WANDB_PROJECT=""
WANDB_ENTITY=""

SERPAPI_KEY= ""
PERPLEXITY_TOKEN = ""

python-dotenv is the library you employ to load the variables declared into the .env file. To put in this library:

pip set up python-dotenv

Now you should utilize the load_dotenv to load the variables. Then get a reference to those variables with the os module.

import os
from dotenv import load_dotenv

load_dotenv()

OPENAI_API_KEY = os.getenv('OPENAI_API_KEY')
OPENAI_MODEL_ID = os.getenv('OPENAI_MODEL_ID')

This methodology will first look into your .env file to load the variables you’ve declared there. If this file doesn’t exist, the variable shall be taken from the host machine. Which means you should utilize the .env file on your native improvement however then when the code is deployed to a number setting like a digital machine or Docker container we’re going to instantly use the setting variables outlined within the host setting.

Pydantic

Pydantic is likely one of the most used libraries in Python for knowledge validation. It is usually used for serializing and deserializing courses into JSON and again. It robotically generates JSON schema, decreasing the necessity for handbook schema administration. It additionally supplies built-in knowledge validation, guaranteeing that the serialized knowledge adheres to the anticipated format. Lastly, it simply integrates with widespread net frameworks like FastAPI.

pydantic-settings is a Pydantic function wanted to load and validate settings or config courses from setting variables.

!pip set up pydantic-settings

We’re going to create a category named Settings. This class will inherit BaseSettings. This makes the default behaviours of figuring out the values of any fields to be learn from the .env file. If no var is discovered within the .env file it is going to be used the default worth if supplied.

from pydantic_settings import BaseSettings, SettingsConfigDict

from pydantic import (
    AliasChoices,
    Subject,
    RedisDsn,
)


class Settings(BaseSettings):
    auth_key: str = Subject(validation_alias="my_auth_key")  
    api_key: str = Subject(alias="my_api_key")  

    redis_dsn: RedisDsn = Subject(
        'redis://consumer:go@localhost:6379/1', #default worth
        validation_alias=AliasChoices('service_redis_dsn', 'redis_url'),  
    )

    model_config = SettingsConfigDict(env_prefix='my_prefix_')

Within the Settings class above now we have outlined a number of fields. The Subject class is used to offer additional data about an attribute.

In our case, we setup a validation_alias. So the variable title to search for within the .env file is overridden. Within the case reported above, the setting variable my_auth_key shall be learn as an alternative of auth_key.

You too can have a number of aliases to search for within the .env file which you could specify by leveraging AliasChoises(choise1, choise2).

The final attribute model_config , incorporates all of the variables concerning a selected matter (e.g connection to a db). And this variable will retailer all .env var that begin with the prefix env_prefix.

Instantiate and use settings

The subsequent step can be to really instantiate and use these settings in your Python venture.

from pydantic_settings import BaseSettings, SettingsConfigDict

from pydantic import (
    AliasChoices,
    Subject,
    RedisDsn,
)


class Settings(BaseSettings):
    auth_key: str = Subject(validation_alias="my_auth_key")  
    api_key: str = Subject(alias="my_api_key")  

    redis_dsn: RedisDsn = Subject(
        'redis://consumer:go@localhost:6379/1', #default worth
        validation_alias=AliasChoices('service_redis_dsn', 'redis_url'),  
    )

    model_config = SettingsConfigDict(env_prefix='my_prefix_')

# create instantly a settings object
settings = Settings()

Now what use the settings in different components of our codebase.

from Settings import settings

print(settings.auth_key)

You lastly have an quick access to your settings, and Pydantic helps you validate that the secrets and techniques have the right format. For extra superior validation suggestions confer with the Pydantic documentation: https://docs.pydantic.dev/newest/

Remaining ideas

Managing the configuration of a venture is a boring however necessary a part of software program improvement. Secrets and techniques like API keys, db connections are what often energy your utility. Naively you possibly can hardcode these variables in your code and it’ll nonetheless work, however for apparent causes, this might not be an excellent observe. On this article, I confirmed you an introduction on the best way to use pydantic settings to have a structured and secure technique to deal with your configurations.

💼 Linkedin ️| 🐦 X (Twitter) | 💻 Web site