Don’t Let Conda Eat Your Arduous Drive

Should you’re an Anaconda consumer, you recognize that conda environments allow you to handle bundle dependencies, keep away from compatibility conflicts, and share your tasks with others. Sadly, they will additionally take over your laptop’s laborious drive.

I write plenty of laptop tutorials and to maintain them organized, every has a devoted folder construction full with a Conda Atmosphere. This labored nice at first, however quickly my laptop’s efficiency degraded, and I seen that my SSD was filling up. At one level I had solely 13 GB free.

Conda helps handle this drawback by storing downloaded bundle information in a single “cache” (pkgs_dirs). If you set up a bundle, conda checks for it within the bundle cache earlier than downloading. If not discovered, conda will obtain and extract the bundle and hyperlink the information to the energetic atmosphere. As a result of the cache is “shared,” totally different environments can use the identical downloaded information with out duplication.

As a result of conda caches each downloaded bundle, pkgs_dirs can develop to many gigabytes. And whereas conda hyperlinks to shared packages within the cache, there may be nonetheless a must retailer some packages within the atmosphere folder. That is primarily to keep away from model conflicts, the place totally different environments want totally different variations of the identical dependency (a bundle required to run one other bundle).

As well as, giant, compiled binaries like OpenCV could require full copies within the atmosphere’s listing, and every atmosphere requires a replica of the Python interpreter (at 100–200 MB). All these points can bloat conda environments to a number of gigabytes.

On this Fast Success Information Science venture, we’ll have a look at some strategies for decreasing the storage necessities for conda environments, together with these saved in default places and devoted folders.

Reminiscence Administration Methods

Under are some Reminiscence Administration strategies that may allow you to scale back conda’s storage footprint in your machine. We’ll talk about every in flip.

1. Cache cleansing
2. Sharing task-based environments
3. Archiving with atmosphere and specs information
4. Archiving environments with conda-pack
5. Storing environments on an exterior drive
6. Relocating the bundle cache
7. Utilizing digital environments (venv)

1. Cleansing the Bundle Cache

Cleansing the bundle cache is the primary and best step for releasing up reminiscence. Even after deleting environments, conda retains the associated bundle information within the cache. You’ll be able to release area by eradicating these unused packages and their related tarballs (compressed bundle information), logs, index caches (metadata saved in conda), and short-term information.

Conda permits an non-compulsory “dry run” to see how a lot reminiscence will probably be reclaimed. You’ll need to run this from both the terminal or Anaconda Immediate in your base atmosphere:

conda clear --all --dry-run

To commit, run:

conda clear --all

Right here’s how this seems on my machine:

This course of trimmed a wholesome 6.28 GB and took a number of minutes to run.

2. Sharing Activity-based Environments

Creating a couple of environments for specialised duties — like laptop imaginative and prescient or geospatial work — is extra reminiscence environment friendly than utilizing devoted environments for every venture. These environments would come with primary packages plus ones for the particular activity (akin to OpenCV, scikit-image, and PIL for laptop imaginative and prescient).

A bonus of this method is you could simply preserve all of the packages updated and hyperlink the environments to a number of tasks. Nonetheless, this gained’t work if some tasks require totally different variations of the shared packages.

3. Archiving with Atmosphere and Specs Recordsdata

Should you don’t have sufficient storage websites or need to protect legacy tasks effectively, think about using atmosphere or specs information. These small information file an atmosphere’s contents, permitting you to rebuild it later.

Saving conda environments on this method reduces their measurement on disk from gigabytes to a couple kilobytes. In fact, you’ll must recreate the atmosphere to make use of it. So, you’ll need to keep away from this system when you ceaselessly revisit tasks that hyperlink to the archived environments.

NOTE: Think about using Mamba, a drop-in substitute for conda, for quicker rebuilds. Because the docs say, “If you recognize conda, you recognize Mamba!”

Utilizing Atmosphere Recordsdata: An environmental file is a small file that lists all of the packages and variations put in in an atmosphere, together with these put in utilizing Python’s bundle installer (pip). This helps you each restore an atmosphere and share it with others.

The atmosphere file is written in YAML (.yml), a human-readable data-serialization format for knowledge storage. To generate an atmosphere file, you should activate after which export the atmosphere. Right here’s how you can make a file for an atmosphere named my_env:

 conda activate my_env
 conda env export > my_env.yml

You’ll be able to title the file any legitimate filename however watch out as an current file with the identical title will probably be overwritten.

By default, the atmosphere file is written to the consumer listing. Right here’s a truncated instance of the file’s contents:

title: C:Usershannaquick_successfed_hikesfed_env
channels:
  - defaults
  - conda-forge
dependencies:
  - asttokens=2.0.5=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - blas=1.0=mkl
  - bottleneck=1.3.4=py310h9128911_0
  - brotli=1.0.9=ha925a31_2
  - bzip2=1.0.8=he774522_0
  - ca-certificates=2022.4.26=haa95532_0
  - certifi=2022.5.18.1=py310haa95532_0
  - colorama=0.4.4=pyhd3eb1b0_0
  - cycler=0.11.0=pyhd3eb1b0_0
  - debugpy=1.5.1=py310hd77b12b_0
  - decorator=5.1.1=pyhd3eb1b0_0
  - entrypoints=0.4=py310haa95532_0

  ------SNIP------

Now you can take away your conda atmosphere and reproduce it once more with this file. To take away an atmosphere, first deactivate it after which run the take away command (the place ENVNAME is the title of your atmosphere):

conda deactivate
conda take away -n ENVNAME --all

If the conda atmosphere exists outdoors of Anaconda’s default envs folder, then embody the listing path to the atmosphere, as so:

conda take away -p PATHENVNAME --all

Be aware that this archiving method will solely work completely when you proceed to make use of the identical working system, akin to Home windows or macOS. It’s because fixing for dependencies can introduce packages which may not be appropriate throughout platforms.

To revive a conda atmosphere utilizing a file, run the next, the place my_env represents your conda atmosphere title and atmosphere.yml represents your atmosphere file:

 conda env create -n my_env -f directorypathtoenvironment.yml

It’s also possible to use the atmosphere file to recreate the atmosphere in your D: drive. Simply present the brand new path when utilizing the file. Right here’s an instance:

conda create --prefix D:my_envsmy_new_env --file atmosphere.yml

For extra on atmosphere information, together with how you can manually produce them, go to the docs.

Utilizing Specs Recordsdata: Should you haven’t put in any packages utilizing pip, you need to use a specs file to breed a conda atmosphere on the identical working system. To create a specification file, activate an atmosphere, akin to my_env, and enter the next command:

 conda checklist --explicit > exp_spec_list.txt

This produces the next output, truncated for brevity:

 # This file could also be used to create an atmosphere utilizing:
 # $ conda create --name <env> --file <this file>
 # platform: win-64
 @EXPLICIT
 https://conda.anaconda.org/conda-forge/win-64/ca-certificates-202x.xx.x-h5b45459_0.tar.bz2
 https://conda.anaconda.org/conda-forge/noarch/tzdata-202xx-he74cb21_0.tar.bz2

------snip------

Be aware that the --explicit flag ensures that the focused platform is annotated within the file, on this case, # platform: win-64 within the third line.

Now you can take away the atmosphere as described within the earlier part.

To re-create my_env utilizing this textual content file, run the next with a correct listing path:

conda create -n my_env -f directorypathtoexp_spec_list.txt

4. Archiving Environments with conda-pack

The conda-pack command allows you to archive a conda atmosphere earlier than eradicating it. It packs all the atmosphere right into a compressed archive with the extension: .tar.gz. It’s useful for backing up, sharing, and transferring environments with out the necessity to reinstall packages.

The next command will protect an atmosphere however take away it out of your system (the place my_env represents the title of your atmosphere):

conda set up -c conda-forge conda-pack
conda pack -n my_env -o my_env.tar.gz

To revive the atmosphere later run this command:

mkdir my_env && tar -xzf my_env.tar.gz -C my_env

This method gained’t save as a lot reminiscence because the textual content file possibility. Nonetheless, you gained’t must re-download packages when restoring an atmosphere, which implies it may be used with out web entry.

5. Storing Environments on an Exterior Drive

By default, conda shops all environments in a default location. For Home windows, that is below the …anaconda3envs folder. You’ll be able to see these environments by working the command conda information --envs in a immediate window or terminal. Right here’s the way it seems on my C: drive (it is a truncated view):

Utilizing a Single Environments Folder: In case your system helps an exterior or secondary drive, you possibly can configure conda to retailer environments there to release area in your major disk. Right here’s the command; you’ll must substitute your particular path:

conda config --set envs_dirs /path/to/exterior/drive

Should you enter a path to your D drive, akin to D:conda_envs, conda will create new environments at this location.

This method works nicely when your exterior drive is a quick SSD and whenever you’re storing packages with giant dependencies, like TensorFlow. The draw back is slower efficiency. In case your OS and notebooks stay on the first drive, you might expertise some learn/write latency when working Python.

As well as, some OS settings could energy down idle exterior drives, including a delay once they spin again up. Instruments like Jupyter could wrestle to find conda environments if the drive letter adjustments, so that you’ll need to use a hard and fast drive letter and be certain that the proper kernel paths are set.

Utilizing A number of Atmosphere Folders: As an alternative of utilizing a single envs_dirs listing for all environments, you possibly can retailer every atmosphere inside its respective venture folder. This allows you to retailer every little thing associated to a venture in a single place.

For instance, suppose you could have a venture in your Home windows D: drive in a folder referred to as D:projectsgeospatial. To position the venture’s conda atmosphere on this folder, loaded with ipykernel for JupyterLab, you’d run:

conda create -p D:projectsgeospatialenv ipykernel

In fact, you possibly can name env one thing extra descriptive, like geospatial_env.

As with the earlier instance, environments saved on a special disk may cause efficiency points.

Particular Be aware on JupyterLab: Relying on the way you launch JupyterLab, its default habits could also be to open in your consumer listing (akin to, C:Usersyour_user_name). Since its file browser is restricted to the listing from which it’s launched, you gained’t see directories on different drives like D:. There are a lot of methods to deal with this, however one of many easiest is to launch JupyterLab from the D: drive.

For instance, in Anaconda Immediate, kind:

D:

adopted by:

jupyter lab

Now, it is possible for you to to select from kernels on the D: drive.

For extra choices on altering JupyterLab’s working listing, ask an AI about “how you can change Jupyter’s default working listing” or “how you can create a Symlink to D: in your consumer folder.”

Shifting Present Environments: You must by no means manually transfer a conda atmosphere, akin to by slicing and pasting to a brand new location. It’s because conda depends on inside paths and metadata that may change into invalid with location adjustments.

As an alternative, you must clone current environments to a different drive. This may duplicate the atmosphere, so that you’ll must manually take away it from its authentic location.

Within the following instance, we use the --clone flag to supply an actual copy of a C: drive atmosphere (referred to as my_env) on the D: drive:

conda create -p D:new_envsmy_env --clone C:pathtooldenv

NOTE: Take into account exporting your atmosphere to a YAML file (as described in Part 3 above) earlier than cloning. This lets you recreate the atmosphere if one thing goes improper with the clone process.

Now, whenever you run conda env checklist, you’ll see the atmosphere listed in each the C: and D: drives. You’ll be able to take away the outdated atmosphere by working the next command within the base atmosphere:

conda take away --name my_env --all -y

Once more, latency points could have an effect on these setups when you’re working throughout two disks.

You could be questioning, is it higher to maneuver a conda atmosphere utilizing an atmosphere (YAML) file or to make use of--clone? The brief reply is that --clone is one of the best and quickest possibility for transferring an atmosphere to a special drive on the identical machine. An atmosphere file is finest for recreating the identical atmosphere on a totally different machine. Whereas the file ensures a constant atmosphere throughout totally different methods, it could take for much longer to run, particularly with giant environments.

6. Relocating the Bundle Cache

In case your major drive is low on area, you possibly can transfer the bundle cache to a bigger exterior or secondary drive utilizing this command:

conda config --set pkgs_dirs D:conda_pkgs

On this instance, packages at the moment are saved on the D drive (D:conda_pkgs) as an alternative of the default location.

Should you’re working in your major drive and each drives are SSD, then latency points shouldn’t be vital. Nonetheless, if one of many drives is a slower HDD, you possibly can expertise slowdowns when creating or updating environments. If D: is an exterior drive related by USB, you may even see vital slowdowns for giant environments.

You’ll be able to mitigate a few of these points by retaining the bundle cache (pkgs_dirs) and ceaselessly used environments on the quicker SSD, and different environments on the slower HDD.

One last item to think about is backups. Major drives could have routine backups scheduled however secondary or exterior drives could not. This places you prone to dropping all of your environments.

7. Utilizing Digital Environments

In case your venture doesn’t require conda’s intensive bundle administration system for dealing with heavy dependencies (like TensorFlow or GDAL), you possibly can considerably scale back disk utilization with a Python digital atmosphere (venv). This represents a light-weight various to a conda atmosphere.

To create a venv named my_env, run the next command:

One of these atmosphere has a small base set up. A minimal conda atmosphere takes up about 200 MB and consists of a number of utilities, akin to conda, pip, setuptools, and so forth. A venv is way lighter, with a minimal set up measurement of solely 5–10 MB.

Conda additionally caches bundle tarballs in pkgs_dirs. These tarballs can develop to a number of GBs over time. As a result of venv installs packages straight into the atmosphere, no additional copies are preserved.

Normally, you’ll need to contemplate venv whenever you solely want primary Python packages like NumPy, pandas, or Scikit-learn. Packages for which conda is strongly really useful, like Geopandas, ought to nonetheless be positioned in a conda atmosphere. Should you use plenty of environments, you’ll in all probability need to follow conda and profit from its bundle linking.

You’ll find particulars on how you can activate and use Python digital environments within the venv docs.

Recap

Excessive affect/low disruption reminiscence administration strategies for conda environments embody cleansing the bundle cache and storing little-used environments as YAML or textual content information. These strategies can save many gigabytes of reminiscence whereas retaining Anaconda’s default listing construction.

Different excessive affect strategies embody transferring the bundle cache and/or conda environments to a secondary or exterior drive. This may resolve reminiscence issues however could introduce latency points, particularly if the brand new drive is a sluggish HDD or makes use of a USB connection.

For easy environments, you need to use a Python digital atmosphere (venv) as a light-weight various to conda.