Reminiscence is undoubtedly changing into an important side of Agentic AI. Because the use circumstances for AI Brokers develop in complexity, so does the necessity for these brokers to be taught from previous experiences, make the most of saved business-specific data, and adapt to evolving situations based mostly on collected data.
In my earlier article, “Reminiscence in AI: Key Advantages and Funding Issues,” I explored why reminiscence is pivotal for AI, discussing its position in recall, reasoning, and steady studying. This piece, nevertheless, will dive instantly into the implementation of reminiscence by inspecting its impression via the “teachability” performance within the widespread agent framework, Autogen.
Word: Whereas this text is technical in nature, it affords worth for each technical professionals and enterprise leaders trying to consider the position of reminiscence in Agentic AI methods. I’ve structured it in order that readers can skip over the code sections and nonetheless grasp the way in which reminiscence can increase the responses of your AI methods. In the event you don’t want to observe the code, you could learn the descriptions of every step to get a way of the method… or simply the important thing findings and suggestions part.
My exploration of Autogen’s Teachable Brokers revealed each their potential and limitations in dealing with each easy and sophisticated reminiscence duties.
Out of the field, Autogen’s TeachableAgent performs much less brilliantly than anticipated. The Agen’ts reasoning capability conflates reminiscences collectively in a non productive approach and the included retrieval mechanism will not be arrange for multi-step searches essential for answering complicated questions. This limitation means that if you need to make use of Autogen’s Teachable Brokers, there must be substantial customization to each complement reasoning capabilities and obtain extra refined reminiscence retrieval.
To construct extra strong reminiscence capabilities, it’s essential to implement multi-step search performance. A single reminiscence search usually falls wanting offering the great data wanted for complicated duties. Implementing a sequence of interconnected searches might considerably improve the agent’s capability to collect and synthesize related data.
The “teachability” characteristic, whereas highly effective, needs to be approached with warning. Steady activation with out oversight dangers information poisoning and compromise of trusted data sources. Enterprise leaders and resolution architects ought to take into account implementing a human-in-the-loop method, permitting customers to approve what the system learns versus treating each inference as floor fact the system ought to be taught from. This oversight in Autogen’s present Teachable Agent design might trigger important dangers related to unchecked studying.
Lastly, the tactic of reminiscence retrieval from a data retailer performs a big position within the system’s effectiveness. Shifting past easy nearest neighbor searches, which is the TeachableAgent’s default, to extra superior methods similar to hybrid search (combining key phrase and vector approaches), semantic search, or data graph utilization might dramatically enhance the relevance and accuracy of retrieved data.
To appropriately exhibit how exterior reminiscence will be helpful, I created a fictitious state of affairs for a automotive components manufacturing plant. Comply with the code under to implement a Teachable Agent your self.
Situation: A automotive components manufacturing facility must put a plan in place in case there are vitality constraints. The plan must be versatile and adapt based mostly on how a lot energy consumption the ability can use and which components and fashions are in demand.
Step 1:
Pre- arrange requires you to pip set up autogen if you happen to don’t have it put in in your energetic setting and create a config JSON file.
Instance of a appropriate config file which makes use of Azure OpenAI’s service mannequin GPT4–o:
[{
"model": "gpt-4o",
"api_key": "<YOUR API KEY>",
"azure_endpoint": "<YOUR ENDPOINT>",
"api_type": "azure",
"api_version": "2024-06-01"
}]
Set up Autogen for python:
pip set up pyautogen
Step 2:
Import the required libraries to your pocket book or file and cargo the config file.
import autogen
from autogen.agentchat.contrib.capabilities.teachability import Teachability
from autogen import ConversableAgent, UserProxyAgentconfig_list = autogen.config_list_from_json(
env_or_file="autogenconfig.json", #the json file identify that shops the config
file_location=".", #this implies the file is in the identical listing
filter_dict={
"mannequin": ["gpt-4o"], #choose a subset of the fashions in your config
},
)
Step 3:
Create the Brokers. We are going to want two brokers due to the way in which that Autogen’s framework works. We use a UserProxyAgent to execute duties and work together with or change human involvement (relying on the specified quantity of human within the loop). We additionally create a Conversable Agent because the “Teachable Agent” which is supposed to work together with different brokers (not the person). You may learn extra in regards to the UserProxyAgents and ConversableAgents right here.
teachable_agent = ConversableAgent(
identify="teachable_agent", # the identify cannot include areas
llm_config={"config_list": config_list, "timeout": 120, "cache_seed": None}, # on this instance we disable caching however whether it is enabled it caches API requests in order that they are often reused when the identical request is used
) person = UserProxyAgent(
identify="person",
human_input_mode="ALWAYS", #I need to have full management over the code executed so I'm setting human_input_mode to ALWAYS. Different choices are NEVER and TERMINATE.
is_termination_msg=lambda x: True if "TERMINATE" in x.get("content material") else False, #setting a termination message is VERY essential as a result of it tells the agent when to complete.
max_consecutive_auto_reply=0, #we do not want this agent to replies a number of instances in a row
code_execution_config={
"use_docker": False
}, # If you're planning on working code look into setting use_docker=True. For this instance I'm not as a result of then I've to stroll via the docker setup, however it's safer than working the code instantly.
)
Step 4:
To have a baseline, let’s ask the identical query to the brokers earlier than and after we “train” it. I purposely determined to exhibit completely different ranges of query problem to point out efficiency on duties that require a single step vs multistep reminiscence retrieval. To efficiently deal with the intricate multi-step query, the retrieval mechanism should present data from two separate reminiscences as context to the language mannequin answerable for producing a closing response.
Easy Query: “The power is experiencing an influence scarcity of 40%. What fashions must be prioritized?”
Advanced Multi Step Query: “The power is experiencing an influence scarcity of 40%. Present me an in depth breakdown of what machines needs to be deactivated and which machines ought to stay energetic.”
As anticipated the responses are generic and never significantly relevant to a person enterprise.
Response previous to enabling long run reminiscence:
SIMPLE QUESTION RESPONSE
===========================
>>>>>>>> USING AUTO REPLY...
teachable_agent (to person):To handle an influence scarcity of 30% successfully, it is essential to prioritize fashions which can be important to essential operations, security, and those who have the best impression on productiveness. Listed below are steps to find out which fashions to prioritize:
1. **Establish Important Techniques:** Prioritize methods which can be very important for security, safety, and important operations. These may embrace:
- Emergency lighting
- Safety methods (alarms, surveillance)
- Essential environmental controls (e.g., refrigeration for hazardous supplies)
- Communication methods
2. **Consider Impression on Core Operations:** Decide which fashions are essential to the ability’s core capabilities. This could embrace equipment, servers, or any processes which can be indispensable for sustaining productiveness and operations.
3. **Power Effectivity:** Prioritize fashions and gear which can be extra energy-efficient and may function throughout the diminished energy capability.
4. **Downtime Value:** Think about the price of downtime for every mannequin. Prioritize these the place downtime would lead to important monetary losses or manufacturing delays.
Primarily based on these ideas, right here’s a generic prioritization framework:
1. **Security and Safety Techniques:**
- Emergency Lighting
- Fireplace Alarms and Sprinkler Techniques
- Safety Cameras and Sensors
2. **Essential Operational Techniques:**
- Important Manufacturing Equipment (that can not be shut down with out main points)
- Servers and IT Infrastructure (particularly these essential for operations and information integrity)
- HVAC Techniques (if essential for sustaining secure working situations)
3. **Help Techniques:**
- Backup Energy Techniques (uninterruptible energy provides, mills)
- Communication Techniques (each inside and exterior communication)
Make sure that non-essential and excessive energy-consuming gear is powered down or scheduled to function throughout off-peak hours if doable.
All the time re-evaluate periodically to make sure that the ability allocation aligns with the dynamic wants of the ability. Communication with key stakeholders relating to priorities and selections can also be important for clean operations throughout an influence scarcity.
COMPLEX MULTISTEP QUESTION RESPONSE
====================================
>>>>>>>> USING AUTO REPLY...
teachable_agent (to person):
To successfully handle a facility experiencing a 30% energy scarcity, we have to prioritize important operations and take into account the criticality and vitality consumption of every machine. Here's a detailed breakdown:
### Step 1: Establish Important Operations
1. **Essential Infrastructure**: Security methods, safety methods, and important IT infrastructure.
2. **Manufacturing Traces**: Machines concerned within the major manufacturing course of.
3. **Help Techniques**: HVAC methods, lighting, and different ancillary gear essential to keep up a secure working setting.
### Step 2: Categorize Machines by Significance and Power Consumption
#### A. Important and Excessive-Power Machines
1. **Security and Safety Techniques**
- Fireplace alarms and sprinklers
- Surveillance cameras
- Safety entry controls
2. **Major Manufacturing Machines**
- Core manufacturing strains
- Central processing items or mainframe servers (for IT-heavy operations)
3. **Essential Help Techniques**
- Emergency lighting
- Important HVAC methods (for sustaining operation-critical temperature and humidity)
#### B. Non-Important and Excessive-Power Machines
1. **Secondary Manufacturing Traces**
- Non-primary or non-critical manufacturing strains
- Tools used for non-essential or ancillary processes
2. **Supplementary IT Infrastructure**
- Servers used for non-essential operations
- Backup methods that aren't in rapid use
3. **Non-Important HVAC and Lighting**
- HVAC methods for non-critical areas
- Lighting in non-essential zones
#### C. Non-Important and Low-Power Machines
1. **Workplace Tools**
- Computer systems, printers, and copiers in non-essential places of work
2. **Non-essential Lab or Analysis Tools**
- Units in a roundabout way linked to present important duties
### Step 3: Prioritize Deactivation
#### Fast Deactivation
1. **Non-Important and Excessive-Power Machines**
- Secondary manufacturing strains
- Non-essential backup servers
2. **Non-Important HVAC and Lighting**
- HVAC in non-critical areas
- Non-essential lighting
3. **Non-Important Workplace Tools**
- Printers, non-essential computer systems, and copiers
#### Restricted Utilization
1. **Important and Excessive-Power Machines**
- Function essential manufacturing strains at diminished capability if doable
- Implement rotational operation schedules for important processes
2. **Essential Help Techniques**
- Optimize HVAC settings (e.g., elevating or reducing set factors inside acceptable ranges)
- Cut back lighting to minimal security requirements
### Step 4: Implement Power Conservation Measures
1. **Optimize Operations**
- Make the most of energy-efficient settings
- Implement energy-saving protocols and encourage employees to preserve vitality
2. **Scheduled Downtime**
- Plan for downtime throughout off-peak hours for important equipment to stability the load
3. **Common Monitoring**
- Monitor vitality consumption in real-time to make fast changes as wanted
### Abstract
#### Stay Lively:
1. Security and safety methods
2. Foremost manufacturing strains
3. Essential IT infrastructure
4. Important HVAC and lighting for operational areas
#### Deactivate or Restrict Use:
1. Non-essential manufacturing strains
2. Redundant IT methods
3. Non-critical HVAC and lighting
4. Workplace gear in non-essential areas
**Word**: These suggestions needs to be adjusted based mostly on the particular operational necessities of your facility. Moreover, it may very well be useful to speak these adjustments to all employees to make sure compliance and understanding.
Code to ask baseline questions:
#easy query
person.initiate_chat(teachable_agent, message="The power is experiencing an influence scarcity of 40%. What fashions must be prioritized?", clear_history=True)
#multistep complicated query
person.initiate_chat(teachable_agent, message="The power is experiencing an influence scarcity of 30%. Present me an in depth breakdown of what machines needs to be deactivated and which machines ought to stay energetic.", clear_history=True)Step 5:
Create the “teachability” functionality that you simply then add to the agent. The Teachability class inherits from the AgentCapabiliy class, which basically lets you add customizable capabilities to the Brokers.
The Teachability class has many optionally available parameters that may be additional explored right here.
The out of the field Teachability class is a fast and handy approach of including long run reminiscence to the brokers, however will doubtless must be custom-made to be used in a manufacturing setting, as outlined in the important thing findings part. It includes sending messages to an Analyzer Agent that evaluates the person messages for potential storage and retrieval. The Analyzer Agent appears for recommendation that may very well be relevant to related duties sooner or later after which summarizes and shops task-advice pairs in a binary database serving because the agent’s “reminiscence”.
teachability = Teachability(
verbosity=0, # 0 for primary data, 1 so as to add reminiscence operations, 2 for analyzer messages, 3 for memo lists.
reset_db=True, # we need to reset the db as a result of we're creating a brand new agent so we do not need any present reminiscences. If we needed to make use of an present reminiscence retailer we might set this to false.
path_to_db_dir="./tmp/pocket book/teachability_db", #that is the default path you should use any path you want
recall_threshold=1.5, # Larger numbers permit extra (however much less related) memos to be recalled.
max_num_retrievals=10 #10 is default bu you'll be able to set the max variety of memos to be retrieved decrease or greater
)teachability.add_to_agent(teachable_agent)
Step 6:
Now that the teachable_agent is configured, we have to present it the knowledge that we would like the agent to “be taught” (retailer within the database and retrieve from).
Consistent with our state of affairs, I needed the agent to have primary understanding of the ability which consisted of:
- the sorts of elements the manufacturing plant produces
- the sorts of automotive fashions the elements must be made for
- which machines are used to make every part
Moreover, I needed to supply some operational steering on the priorities of the ability relying on how energy constrained it’s. This contains:
- Steering in case of vitality capability constraint of greater than 50%
- Steering in case of vitality capability constraint between 25–50%
- Steering in case of vitality capability constraint between 0–25%
business_info = """
# This manufacturing plant manufactures the next car components:
- Physique panels (doorways, hoods, fenders, and many others.)
- Engine elements (pistons, crankshafts, camshafts)
- Transmission components
- Suspension elements (springs, shock absorbers)
- Brake system components (rotors, calipers, pads)# This manufactoring plant produces components for the next fashions:
- Ford F-150
- Ford Focus
- Ford Explorer
- Ford Mustang
- Ford Escape
- Ford Edge
- Ford Ranger
# Tools for Particular Automotive Components and Their Makes use of
## 1. Physique Panels (doorways, hoods, fenders, and many others.)
- Stamping presses: Type sheet steel into physique panel shapes
- Die units: Used with stamping presses to create particular panel shapes
- Hydraulic presses: Form and kind steel panels with excessive strain
- Robotic welding methods: Automate welding of physique panels and buildings
- Laser chopping machines: Exactly minimize sheet steel for panels
- Sheet steel forming machines: Form flat sheets into curved or complicated types
- Hemming machines: Fold and crimp edges of panels for power and security
- Metallic ending gear (grinders, sanders): Clean surfaces and take away imperfections
- Paint cubicles and spraying methods: Apply paint and protecting coatings
- Drying ovens: Remedy paint and coatings
- High quality management inspection methods: Examine for defects and guarantee dimensional accuracy
## 2. Engine Elements (pistons, crankshafts, camshafts)
- CNC machining facilities: Mill and drill complicated engine components
- CNC lathes: Create cylindrical components like pistons and camshafts
- Boring machines: Enlarge and end cylindrical holes in engine blocks
- Honing machines: Create a positive floor end on cylinder bores
- Grinding machines: Obtain exact dimensions and clean surfaces
- EDM gear: Create complicated shapes in hardened supplies
- Forging presses: Form steel for crankshafts and connecting rods
- Die casting machines: Produce engine blocks and cylinder heads
- Warmth remedy furnaces: Alter materials properties for power and sturdiness
- Quenching methods: Quickly cool components after warmth remedy
- Balancing machines: Guarantee rotating components are completely balanced
- Coordinate Measuring Machines (CMMs): Confirm dimensional accuracy
## 3. Transmission Components
- Gear chopping machines: Create exact gear tooth on transmission elements
- CNC machining facilities: Mill and drill complicated transmission housings and components
- CNC lathes: Produce shafts and different cylindrical elements
- Broaching machines: Create inside splines and keyways
- Warmth remedy gear: Harden gears and different elements
- Precision grinding machines: Obtain extraordinarily tight tolerances on gear surfaces
- Honing machines: End inside bores in transmission housings
- Gear measurement methods: Confirm gear geometry and high quality
- Meeting strains with robotic methods: Put collectively transmission elements
- Take a look at benches: Consider accomplished transmissions for efficiency and high quality
## 4. Suspension Elements (springs, shock absorbers)
- Coil spring winding machines: Produce coil springs to actual specs
- Leaf spring forming gear: Form and kind leaf springs
- Warmth remedy furnaces: Strengthen springs and different elements
- Shot peening gear: Enhance fatigue power of springs
- CNC machining facilities: Create precision components for shock absorbers
- Hydraulic cylinder meeting gear: Assemble shock absorber elements
- Gasoline charging stations: Fill shock absorbers with pressurized gasoline
- Spring testing machines: Confirm spring charges and efficiency
- Sturdiness take a look at rigs: Simulate real-world situations to check longevity
## 5. Brake System Components (rotors, calipers, pads)
- Excessive-precision CNC lathes: Machine brake rotors to actual specs
- Grinding machines: End rotor surfaces for smoothness
- Die casting machines: Produce caliper our bodies
- CNC machining facilities: Mill and drill calipers for exact match
- Precision boring machines: Create correct cylinder bores in calipers
- Hydraulic press: Compress and kind brake pad supplies
- Powder coating methods: Apply protecting finishes to calipers
- Meeting strains with robotic methods: Put collectively brake elements
- Brake dynamometers: Take a look at brake system efficiency and sturdiness
"""
business_rules_over50 = """
- The engine elements are essential and equipment needs to be saved on-line that corresponds to producing these elements when capability constraint is extra or equal to 50%: engine elements
- Elements for the next fashions needs to be prioritized when capability constraint is extra or equal to 50%: 1.Ford F-150
"""
business_rules_25to50 = """
- The next elements are essential and equipment needs to be saved on-line that corresponds to producing these elements when capability constraint is between 25-50%: engine elements and transmission components
- Elements for the next fashions needs to be prioritized when capability constraint is between 25-50%: 1.Ford F-150 2.Ford Explorer
"""
business_rules_0to25 = """
- The next elements are essential and equipment needs to be saved on-line that corresponds to producing these elements when capability constraint is between 0-25%: engine elements,transmission components, Brake System Components
- Elements for the next fashions needs to be prioritized when capability constraint is between 0-25%: 1.Ford F-150 2.Ford Explorer 3.Ford Mustang 4.Ford Focus
"""
person.initiate_chat(teachable_agent, message=business_info, clear_history=True)
person.initiate_chat(teachable_agent, message=business_rules_over50, clear_history=True)
person.initiate_chat(teachable_agent, message=business_rules_25to50, clear_history=True)
person.initiate_chat(teachable_agent, message=business_rules_0to25, clear_history=True)
Step 7:
Now that we’ve added essential data to long run reminiscence, let’s ask the Agent the identical query and see the distinction in its responses.
Easy Query: “The power is experiencing an influence scarcity of 40%. What fashions must be prioritized?”
Advanced Multi Step Query: “The power is experiencing an influence scarcity of 40%. Present me an in depth breakdown of what machines needs to be deactivated and which machines ought to stay energetic.”
Outcomes after implementing long run reminiscence:
The Teachable Agent was in a position to partially reply the easy query. It was appropriate on the automotive fashions that must be prioritized, but it surely offered further incorrect details about which elements wanted to be included.
The complicated multi-step query was not adequately addressed. It erroneously merged two distinct enterprise tips for 0–25% and 25–50% capability constraints right into a single, generalized guideline, leading to an inaccurate response. Moreover, it solely offered a partial listing of machines that ought to stay energetic or be deactivated for every part’s manufacturing, omitting essential particulars from the entire operational protocol. This demonstrates the challenges in precisely retrieving and synthesizing data from a number of, separate reminiscence sources for complicated queries.
SIMPLE QUESTION RESPONSE
===========================
>>>>>>>> USING AUTO REPLY...
teachable_agent (to person):
Given the ability is experiencing an influence scarcity of 40%, this falls throughout the 25-50% capability constraint vary. In accordance with your beforehand established priorities for such a state of affairs, the next fashions and elements must be prioritized:### Elements to Prioritize
- Engine elements
- Transmission components
- Brake System Components
### Fashions to Prioritize
1. Ford F-150
2. Ford Explorer
It is best to be sure that the equipment essential to supply these essential elements for the Ford F-150 and Ford Explorer is saved on-line.
COMPLEX MULTISTEP QUESTION RESPONSE
====================================
>>>>>>>> USING AUTO REPLY...
teachable_agent (to person):
In response to the 40% energy scarcity, let's create an in depth plan to make sure probably the most essential equipment stays operational whereas deactivating much less important ones.
### Essential Prioritization (0-50% capability constraint):
- **Essential Elements**:
- Engine elements (pistons, crankshafts, camshafts)
- Transmission components
- Brake System Components (rotors, calipers, pads)
- **Prioritized Fashions**:
- Ford F-150
- Ford Explorer
### Machines to Stay Lively:
1. **Engine Part Machines**:
- Engine Block Milling Machine
- Crankshaft Grinding Machine
- Camshaft Milling Machine
2. **Transmission Half Machines**:
- Transmission Housing Caster
- Gear Reducing Machine
- Transmission Meeting Line
3. **Brake System Half Machines**:
- Brake Rotor Casting Machine
- Brake Caliper Milling Machine
- Brake Pad Press
### Machines to Deactivate:
1. **Physique Panel Machines**:
- Door Stamping Machine
- Hood Meeting Machine
- Fender Bending Machine
2. **Suspension Part Machines**:
- Coil Spring Forming Machine
- Shock Absorber Meeting Line
### Detailed Breakdown:
#### Lively Machines:
- **Engine Block Milling Machine**:
- Produces essential engine blocks, important for Ford F-150 and Ford Explorer.
- **Crankshaft Grinding Machine**:
- Ensures excessive precision crankshafts; essential for the efficiency of Ford F-150 and Ford Explorer.
- **Camshaft Milling Machine**:
- Produces camshafts, essential for Ford F-150 and Ford Explorer engines.
- **Transmission Housing Caster**:
- Vital for casting the housing of transmissions utilized in Ford F-150 and Ford Explorer.
- **Gear Reducing Machine**:
- Precision gears manufacturing, very important for Ford F-150 and Ford Explorer transmissions.
- **Transmission Meeting Line**:
- Meeting of full transmission items for Ford F-150 and Ford Explorer.
- **Brake Rotor Casting Machine**:
- Produces brake rotors, essential for the secure braking methods in Ford F-150 and Ford Explorer.
- **Brake Caliper Milling Machine**:
- Machining of brake calipers, important for Ford F-150 and Ford Explorer.
- **Brake Pad Press**:
- Manufacturing of brake pads, essential for guaranteeing braking effectivity in Ford F-150 and Ford Explorer.
#### Deactivated Machines:
- **Door Stamping Machine**:
- Whereas essential, physique panels like doorways are much less essential than engine and transmission elements beneath present constraints.
- **Hood Meeting Machine**:
- Deactivating hood meeting to unencumber energy for essential elements manufacturing.
- **Fender Bending Machine**:
- Non-critical in comparison with engine and brake elements throughout a big energy scarcity.
- **Coil Spring Forming Machine**:
- Suspension springs are decrease priorities beneath present circumstances.
- **Shock Absorber Meeting Line**:
- Suspension elements aren’t essential sufficient to maintain operational throughout this degree of energy scarcity.
This plan ensures that manufacturing focuses on sustaining the essential elements wanted for major fashions, the Ford F-150 and Ford Explorer, thereby maximizing the utility of the remaining energy whereas managing manufacturing constraints successfully.
Code:
#easy query
person.initiate_chat(teachable_agent, message="The power is experiencing an influence scarcity of 40%. What fashions must be prioritized?", clear_history=True)
#multistep complicated query
person.initiate_chat(teachable_agent, message="The power is experiencing an influence scarcity of 30%. Present me an in depth breakdown of what machines needs to be deactivated and which machines ought to stay energetic.", clear_history=True)
Whereas Autogen offers an easy introduction to AI methods with reminiscence, it falls quick in dealing with complicated duties successfully.
When creating your personal AI Agent System with reminiscence capabilities, take into account specializing in these key capabilities:
- Implement multi-step searches to make sure complete and related outcomes. This enables the agent to evaluate the usefulness of search outcomes and deal with all elements of a question utilizing the retrieved data. Moreover, think about using extra superior retrieval approaches similar to semantic search, hybrid search, or data graphs for the most effective outcomes.
- To restrict the potential for information poisoning, develop a considerate method to who ought to be capable of “train” the agent and when the agent ought to “studying”. Primarily based on tips set by the enterprise or developer, one also can use agent reasoning to find out if one thing needs to be added to reminiscence and by whom.
- Take away the probability of retrieving old-fashioned data by including a reminiscence decaying mechanism that determines when a reminiscence is now not related or a more moderen reminiscence ought to change it.
- For multi-agent methods involving group chats or inter-agent data sharing, discover varied communication patterns. Decide the simplest strategies for transferring supplemental data and set up limits to forestall data overload.