Which is the Higher SLM?

The evolution of AI fashions has reached new heights, notably in small language fashions (SLMs), the place effectivity and efficiency are key. Among the many newest contenders, Phi-4-mini and o1-mini stand out as superior and environment friendly fashions. On this article, we’ll be doing a Phi-4 mini vs o1-mini comparability to take a look at their consumer expertise, pace, and efficiency on STEM purposes and coding duties. We will likely be assessing their strengths in programming, debugging, and total effectivity to see which mannequin performs higher. By the tip, you’ll have a transparent perspective on which mannequin aligns along with your wants.

What’s Phi-4-mini?

Phi-4-mini is a state-of-the-art SLM designed for high-performance reasoning and coding duties. It strikes a steadiness between effectivity and accuracy, making it a robust contender in AI-driven purposes. The mannequin is designed for high-accuracy textual content era and complicated reasoning duties whereas being computationally environment friendly, making it well-suited for edge computing environments

Structure Overview

Phi-4-mini is a dense, decoder-only transformer mannequin with 3.8 billion parameters and a 128K token context window. It helps a vocabulary measurement of 200,064 tokens and incorporates Grouped Question Consideration (GQA) to optimize useful resource effectivity whereas sustaining excessive efficiency.

Grouped Question Consideration (GQA) is an environment friendly consideration mechanism that balances the pace of multi-query consideration (MQA) with the standard of multi-head consideration (MHA) by grouping question heads and sharing key/worth heads, enhancing inference pace for language fashions

Key Options

• Shared Enter-Output Embeddings: Reduces reminiscence overhead by reusing embeddings for each enter and output.
• Coaching Information: Skilled on 5 trillion tokens, together with high-quality instructional materials, coding examples, and artificial information tailor-made for reasoning.
• Efficiency: Excels in reasoning, arithmetic, coding, and instruction-following, with the flexibility to combine exterior APIs by operate calling.

Additionally Learn: Phi-4 vs GPT-4o-mini Face-Off

What’s o1-mini?

o1-mini is a light-weight and cost-efficient SLM aimed toward balancing affordability and efficiency. It prioritizes environment friendly processing whereas sustaining an inexpensive stage of accuracy for normal AI purposes.

Structure Overview

o1-mini follows a normal transformer structure, with fewer parameters than Phi-4-mini (actual measurement undisclosed). It additionally helps a 128K token context window however focuses on cost-effective processing relatively than architectural optimizations like GQA.

Additionally Learn: OpenAI’s o1-preview vs o1-mini: A Step Ahead to AGI

Mannequin Comparability: Phi-4-mini vs o1-mini

Phi-4-mini is a robust mannequin designed for duties like reasoning, math, and coding, whereas o1-mini follows a less complicated design targeted on cost-effective coding. The desk beneath highlights their key variations:

Phi-4-mini stands out with superior options like GQA and shared embeddings, making it superior in reasoning, coding, and API integration. In distinction, o1-mini is a lighter, cost-effective different optimized for coding, although it lacks the architectural refinements seen in Phi-4-mini. Selecting between the 2 is dependent upon whether or not the precedence is excessive accuracy and reasoning energy or affordability and effectivity in particular duties.

Analysis of Reasoning Efficiency

This part appears at how the Phi-4-mini and o3-mini fashions carry out in reasoning in comparison with their bigger fashions. It focuses on how nicely they remedy advanced issues and make logical conclusions, highlighting the variations in accuracy, effectivity, and readability between the smaller and bigger fashions.

Phi-4-mini and o1-mini vs Bigger Fashions

The reasoning capabilities of the reasoning-enhanced Phi-4-mini and o1-mini have been evaluated throughout a number of benchmarks, together with AIME 2024, MATH-500, and GPQA Diamond. These benchmarks assess superior mathematical reasoning and normal problem-solving expertise, offering a foundation for comparability in opposition to a number of bigger fashions from DeepSeek, Bespoke, and OpenThinker.

Regardless of having solely 3.8 billion parameters, the reasoning-trained Phi-4-mini demonstrates robust efficiency, surpassing bigger fashions resembling:

• DeepSeek-R1-Distill-Llama-8B (8B parameters)
• Bespoke-Stratos-7B (7B parameters)
• OpenThinker-7B (7B parameters)

Moreover, it achieves efficiency similar to DeepSeek-R1-Distill-Qwen-7B, a considerably bigger 7B mannequin, additional highlighting its effectivity. Nonetheless, o1-mini, regardless of its undisclosed parameter measurement, leads throughout a number of benchmarks, making it a robust contender in AI reasoning duties.

Benchmark Comparisons

The efficiency of each fashions, as proven within the offered picture, highlights their competitiveness in opposition to bigger fashions:

• AIME Benchmark:
  • o1-mini scores 63.6, the best amongst all fashions.
  • Phi-4-mini (reasoning skilled) scores 50.0, a fivefold enchancment over its base model (10.0).
• MATH-500 Benchmark:
  • Phi-4-mini (90.4) barely outperforms o1-mini (90.0), making it extremely efficient for advanced mathematical reasoning duties.
• GPQA Diamond:
  • o1-mini leads with 60.0, showcasing superior normal problem-solving capabilities.
  • Phi-4-mini (49.0) outperforms a number of 7B and 8B fashions, proving its effectivity in reasoning duties.

These outcomes point out that o1-mini dominates generally problem-solving and reasoning, whereas Phi-4-mini (reasoning-trained) excels in mathematical benchmarks regardless of its smaller measurement (3.8B parameters). Each fashions reveal distinctive effectivity, difficult and even outperforming considerably bigger fashions throughout key AI benchmarks.

Phi-4-mini vs o1-mini: Reasoning and Coding Capabilities

Now we’ll evaluate the reasoning and programming capabilities of Phi-4-mini and o1-mini. For that, we’re going to give the identical immediate to each fashions and consider their responses and we will likely be utilizing API to load the mannequin. Listed below are the duties we’ll be making an attempt out on this comparability:

1. Analyzing constructing order relationships
2. Logical reasoning in Math
3. Discovering the Longest Substring

Process 1: Analyzing Constructing Order Relationships

This activity requires the mannequin to infer the relative positions of buildings primarily based on the given constraints and determine the center constructing.

Immediate: “There are 5 buildings referred to as V, W, X, Y and Z in a row (not essentially in that order). V is to the West of W. Z is to the East of X and the West of V, W is to the West of Y. Which is the constructing within the center?
Choices:
A) V
B) W
C) X
D) Y”

Enter to o1-mini

from openai import OpenAI
import time
import tiktoken
from IPython.show import show, Markdown
with open("path_to_api_key") as file:
   api_key = file.learn().strip()


task1_start_time = time.time()
shopper = OpenAI(api_key=api_key)
messages = [
   {
       "role": "user",
       "content": """
   There are five buildings called V, W, X, Y and Z in a row (not necessarily in that order).
   V is to the West of W. Z is to the East of X and the West of V, W is to the West of Y.
   Which is the building in the middle?
   Options:
   A) V
   B) W
   C) X
   D) Y
"""
   }
]
completion = shopper.chat.completions.create(
   mannequin="o1-mini-2024-09-12",
   messages=messages
)
task1_end_time = time.time()
# Print outcomes
print(completion.selections[0].message)
print("----------------=Whole Time Taken for activity 1:----------------- ", task1_end_time - task1_start_time)
# Show outcome
from IPython.show import Markdown
show(Markdown(completion.selections[0].message.content material))

Response by o1-mini

Enter to Phi 4-mini

from transformers import pipeline
import time
from IPython.show import show, Markdown


from transformers import AutoModelForCausalLM, BitsAndBytesConfig
quantization_config = BitsAndBytesConfig(load_in_8bit=True)


# Load mannequin immediately
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-4-mini-instruct", trust_remote_code=True)
mannequin = AutoModelForCausalLM.from_pretrained("microsoft/Phi-4-mini-instruct", trust_remote_code=True, quantization_config=quantization_config) 


task1_start_time = time.time()
messages = [
   {"role": "system", "content": "You are an expert in solving numerical and general reasoning questions."},
   {"role": "user", "content": """There are five buildings called V, W, X, Y and Z in a row (not necessarily in that order).
   V is to the West of W. Z is to the East of X and the West of V, W is to the West of Y.Which is the building in the middle? Options:
A) V
B) W
C) X
D) Y"""},
]


pipe = pipeline(
   "text-generation",
   mannequin=mannequin,
   tokenizer=tokenizer,
)
generation_args = {
   "max_new_tokens": 1024,
   "return_full_text": False,
   "temperature": 0.0,
   "do_sample": False,
}
output = pipe(messages, **generation_args) #,
task1_end_time = time.time()
print("----------------=Whole Time Taken for activity 1:----------------- ", task1_end_time - task1_start_time)
show(Markdown((output[0]['generated_text'])))
pipe = pipeline("text-generation", mannequin="microsoft/Phi-4-mini-instruct", trust_remote_code=True)
pipe(messages)

Response by Phi 4-mini

Comparative Evaluation

o1-mini is healthier than Phi-4-mini in each pace and accuracy for this activity. o1-mini shortly figures out the right reply (“V”) with just some steps, whereas Phi-4-mini takes for much longer as a result of it goes by every element step-by-step. Even with all that effort, Phi-4-mini nonetheless will get the fallacious reply (“Z”), which isn’t even one of many selections. This reveals that Phi-4-mini struggles with easy logic issues, whereas o1-mini handles them shortly and accurately. Phi-4-mini’s detailed considering could be helpful for tougher issues, however on this case, it solely prompted delays and errors.

Process 2:  Logical Reasoning in Math

This activity requires the mannequin to acknowledge the sample in a given quantity sequence and determine the lacking quantity.

Immediate: “Choose the quantity from among the many given choices that may exchange the query mark (?) within the following sequence:16, 33, 100, 401, ?
Choices:A) 1235
B) 804
C) 1588
D) 2006″

Enter to o1-mini

task2_start_time = time.time()

shopper = OpenAI(api_key=api_key)

messages = [
   {
       "role": "user",
       "content": """Select the number from among the given options that can replace the question mark (?) in the following series.16, 33, 100, 401, ?
       A) 1235
       B) 804
       C) 1588
       D) 2006"""
   }
]


# Use a appropriate encoding (cl100k_base is the most suitable choice for brand new OpenAI fashions)
encoding = tiktoken.get_encoding("cl100k_base")


# Calculate token counts
input_tokens = sum(len(encoding.encode(msg["content"])) for msg in messages)
completion = shopper.chat.completions.create(
   mannequin="o1-mini-2024-09-12",
   messages=messages
)
output_tokens = len(encoding.encode(completion.selections[0].message.content material))


task2_end_time = time.time()
# Print outcomes
print(completion.selections[0].message)
print("----------------=Whole Time Taken for activity 2:----------------- ", task2_end_time - task2_start_time)
# Show outcome
from IPython.show import Markdown
show(Markdown(completion.selections[0].message.content material))

Response by o1-mini

Enter to Phi 4-mini

task2_start_time = time.time()
messages = [
   {"role": "system", "content": "You are an expert in solving numerical and general reasoning questions."},
   {"role": "user", "content": """Select the number from among the given options
   that can replace the question mark (?) in the following series.16, 33, 100, 401, ?
A) 1235
B) 804
C) 1588
D) 2006"""},
]
pipe = pipeline(
   "text-generation",
   mannequin=mannequin,
   tokenizer=tokenizer,
)
generation_args = {
   "max_new_tokens": 1024,
   "return_full_text": False,
   "temperature": 0.0,
   "do_sample": False,
}
output = pipe(messages, **generation_args) #,
task2_end_time = time.time()
print("----------------=Whole Time Taken for activity 2:----------------- ", task2_end_time - task2_start_time)
show(Markdown((output[0]['generated_text'])))

Response by Phi 4-mini

Comparative Evaluation

o1-mini carried out higher than Phi-4-mini in each pace and accuracy for this quantity sample activity. o1-mini shortly acknowledged the sample and accurately selected 2006 in simply 10.77 seconds. Alternatively, Phi-4-mini took for much longer (50.25 seconds) and nonetheless received the fallacious reply (120). In the meantime, o1-mini adopted a transparent and direct method, fixing the issue accurately and effectively. This reveals that o1-mini is healthier at recognizing quantity patterns shortly, whereas Phi-4-mini tends to overcomplicate easy issues, resulting in errors and delays.

Process 3: Discovering the Longest Substring

This downside asks you to seek out the size of the longest substring inside a given string that doesn’t comprise any repeating characters. For instance, within the string “abcabcbb”, the longest substring with out repeating characters could be “abc”, and its size is 3.

Immediate: “Given a string s, discover the size of the longest substring with out repeating characters.
Write a operate lengthOfLongestSubstring(s: str) -> int that returns the size of the longest substring with out repeating characters.”

Enter to o1-mini

task3_start_time = time.time()
shopper = OpenAI(api_key=api_key)
messages = [
   {
       "role": "user",
       "content": """
Given a string s, find the length of the longest substring without repeating characters.


Write a function lengthOfLongestSubstring(s: str) -> int that returns the length of the longest substring without repeating characters.
"""
   }
]
# Use a appropriate encoding (cl100k_base is the most suitable choice for brand new OpenAI fashions)
encoding = tiktoken.get_encoding("cl100k_base")
# Calculate token counts
input_tokens = sum(len(encoding.encode(msg["content"])) for msg in messages)
completion = shopper.chat.completions.create(
   mannequin="o1-mini-2024-09-12",
   messages=messages
)
output_tokens = len(encoding.encode(completion.selections[0].message.content material))
task3_end_time = time.time()
# Print outcomes
print(completion.selections[0].message)
print("----------------=Whole Time Taken for activity 3:----------------- ", task3_end_time - task3_start_time)
# Show outcome
from IPython.show import Markdown
show(Markdown(completion.selections[0].message.content material))

Response by o1-mini

def lengthOfLongestSubstring(s: str) -> int:
   """
   Returns the size of the longest substring with out repeating characters within the string s.


   Parameters:
   s (str): The enter string.


   Returns:
   int: The size of the longest substring with out repeating characters.
   """
   # Dictionary to retailer the final positions of every character.
   char_map = {}
   # Initialize the left pointer of the window.
   left = 0
   # Initialize the utmost size.
   max_length = 0


   # Iterate over the string with the correct pointer.
   for proper, char in enumerate(s):
       # If the character is present in char_map and its final place is >= left pointer,
       # transfer the left pointer to at least one place proper of its final incidence to keep away from duplicates.
       if char in char_map and char_map[char] >= left:
           left = char_map[char] + 1


       # Replace/insert the character's newest index.
       char_map[char] = proper


       # Calculate the size of the present window and replace max_length if essential.
       current_window_length = proper - left + 1
       if current_window_length > max_length:
           max_length = current_window_length


       # Debug statements to hint the interior state (might be eliminated in manufacturing)
       # print(f"Present character: {char}, Left: {left}, Proper: {proper}, Present Window: '{s[left:right+1]}', Max Size: {max_length}")


   return max_length


# Instance utilization:
if __name__ == "__main__":
   test_strings = [
       "abcabcbb",
       "bbbbb",
       "pwwkew",
       "",
       "a",
       "dvdf"
   ]
   for s in test_strings:
       print(f"Longest substring with out repeating characters in '{s}' is of size {lengthOfLongestSubstring(s)}.")

Output

Enter to Phi 4-mini

from transformers import pipeline
messages = [
   {"role": "user",
    "content":"""Given a string s, find the length of the longest substring without repeating characters.
Write a function lengthOfLongestSubstring(s: str) -> int that returns the length of the longest substring without repeating characters."""},
]
pipe = pipeline("text-generation", mannequin="microsoft/Phi-4-mini-instruct", trust_remote_code=True)
pipe(messages)

Response by Phi 4-mini

def lengthOfLongestSubstring(s: str) -> int:
   char_index_map = {}
   left = 0
   max_length = 0
   for proper in vary(len(s)):
       if s[right] in char_index_map and char_index_map[s[right]] >= left:
           left = char_index_map[s[right]] + 1
       char_index_map[s[right]] = proper
       max_length = max(max_length, proper - left + 1)


   return max_length
print("Longest substring with out repeating characters in 'abcabcbb' is of size ",lengthOfLongestSubstring("abcabcbb"))
print("Longest substring with out repeating characters in 'bbbbb' is of size  ",lengthOfLongestSubstring("bbbbb"))

Output

Comparative Evaluation

Each o1-mini and Phi-4-mini used the sliding window technique accurately, however o1-mini’s code was extra organized and simpler to know. It included clear explanations, feedback, check circumstances, and easy-to-read variable names. Phi-4-mini’s resolution was shorter however lacked explanations and construction, making it tougher to observe in greater tasks. o1-mini was additionally quicker and produced a cleaner, extra readable resolution, whereas Phi-4-mini targeted extra on preserving the code transient.

General Comparative Evaluation

Right here’s the general comparative evaluation for all 3 duties:

Conclusion

General, o1-mini excelled in structured reasoning, accuracy, and coding greatest practices, making it extra appropriate for advanced problem-solving and maintainable code. Whereas Phi-4-mini was quicker, its exploratory method sometimes led to inefficiencies or incorrect conclusions, particularly in reasoning duties. In coding, o1-mini offered well-documented and readable options, whereas Phi-4-mini prioritized brevity at the price of readability. If pace is the principle concern, Phi-4-mini is a strong selection, however for precision, readability, and structured problem-solving, o1-mini stands out as the higher choice.

Incessantly Requested Questions