Migrating from Hugging Face

​

Introduction

​

Comparing Hugging Face and Outerscope

​

Benefits of migrating to Outerscope

1. Faster build times: Outerscope significantly reduces build times by up to 95%, especially for subsequent builds (an additional 56% reduction). This can greatly improve iteration speed and the cost of running experiments with complex ML apps.
2. Flexible cooldown period: With a minimum cooldown period of just 1 second (compared to Hugging Face’s 15 minutes), Outerscope allows for more efficient resource utilization and cost management.
3. Improved cold start handling: When encountering a cold start, Outerscope waits for the infrastructure to become available instead of throwing an error. This results in a better user experience and fewer failed requests.
4. Model colocation flexibility: Outerscope doesn’t require a separate repository for each inference endpoint, simplifying the management of models. Each function in your app becomes an endpoint automatically, which means that you can run multiple models from the same app to save costs.
5. Pay-per-use model: Outerscope’s pricing model ensures you pay only for the compute resources you actually use. This can lead to cost savings, especially for sporadic or low-volume inference needs.
6. Competitive performance: Outerscope only adds up to 50ms of latency to your inference requests. This results in competitive response times from a warm start. Caching mechanisms and highly optimized orchestration pipelines help apps start from a cold state in an average of 2-5 seconds.
7. Customizable infrastructure: Outerscope allows for fine-grained control over the infrastructure specifications, enabling you to optimize for your specific use case.

​

Migration process

​

1. Outerscope setup and configuration

​

1.1 Install Outerscope CLI

pip install Outerscope --upgrade

​

1.2 Update your requirements file

[Outerscope.deployment]
name = "llama-8b-vllm"
python_version = "3.11"
docker_base_image_url = "debian:bookworm-slim"
include = ["./*", "main.py", "Outerscope.toml"]
exclude = [".*"]

[Outerscope.hardware]
cpu = 2
memory = 12.0
compute = "AMPERE_A10"

[Outerscope.scaling]
min_replicas = 0
max_replicas = 5
cooldown = 30

[Outerscope.dependencies.pip]
sentencepiece = "latest"
torch = "latest"
transformers = "latest"
accelerate = "latest"
xformers = "latest"
pydantic = "latest"
bitsandbytes = "latest"

• Outerscope.deployment: Specifies the project name, Python version, base Docker image, and which files to include/exclude as project files.
• Outerscope.hardware: Defines the CPU, memory, and GPU requirements for your deployment.
• Outerscope.scaling: Configures auto-scaling behavior, including minimum and maximum replicas, and cooldown period.
• Outerscope.dependencies.pip: Lists the Python packages required for your project.

​

1.3 Update your code

import torch
import os
from huggingface_hub import login
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig

# Log into Hugging Face Hub
login(token=os.environ.get("HF_AUTH_TOKEN"))

model_path = "meta-llama/Meta-Llama-3.1-8B-Instruct"
cache_directory = "/persistent-storage"

# Set up tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_path, cache_dir=cache_directory)
tokenizer.pad_token_id = 0
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
    cache_dir=cache_directory,
)

class Item(BaseModel):
    prompt: str
    temperature: float
    top_p: float
    top_k: int
    max_tokens: int
    frequency_penalty: float

def run(
        prompt, temperature=0.6, top_p=0.9, top_k=0, max_tokens=512, frequency_penalty=1
):
    item = Item(
        prompt=prompt,
        temperature=temperature,
        top_p=top_p,
        top_k=top_k,
        max_tokens=max_tokens,
        frequency_penalty=frequency_penalty,
    )

    # Place prompt in template
    inputs = tokenizer(
        item.prompt, return_tensors="pt", max_length=512, truncation=True, padding=True
    )
    input_ids = inputs["input_ids"].to("cuda")

    # Set up generation config
    generation_config = GenerationConfig(
        temperature=temperature,
        top_p=top_p,
        top_k=top_k,
        max_tokens=max_tokens,
    )
    with torch.no_grad():
        outputs = model.generate(
            input_ids=input_ids,
            generation_config=generation_config,
            return_dict_in_generate=True,
            output_scores=True,
        )
    result = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)

    return {"result": result}

1. Authenticates with Hugging Face using a secret token. Add this secret in the Outerscope dashboard.
2. Initializes the Llama 3.1 8B model using vLLM for efficient inference.
3. Defines an Item class to structure and validate (using Pydantic) the input parameters.
4. Implements a run function that generates text based on the provided prompt and parameters.

​

2. Deployment

Outerscope deploy

​

3. Using the deployed model

import requests
import json

url = "https://api.Outerscope/v4/p-xxxxxxxx/llama-8b-vllm/run"

payload = json.dumps({"prompt": "tell me about yourself"})

headers = {
  'Authorization': 'Bearer [OUTERSCOPE_API_KEY]',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

​

Additional considerations

1. API structure: The Outerscope implementation uses a different API structure compared to Hugging Face
2. Authentication: Ensure you have set up the HF_AUTH_TOKEN secret in Outerscope for authenticating with Hugging Face
3. Model permissions: The example uses the Llama 3.1 8B Instruct model. Ensure you have the necessary permissions to use this model
4. Hardware optimization: The Outerscope.toml file specifies the hardware requirements. Adjust these based on your specific model and performance needs
5. Dependency management: Regularly review and update the dependencies listed in Outerscope.toml to ensure you’re using the latest compatible versions
6. Scaling configuration: The example sets up auto-scaling with 0 to 5 replicas and a 30-second cooldown. Monitor your usage patterns and adjust these parameters as needed
7. Cold starts: While Outerscope handles cold starts more gracefully than Hugging Face, be aware that the first request after a period of inactivity may still take longer to process
8. Monitoring and logging: Familiarize yourself with Outerscope’s monitoring and logging capabilities to track your model’s performance and usage
9. Cost management: Although Outerscope’s pay-per-use model can be more cost-effective, set up proper monitoring and alerts to avoid unexpected costs
10. Testing: Thoroughly test your migrated models to ensure they perform as expected on the new platform

​

Conclusion

• Secrets
• Model scaling
• Keeping models warm