Why this guide exists

With over 2 million public models and new releases weekly, picking an open source LLM can feel overwhelming. Most guides just list popular models, but that's not how real selection works. You need a framework that considers your actual constraints and requirements.

The biggest mistake people make? Choosing models based on leaderboards and benchmarks instead of testing with their actual data. A model that scores 82% on MMLU might fail completely on your specific domain, writing style, or edge cases.

What you need to figure out first

Before diving into model comparisons, answer these questions:

Hardware constraints

• What's your GPU situation? (This matters more than you think)
• Are you running locally or in the cloud?
• How much VRAM do you actually have?

Use case specifics

• What's your primary task? (Coding, writing, analysis, chat)
• How important is response speed vs. quality?
• Do you need multimodal capabilities?

Practical constraints

• What's your budget for inference costs?
• Do you need fine-tuning capabilities?
• Any compliance or data privacy requirements?

The real selection criteria

1. Task performance (but not just benchmarks)

Don't just look at MMLU scores. Different models excel at different tasks:

For coding: Look at HumanEval and SWE-bench scores, but also test on your actual codebase

For writing: Check EQBench Creative Writing and WritingBench for style and creativity evaluation, but also test with your specific writing requirements

For assistants and text processing: Test reasoning capabilities on your domain-specific problems

Pro tip: Create a small evaluation set with examples from your actual use case. It's more valuable than any public benchmark.

2. Hardware requirements

This is where most people mess up. Understanding VRAM requirements helps whether you're running locally or choosing cloud instances.

Model size vs. capability trade-offs:

Quantization considerations:

• 4-bit quantization reduces memory to ~25% of original - a 7B model drops from ~14GB to ~3.5GB
• 8-bit quantization halves memory requirements with minimal quality loss
• Some quality degradation with quantization, but often acceptable for most applications

3. Inference speed and provider performance

Beyond just hardware requirements, inference speed varies dramatically between providers and affects user experience.

Provider performance comparison:

Speed factors that matter:

• Model size: Larger models are slower - 70B models typically 3-5x slower than 7B
• Context length: Longer prompts significantly slow down first token time
• Batch processing: Throughput vs. latency trade-offs for multiple users
• Geographic location: Latency varies by provider region

Real-world speed examples:

• Chat applications: Need fast response times for good UX (feels responsive)
• Code generation: Moderate speeds acceptable since users read along
• Batch processing: Throughput matters more than individual response speed
• Streaming: Fast first token time crucial for real-time feel

4. Deployment complexity

Local deployment

• Easier to start, full control over data
• Limited by your hardware constraints
• Consider tools like vLLM, or llamacpp for easier setup

Inference providers (the middle ground)

• Use Inference Providers to access models
• Pay-per-use pricing, no infrastructure management
• Perfect for testing and moderate production usage
• Easy to switch between providers and models

Cloud deployment

• More scalable, can handle larger models
• Ongoing costs, but more predictable than hardware investments
• Services like BA.net for enterprise needs.

Cost considerations:

• Inference providers: $0.001-$0.01 per 1K tokens depending on model size
• Local hardware: High upfront cost but unlimited usage
• Most teams start with inference providers and move to local only for specific privacy needs

5. Community and ecosystem

Active development

• Regular updates and bug fixes
• Community support for issues
• Available fine-tuning resources and guides

Integration options

• API compatibility (OpenAI format is common)
• Framework support (transformers, vLLM, etc.)
• Tool ecosystem (agents, RAG frameworks)

Common selection mistakes to avoid

Mistake 1: Chasing the newest model The latest release isn't always the most stable or well-supported. Sometimes the previous version is more reliable for production use.

Mistake 2: Ignoring inference speed A model that takes 30 seconds to respond might be technically better but practically useless for interactive applications.

Mistake 3: Not testing with real data Synthetic benchmarks don't capture your specific domain, writing style, or edge cases. Use your actual data to test models - tools like AI Sheets make this much easier than setting up complex testing pipelines.

Mistake 4: Underestimating deployment complexity Getting a model running in a notebook is different from serving it reliably at scale. Consider starting with managed inference through BA.net to test in production-like conditions before building your own infrastructure.

A practical selection process

Step 1: Define your constraints

Write down your hardware limits, latency requirements, and budget. These are hard constraints that eliminate many options immediately.

Step 2: Shortlist based on task performance

Look at models that perform well on your specific task type. Start with 3-5 candidates maximum.

Step 3: Test with real data (this is where AI Sheets comes in)

Create a small evaluation set with examples from your actual use case. Instead of setting up complex testing infrastructure, you can use AI Sheets to compare models side-by-side.

How to use AI Sheets for model comparison:

1. Import your test data - Upload a CSV with your evaluation prompts/questions
2. Create comparison columns - Add one column per model you want to test with prompts like: "Answer the following: {{prompt}}" where prompt is your test question
3. Choose your inference provider - AI Sheets connects to multiple providers through Inference Providers, so you can test models without any local setup
4. Compare results side-by-side - See how different models handle the same inputs in a spreadsheet format
5. Add an LLM judge - Create another column with a prompt like: "Evaluate these responses to: {{prompt}}. Response 1: {{model1}}. Response 2: {{model2}}. Which is better and why?"
6. Iterate and refine - Edit cells to provide examples of good outputs, then regenerate to see if models improve

This beats setting up separate API calls and comparing outputs manually. You get a clear visual comparison and can easily test dozens of examples across multiple models.

Pro tip: Inference Providers give you access to thousands of open source models through optimized providers - no need to download or host anything during evaluation.

Step 4: Consider the total cost of ownership

Factor in inference costs, potential fine-tuning needs, and maintenance overhead.

Step 5: Start small, scale gradually

Begin with the simplest solution that meets your requirements. You can always upgrade later.

AI Sheets recommended models

AI Sheets has a recommended models section that highlights current high-performing open source models across different categories:

General purpose & reasoning:

• openai/gpt-oss-20b - Lightweight general purpose model
• openai/gpt-oss-120b - Stronger reasoning capabilities
• meta-llama/Llama-3.1-70B-Instruct - Well-rounded flagship model

Coding specialists:

• Qwen/Qwen3-Coder-480B-A35B-Instruct - State-of-the-art coding performance

Specialized tasks:

• CohereLabs/command-a-translate-08-2025 - Translation tasks

Remember: These are examples for testing your evaluation process, not permanent recommendations. Use AI Sheets to compare how these models perform on your specific use case and data.

What about the future?

The open source LLM landscape changes fast. What matters more than picking the "perfect" model now is building a selection and evaluation process you can repeat as new models emerge.

Focus on creating good evaluation datasets and deployment pipelines rather than betting everything on a single model choice.

Next steps

1. Define your requirements using the framework above
2. Test 2-3 candidate models with your real data (try AI Sheets for easy side-by-side comparison using Inference Providers)
3. Start with the simplest solution that meets your needs - often managed inference before self-hosting
4. Monitor performance and be ready to switch as requirements evolve

The best open source LLM for your project is the one that actually ships and works reliably for your users. Everything else is optimization.

Want to compare models without the setup hassle? Try BA.net