LoRA Merge and Combine Multiple Models - Complete Guide

When you have a LoRA that captures the perfect art style and another that nails a character's appearance, using both simultaneously often creates conflicts. One pushes the generation toward warm colors while the other emphasizes cool tones. One shapes faces a certain way while the other pulls in a different direction. Reducing their individual strengths to avoid conflicts weakens the contribution of each, leaving you with diluted results that capture neither capability well. LoRA merge techniques solve these problems elegantly.

LoRA merge processes combine multiple LoRAs into a single unified model before generation time. Rather than loading separate LoRAs that compete during inference, you create one LoRA through LoRA merge that contains the balanced contributions of all sources. The LoRA merge process resolves conflicts during the merge itself rather than during generation, and sophisticated LoRA merge algorithms can produce results that exceed what stacking achieves. If you're new to ComfyUI, start with our ComfyUI basics guide to understand the fundamentals.

This guide covers everything from basic weighted merging to advanced algorithms like DARE and TIES, with practical workflows using available tools. You'll understand when merging beats stacking, how to choose weights and algorithms, and how to evaluate whether your merged LoRA meets your needs.

Why LoRA Merge Instead of Stacking

Before investing effort in LoRA merge, understand why LoRA merge is often better than simply loading multiple LoRAs at generation time.

When you stack LoRAs, each applies its learned modifications independently during inference. If you load a style LoRA at strength 0.7 and a character LoRA at strength 0.8, both modify the same base model weights during generation. Where their modifications align, effects compound. Where they conflict, you get unpredictable averaging that often produces muddy, inconsistent results.

The core problem is that stacked LoRAs have no knowledge of each other. The style LoRA learned to modify certain weights to achieve its effect without knowing that a character LoRA will also be modifying those same weights. When both apply their modifications, neither achieves its intended effect cleanly.

LoRA merge addresses this by combining the LoRAs into a single model that balances their contributions through explicit weight allocation. When you perform a LoRA merge at 0.6 style and 0.4 character weights, you're creating a new LoRA where each weight contains 60% of the style LoRA's modification plus 40% of the character LoRA's modification for that weight. The LoRA merge result blends both capabilities coherently rather than having them compete.

Beyond conflict resolution, LoRA merge provides practical benefits. Loading one merged LoRA uses less memory and loads faster than loading multiple separate LoRAs. Workflows become simpler when you reference one LoRA instead of juggling multiple with their respective strengths. Sharing your specific blend becomes trivial - hand over one file instead of multiple files plus strength instructions. For VRAM optimization when working with LoRA merge results, check our VRAM optimization guide.

That said, stacking isn't always worse. If you need to dynamically adjust the balance between components during a session, stacking provides that flexibility while merging commits to a fixed blend. If different generations need different component balances, maintaining separate LoRAs makes sense. Merging excels when you've found a specific blend you want to standardize and reuse.

Understanding Merge Algorithms

Merging isn't just adding weights together. Different algorithms produce different results, and understanding your options helps you choose appropriately.

Weighted Sum is the simplest algorithm and often a good starting point. For each weight position in the LoRA, it calculates the weighted average of the source LoRAs' values. If you merge LoRA_A and LoRA_B with weights 0.6 and 0.4:

merged_weight[i] = 0.6 * A_weight[i] + 0.4 * B_weight[i]

This is straightforward and predictable. The result directly reflects your weight choices. If your sources are compatible and you choose appropriate weights, weighted sum produces good results. Its limitation is that it doesn't handle conflicting modifications intelligently - it just averages them.

Add Difference is useful when you have a base LoRA and want to add specific capabilities from another. Instead of blending both equally, it takes one LoRA as the base and adds the difference contributed by the second:

merged_weight[i] = A_weight[i] + factor * (B_weight[i] - base_weight[i])

This preserves the first LoRA fully while adding scaled contributions from the second. It's useful for adding targeted modifications without diluting the primary LoRA's capabilities.

DARE (Drop and Rescale) randomly drops a portion of delta parameters and rescales the remaining ones to compensate. This reduces redundancy between merged sources and can produce cleaner results than straight weighted sum. The intuition is that not all parameters contribute equally to a LoRA's effect, and some redundant parameters between sources cause conflicts. Dropping parameters randomly and rescaling eliminates some redundancy while preserving overall contribution magnitude.

# DARE pseudocode for each LoRA
mask = random_mask(probability=p)  # e.g., p=0.3 drops 30%
scaled_delta = lora_delta * mask / (1 - p)  # Rescale remaining

DARE typically improves merge quality when sources have significant overlap in which parameters they modify. It's worth trying when weighted sum produces muddy results.

TIES (Trim, Elect Sign, Disjoint Merge) addresses a specific problem: when source LoRAs modify the same weight in opposite directions (positive vs negative), averaging produces near-zero values that represent neither source's intent. TIES resolves sign conflicts by trimming small values, electing the dominant sign for conflicting parameters, and merging only parameters with consistent signs.

# TIES pseudocode
trimmed = trim_small_values(lora_deltas, threshold)
signs = elect_signs(trimmed)  # Vote on sign per position
merged = merge_consistent_signs(trimmed, signs)

TIES often produces cleaner merges than weighted sum, especially when sources have conflicting modifications. It's more computationally complex but worth using for high-quality merges.

Block-Weighted Merging applies different weights to different parts of the model architecture. Neural networks have different layers with different functions - early layers capture low-level features while later layers handle high-level concepts. By assigning different merge weights to different blocks, you control which source dominates different aspects of the output.

For example, you might merge a style LoRA at high weight in the layers that affect rendering style while merging a character LoRA at high weight in layers that affect structural composition. This requires understanding which layers do what, but enables more precise control than whole-LoRA weights.

Tools for LoRA Merging

Several tools implement these algorithms with varying degrees of user-friendliness.

SuperMerger is a popular extension for Automatic1111's Web UI. It provides a graphical interface for merging with multiple algorithm options. You select source LoRAs, set weights for each, choose an algorithm, and execute the merge. Results save as new LoRA files. SuperMerger supports weighted sum, add difference, and several other methods. It's accessible and well-documented, making it a good starting point.

Kohya's merge scripts are command-line tools from the same project that provides LoRA training. They offer more algorithm options and precise control than GUI tools but require comfort with command-line operation. A basic weighted merge looks like:

python networks/merge_lora.py \
  --save_to /path/to/merged_lora.safetensors \
  --models /path/to/lora_a.safetensors /path/to/lora_b.safetensors \
  --ratios 0.6 0.4 \
  --save_precision fp16

Kohya scripts support advanced algorithms and detailed configuration, making them powerful for users who need precise control.

ComfyUI custom nodes bring merging into ComfyUI workflows. Nodes like those in ComfyUI-Model-Manager or dedicated merge node packs let you merge LoRAs as part of your generation workflow, immediately test results, and iterate quickly. This integration is convenient for experimentation but may lack the algorithm options of dedicated tools.

Standalone Python scripts offer maximum flexibility if you're comfortable writing code. Libraries like safetensors make loading and saving LoRA files straightforward, and implementing algorithms yourself lets you customize behavior exactly:

from safetensors.torch import load_file, save_file
import torch

def weighted_merge(lora_a_path, lora_b_path, weight_a, weight_b, output_path):
    lora_a = load_file(lora_a_path)
    lora_b = load_file(lora_b_path)

    merged = {}
    for key in lora_a.keys():
        if key in lora_b:
            merged[key] = weight_a * lora_a[key] + weight_b * lora_b[key]
        else:
            merged[key] = lora_a[key]

    # Include keys only in lora_b
    for key in lora_b.keys():
        if key not in lora_a:
            merged[key] = lora_b[key]

    save_file(merged, output_path)

weighted_merge(
    "style_lora.safetensors",
    "character_lora.safetensors",
    0.6, 0.4,
    "merged_result.safetensors"
)

This basic example demonstrates the pattern - load weights, combine them, save the result. You'd extend this with DARE, TIES, or other algorithms as needed.

Practical Merging Workflow

Here's a workflow for merging LoRAs that produces good results reliably.

Start by testing your source LoRAs individually. Generate images with each LoRA alone at full strength to understand what each contributes. Note specific characteristics you want to preserve - the style LoRA's color palette, the character LoRA's facial structure, whatever makes each valuable. This baseline helps you evaluate whether your merge captured these characteristics.

Choose two LoRAs for your first merge. While you can merge more, complexity increases with each source. Start with two, evaluate the result, then consider adding a third if needed. Successful merging builds incrementally.

Consider compatibility before merging. LoRAs trained on the same base model merge better than cross-base merges. An SDXL LoRA and an SD1.5 LoRA cannot merge meaningfully - their weight structures don't correspond. Even within the same base model, LoRAs trained with similar settings (network rank, alpha values, training approaches) merge more predictably than disparate training configurations.

Start with equal weights (0.5/0.5) using weighted sum algorithm. This provides a neutral starting point. Generate test images with your merged LoRA and compare to your individual LoRA baselines. Does the result show characteristics from both sources? Are any characteristics missing or distorted?

Adjust weights based on results. If the style LoRA's characteristics dominate too much, reduce its weight and increase the character LoRA's weight. If neither seems strong enough, the weights might both be too low (though they should sum to 1.0 for weighted sum). Iterate through a few weight combinations, generating test images each time.

If weighted sum produces muddy results or loses desired characteristics, try DARE or TIES algorithms. These often handle conflicting modifications better. Compare results to your weighted sum attempt and your individual LoRA baselines.

Document your successful merges. Record which source LoRAs, what weights, which algorithm, and notes on the result. This lets you reproduce good merges and iterate on them later. Naming merged LoRAs descriptively (style-char-merge-60-40-v1.safetensors) helps track what you have.

Advanced Merging Techniques

Once you're comfortable with basic merging, these techniques provide more control.

Block-weighted merging assigns different weights to different model blocks. This requires tools that support block weights, like LoRA Block Merge extensions. The approach is:

1. Understand the block structure of your base model architecture
2. Identify which blocks affect which aspects of generation
3. Assign per-block weights for each source LoRA
4. Generate tests and adjust block weights

For SDXL, input blocks (IN01-IN08) affect low-level features, mid blocks (MID) affect global composition, and output blocks (OUT0-OUT07) affect final rendering. A common pattern is weighting a style LoRA heavily in output blocks (affecting appearance) while weighting a character LoRA heavily in input blocks (affecting structure).

Block weighting requires significant experimentation to dial in. Start with simple whole-LoRA weights until you need the control, then investigate block weights for fine-tuning.

Iterative merging builds complex merges step by step. Instead of merging five LoRAs at once, merge two, evaluate, merge the result with a third, evaluate, and continue. This gives you checkpoints where you understand what's happening, makes debugging easier when results go wrong, and often produces better results than trying to balance many sources simultaneously.

Merge pruning removes low-magnitude weights from the merged result to reduce file size and potentially improve generalization. After merging, weights below a threshold are zeroed out:

def prune_merge(merged_dict, threshold=0.01):
    pruned = {}
    for key, tensor in merged_dict.items():
        mask = torch.abs(tensor) > threshold
        pruned[key] = tensor * mask
    return pruned

Aggressive pruning can degrade quality, so test results after pruning.

Evaluating Merge Quality

Determining whether a merge succeeded requires systematic evaluation.

Generate images with identical prompts and settings using: each source LoRA individually at the strength you'd normally use, the merged LoRA at strength 1.0, and the stacked source LoRAs at their respective strengths. This gives you direct comparisons.

Check that the merge preserves key characteristics from each source. Does the style LoRA's distinctive color treatment appear? Does the character LoRA's facial structure come through? Lost characteristics indicate the corresponding weight might be too low or the merge algorithm might be suppressing that source's contributions.

Check that new artifacts haven't appeared. Poor merges sometimes introduce distortions that neither source LoRA produces individually. If the merge creates problems not present in either source, try different weights or algorithms.

Compare the merge to stacking. The merge should produce equal or better results than loading both LoRAs simultaneously at adjusted strengths. If stacking produces better results despite conflicts, your merge isn't successfully combining the sources - revisit weights and algorithm choice.

Test across varied prompts. A merge that works well for one prompt might fail on others if it overfit to specific characteristics. Generate diverse images to verify the merge generalizes properly.

Subjective evaluation matters. Technical metrics can't capture whether the merged result achieves your creative intent. Trust your judgment about whether the merge produces what you wanted.

Troubleshooting Common Issues

Muddy or undefined results suggest conflicting modifications are averaging toward zero. Try DARE or TIES algorithms that handle conflicts better than weighted sum. Also verify that source LoRAs are compatible - dramatically different training configurations often don't merge well.

One source dominates completely despite balanced weights usually means that source has stronger weight modifications. Reduce its weight further than you'd expect needed - if one LoRA is "stronger" than another, you might need 0.3/0.7 rather than 0.5/0.5 for balanced appearance.

Artifacts appearing in merge but not sources indicates problematic weight interactions. Try lower total weights (e.g., 0.4/0.4 instead of 0.5/0.5) to reduce modification magnitude. Block-weighted merging can also help by isolating the problematic interactions to specific blocks.

Merged LoRA produces errors when loading usually means file corruption or incompatible architectures. Verify source LoRAs are for the same base model and have compatible shapes. Check the merge tool didn't error during creation.

Results are worse than stacking means the merge isn't successfully combining sources. This happens when sources are too incompatible for the algorithm to reconcile. Try different algorithms, significantly different weights, or accept that these specific LoRAs may not merge well.

Frequently Asked Questions

Can I merge LoRAs for different base models?

No. A Flux LoRA and an SDXL LoRA have completely different weight structures and cannot be meaningfully merged. Only merge LoRAs trained on the same base model architecture.

What's the best weight ratio to start with?

Start with equal weights (0.5/0.5 for two sources) and adjust based on results. If one source should dominate, start with 0.7/0.3. There's no universal best - it depends on your sources and goals.

How many LoRAs can I merge?

Technically unlimited, but practically 2-4 works well. More sources create more complexity in balancing contributions and more opportunities for conflicts. Build incrementally if you need many sources.

Does merging reduce file size?

Not significantly compared to the largest source. The merged LoRA has the same number of parameters as each source. You save memory during inference by loading one file instead of multiple, but the file on disk is similar size.

Will merged LoRAs work with new base model versions?

Usually yes if the architecture is compatible. LoRAs for one SDXL checkpoint generally work with other SDXL checkpoints. However, fine-tuned bases may have better or worse compatibility depending on how much they diverged from the original architecture.

Can I merge a LoRA with itself at different weights?

Technically yes, but it's equivalent to using that LoRA at a different strength during inference. There's no benefit to merging a LoRA with itself.

Is DARE or TIES always better than weighted sum?

Not always. They often produce better results when sources have conflicting modifications, but weighted sum is fine when sources are compatible. Start with weighted sum for simplicity and try others if results aren't satisfactory.

How do I merge block weights in ComfyUI?

Install node packs that support block-weighted LoRA operations. ComfyUI-Inspire-Pack and similar extensions provide nodes for detailed LoRA control including block weights.

Can I undo a merge to get the original LoRAs back?

No, merging is one-way. Always keep your original LoRAs. The merged file doesn't contain enough information to recover individual sources.

Does the merge order matter?

For basic weighted sum, no - addition is commutative. For some advanced algorithms that apply sources sequentially, order might matter. Check your tool's documentation for specific algorithms.

Conclusion

LoRA merge creates unified models that combine capabilities from multiple sources without the conflicts that stacking produces. The LoRA merge process involves choosing compatible sources, selecting appropriate weights and algorithms, and evaluating results against your sources and goals.

Start simple with weighted sum LoRA merge of two compatible LoRAs. Evaluate carefully and adjust weights until you achieve the balance you want. Graduate to advanced LoRA merge algorithms like DARE and TIES when weighted sum doesn't handle conflicts well enough.

LoRA merge is a creative tool that expands what you can achieve with trained LoRAs. The merged models you create through LoRA merge represent new capabilities that didn't exist in any single source. With practice, you'll develop intuition for how different sources combine and what LoRA merge algorithms work best for different situations.

For users who want combined LoRA effects without manual LoRA merge, Apatero.com provides access to professionally merged and tuned LoRA combinations that deliver specific effects reliably.

Advanced Merging Strategies

Beyond basic techniques, advanced strategies unlock more sophisticated merge capabilities.

Hierarchical Merging for Complex Combinations

When combining many LoRAs, hierarchical approaches produce better results:

Hierarchical Structure:

1. Group similar LoRAs (e.g., all style LoRAs)
2. Merge within groups first
3. Then merge group results together
4. Evaluate at each level

Benefits:

• Easier to debug problems
• Better balance within categories
• More predictable results
• Checkpoints for rollback

Example:

Style_A + Style_B → Merged_Styles
Character_A + Character_B → Merged_Characters
Merged_Styles + Merged_Characters → Final_Merge

This produces better results than merging all four simultaneously.

Concept-Aware Weight Distribution

Different LoRA types benefit from different weight strategies:

Style LoRAs:

• Often need higher weights (0.6-0.8) to maintain aesthetic
• Style characteristics are subtle and easily diluted
• Test across varied prompts for consistency

Character LoRAs:

• Can work at lower weights (0.4-0.6)
• Core identity preserved at lower strengths
• Too high can cause artifacts

Concept LoRAs:

• Very dependent on training quality
• Test at multiple weights to find threshold
• Often work well at 0.3-0.5

Understanding these patterns helps you set initial weights more effectively.

DARE and TIES Deep Dive

For challenging merges, understanding DARE and TIES parameters helps:

DARE Configuration:

# DARE parameters
drop_rate = 0.3  # Drop 30% of parameters
rescale = True   # Rescale remaining by 1/(1-drop_rate)

Higher drop rates reduce redundancy more aggressively. Start at 0.3 and increase if results are muddy. Too high causes loss of important parameters.

TIES Configuration:

# TIES parameters
trim_fraction = 0.2  # Trim smallest 20% of values
sign_election = "majority"  # How to resolve sign conflicts

TIES is particularly effective when source LoRAs have opposing modifications to the same weights. The sign election prevents cancellation that causes muddy results.

Creating Production-Quality Merges

For professional use, additional considerations apply.

Merge Testing Protocol

Systematic testing ensures merge quality:

Test Suite:

1. 10 diverse prompts covering common use cases
2. Edge case prompts (extremes that stress the LoRA)
3. Comparison to each source LoRA
4. Comparison to stacked configuration
5. Multiple seeds per prompt for variance check

Evaluation Criteria:

• Characteristic preservation from each source
• No new artifacts introduced
• Consistent quality across test suite
• Equal or better results than stacking

Document test results for future reference.

Version Control for Merges

Maintain organized merge history:

Naming Convention:

{sources}_{algorithm}_{weights}_{version}.safetensors

Example:
style-char_weighted_60-40_v3.safetensors

Documentation:

Merge: style-char_weighted_60-40_v3
Sources: art_style_v2.safetensors, character_face_v1.safetensors
Algorithm: Weighted Sum
Weights: 0.6, 0.4
Notes: Better balance than v2, reduced style bleeding
Date: 2025-01-15

This enables reproducing successful merges and learning from iterations.

Merge Validation Workflow

Before deploying a merge:

Validation Steps:

1. Technical validation (file loads, no errors)
2. Visual validation (test images look correct)
3. Range validation (works across parameter ranges)
4. Edge validation (handles unusual prompts)
5. Performance validation (inference speed acceptable)

Any failure sends the merge back for revision.

Integration with Broader Workflows

Merged LoRAs fit into larger generation systems.

ComfyUI Workflow Integration

Incorporate merged LoRAs into ComfyUI workflows:

Workflow Structure:

• Load merged LoRA in standard LoRA loader
• Apply at strength 0.8-1.0 (internal balance already set)
• Combine with other non-merged LoRAs if needed

For ComfyUI basics and LoRA loading, our essential nodes guide covers fundamental workflow construction.

Batch Processing with Merged LoRAs

Merged LoRAs simplify batch workflows:

Without Merging:

For each image:
  Load LoRA A
  Load LoRA B
  Apply strength A
  Apply strength B
  Generate

With Merged LoRA:

Load Merged LoRA
For each image:
  Generate

Simplified workflow, faster processing, consistent results.

Training Considerations

If you're training LoRAs specifically for merging:

Training for Merge Compatibility:

• Use similar training parameters across LoRAs
• Same network rank and alpha
• Compatible learning rates
• Similar training duration

LoRAs trained consistently merge more predictably. Our Flux LoRA training guide covers training parameters that affect merge compatibility.

Troubleshooting Complex Merges

When basic troubleshooting doesn't resolve issues:

Analyzing Weight Distributions

Examine weight statistics to understand merge behavior:

import torch
from safetensors.torch import load_file

lora = load_file("merged.safetensors")

for key in list(lora.keys())[:5]:  # Sample first 5 keys
    tensor = lora[key]
    print(f"{key}:")
    print(f"  Mean: {tensor.mean().item():.4f}")
    print(f"  Std: {tensor.std().item():.4f}")
    print(f"  Min: {tensor.min().item():.4f}")
    print(f"  Max: {tensor.max().item():.4f}")

What to look for:

• Very small means indicate cancellation (try TIES)
• High variance suggests conflicting sources
• Zero or near-zero tensors indicate pruned parameters

Block-by-Block Analysis

When results are partially correct, analyze by block:

Approach:

1. Generate with each source LoRA
2. Generate with merge
3. Compare which aspects match/differ
4. Map differences to block functions
5. Adjust block weights accordingly

For example, if colors match but faces don't, adjust weights in blocks that handle facial features.

Iterative Refinement

For difficult merges, use iterative refinement:

Process:

1. Create initial merge
2. Identify specific problems
3. Create new merge adjusting for those problems
4. Repeat until satisfactory

Keep notes on each iteration to avoid repeating failed approaches.

Future of LoRA Merging

The field continues evolving.

Emerging Techniques

Model Souping: Training multiple LoRAs with different initializations, then averaging. Can produce more robust merged models.

Learned Merge Functions: Neural networks that learn optimal merge functions for specific source types. Early research but promising.

Automated Merge Optimization: Tools that automatically search for optimal merge weights based on evaluation metrics. Reduces manual iteration.

Tool Development

Expected Improvements:

• Better GUIs for complex merges
• Automated quality evaluation
• Block weight presets for common scenarios
• Integration with training workflows

The barrier to quality merging continues lowering as tools improve.

Performance Optimization for Merged LoRAs

Ensure your merged LoRAs perform optimally.

File Size Optimization

Pruning Merged LoRAs: After merging, remove low-magnitude weights:

def prune_lora(lora_dict, threshold=0.01):
    pruned = {}
    for key, tensor in lora_dict.items():
        mask = torch.abs(tensor) > threshold
        pruned[key] = tensor * mask
    return pruned

Typical Savings:

• 10-30% size reduction
• Minimal quality impact at low thresholds

Inference Speed

Merged LoRAs load faster than multiple separate LoRAs:

Benchmarks:

• 2 separate LoRAs: ~500ms load time
• 1 merged LoRA: ~300ms load time

For batch processing or interactive use, this compounds into significant time savings.

For comprehensive generation speed optimization, our performance guide covers techniques beyond LoRA loading.

Final Conclusion

LoRA merge creates unified models that combine capabilities from multiple sources without the conflicts that stacking produces. The LoRA merge process involves choosing compatible sources, selecting appropriate weights and algorithms, and evaluating results against your sources and goals.

Start simple with weighted sum LoRA merge of two compatible LoRAs. Evaluate carefully and adjust weights until you achieve the balance you want. Graduate to advanced LoRA merge algorithms like DARE and TIES when weighted sum doesn't handle conflicts well enough.

LoRA merge is a creative tool that expands what you can achieve with trained LoRAs. The merged models you create through LoRA merge represent new capabilities that didn't exist in any single source. With practice, you'll develop intuition for how different sources combine and what LoRA merge algorithms work best for different situations. For complete beginners, our beginner's guide to AI image generation provides essential context.

For users who want combined LoRA effects without manual LoRA merge, Apatero.com provides access to professionally merged and tuned LoRA combinations that deliver specific effects reliably.