Scheduling + memory overcommit for KVM fleets

Stop paying for empty space.

Run more VMs on every worker you already have, without missing your launch latency, job runtime or reliability targets. Binpacked is a scheduling and memory overcommit layer for bare-metal and nested KVM fleets.

Same workers. More VMs. Targets met.

Fig. 1One fleet, placed three ways

Placement

VMs sit wherever there was room at launch. Every worker is partly empty, and most guests hold memory they never touch.

VMs per worker
10
Memory allocated
–of RAM on busy workers
In use at peak
–of RAM on busy workers
  • In use
  • Held, never touched
  • Unallocated
View as table
Memory on each worker for the selected placement, as a share of its 64 GiB
WorkerVMsAllocatedIn use at peak
Simulation. The same VMs on 64 GiB workers appear in every frame. Only their placement changes.

01The problem

Two kinds of empty.

A worker runs out of memory on paper long before it runs out in practice. Two kinds of empty space sit in between, and each one needs a different fix.

Fig. 2Anatomy of one worker

  • 23% In useMemory guests are actually touching.
  • 33% Held, never touchedGiven to a guest that is not using it.
  • 44% UnallocatedNot given to any guest.
Illustration. Each cell is one gibibyte of RAM on a 64 GiB worker.

Kind one · Gaps between guests

Space no guest was given.

Launches land wherever there happens to be room. Each worker is left with a gap that is too small for the next VM, so that VM goes elsewhere and leaves a gap of its own. Across a fleet, the leftovers add up to whole workers.

Removed by bin packing

Kind two · Idle memory

Space a guest holds and never touches.

A VM is sized for its worst moment, and its worst moment is rare. The rest of the time a good part of its RAM goes untouched. If every allocation is backed one for one, that memory is spoken for and doing nothing.

Removed by memory overcommit

Packing allocations tightly still packs air. Overcommitting scattered workers still strands capacity. You need both.

02How it works

Know what fits. Then make room.

Binpacked has two parts. A scheduling layer decides where each launch goes and holds its place. A host agent on every worker makes memory overcommit something you can rely on.

Part one · Scheduling

Every launch gets a place that is really there.

Binpacked tracks what fits on each worker, so launches can fill workers up instead of spreading thin. The moment a launch is placed, its capacity is reserved.

  • One launch, one reservation. Capacity is set aside for each launch as it is placed, so two concurrent requests can never claim the same resources.
  • Worker by worker. What is placed, what is reserved and what is still free is tracked for every worker in the fleet.
  • Memory first. RAM is promised to a guest whether or not it is ever touched, which usually makes it the first thing a worker runs out of. Binpacked starts there.

Fig. 3Two launches, one gap

Without reservations

worker-a12 GiB free

Before81%
After106%

Over its limit. Both launches read 12 GiB free, and both land here.

worker-b24 GiB free

Before63%
After63%

Had room the whole time.

A reservation per launch

worker-a12 GiB free

Before81%
After94%

The first launch reserves its 8 GiB as it is placed.

worker-b24 GiB free

Before63%
After75%

The second sees 4 GiB left on worker-a and is placed here.

Illustration. Two launches arrive at the same moment, each needing 8 GiB. Percentages are of each worker’s limit, 64 GiB here.

Two ways in

Replace your scheduler, or keep it.

Binpacked can be your VM scheduler, or it can work alongside the one you have. Either way it tracks what fits on each worker and reserves capacity for each launch.

Option one

Binpacked schedules.

Replace your VM scheduler with ours. Binpacked decides where each launch goes and reserves the capacity for it.

Option two

Your scheduler asks.

Keep your VM scheduler and have it use our capacity API. Binpacked still tracks what fits and still holds the reservation.

Part two · Memory overcommit

Guests hold more than they touch.

Overcommit means promising guests more memory than a worker has, on the grounds that they never touch all of it at once. On its own that is a bet. The host agent on each worker is what makes it practical.

  • Guaranteed where it matters. Sensitive workloads keep guaranteed memory, and the host agent protects it.
  • Borrowed where it is spare. Eligible VMs can borrow RAM that is sitting idle on the worker.
  • Back before it is reused. When memory is reclaimed from a guest, the host agent verifies that it has returned to the host before more work is admitted.

Fig. 4Five guests on one worker, over one day

  • Combined memory in use
  • Each guest, with its own peak
  • Sum of those peaks
Sum of the five peaks
–what the guests are allocated
Most in use at once
–what the worker has to back
Never in use at once
–of what was allocated
View as table
Memory in use by each guest, in GiB, hour by hour
Illustration with synthetic guests. Each one is allocated exactly what it uses at its own peak, and together they still never need all of it at once. That gap is only usable if memory a guest stops using really does return to the host.

03Safety

Overcommit is a dial, not a cliff.

Memory overcommit earns its bad reputation when it is done blind: one ratio, set once, applied to every host. Binpacked’s limits are specific, and they tighten when a host gives them reason to.

  • Guaranteed memory

    Sensitive workloads keep the memory they were promised. The host agent protects it, whatever their neighbours are doing.

  • Headroom for boot bursts

    A guest needs more while it boots than once it settles. Capacity is held back so that launches have somewhere to land.

  • Tighter when a host gives reason

    When a host fails or its telemetry goes stale, admission tightens. New work is not placed on numbers nobody can vouch for.

  • Held to your targets

    Density only counts if launches stay fast, jobs finish on time and hosts stay up. Your launch latency, job runtime and reliability targets are the test.

04Calculator

Run your numbers.

Five inputs and three formulas. Nothing leaves your browser.

An average across a worker pool is fine.

What the workers themselves cost. Clear it to leave money out.

75%

The memory your VMs are configured with, added up, as a share of the RAM on the worker.

60%

A guest’s resident memory at its busiest, as a share of what it was given.

75%

Memory really in use, as a share of the worker’s RAM, at its busiest moment. Whatever is left is headroom.

66VMs per worker, where 40 run today. That is 1.6 times as many on the workers you already have. Hosting the VMs you run today would take $19,697 a month less in workers, or $236K a year.

Where a worker’s RAM goes today

  • In use at peak
  • Held, never touched
  • Unallocated
Today
40VMs per worker$50,000 a month
Packed tighter
488 more per worker$8,333 a month less
With overcommit
6626 more per worker$19,697 a month less

Rather have real numbers? We will collect them from your fleet and tell you whether Binpacked is a good fit.

How this is calculated

packed tighter = today × 90% ÷ allocated

with overcommit = today × target ÷ (allocated × touched)

spend at that density = spend × today ÷ new VMs per worker

This is arithmetic on your own inputs, not a forecast. It treats guests as roughly the same size, rounds down to whole VMs, and assumes packing by allocation tops out at 90% of a worker’s RAM, because odd sizes and placement rules always leave some gaps. It looks at memory only. CPU, disk or anything else can cap a worker first, and your fleet will move every one of these numbers.

The dollar figures are what it would cost to host the VMs you run today at each density. They turn into savings only if the workers you no longer need are handed back or given other work. On hardware you own, read them as capacity you do not have to buy.

05Getting started

From first look to fully packed.

  1. Collect

    We collect data from your fleet as it runs today: what is on each worker, how much memory guests are allocated, and how much of it they touch. Nothing changes yet, and you find out whether Binpacked is a good fit.

  2. Run in shadow

    In shadow mode Binpacked makes its own placement decisions next to your current scheduler, and none of them take effect.

  3. Compare

    Set its decisions against the ones your scheduler made, for as long as it takes to trust them.

  4. Switch one pool

    Move a single worker pool across and watch your launch latency, job runtime and reliability. Then decide about the next one.

06Questions

Fair questions.

What is Binpacked?

A scheduling and memory overcommit layer for bare-metal and nested KVM fleets. It tracks what fits on each worker, reserves capacity for each launch, and puts a host agent on every worker to make memory overcommit practical. The point is more VMs per worker with your targets still met.

Do we have to replace our scheduler?

No. You can replace your VM scheduler with ours, or keep yours and use our capacity API. Either way Binpacked tracks what fits on each worker and reserves capacity for each launch, so concurrent requests cannot claim the same resources.

Does it need bare metal?

No. It runs on bare-metal KVM fleets and on nested KVM, where each worker is itself a virtual machine.

What is memory overcommit?

Promising guests more memory in total than a worker physically has, on the grounds that they never touch all of it at the same moment. The engineering is in deciding which guests take part, how far to go, and what happens when memory gets tight.

Isn’t memory overcommit risky?

It is when it is done blind. The host agent is there so that it is not. It protects guaranteed memory for sensitive workloads, lets only eligible VMs borrow spare RAM, and verifies that reclaimed memory has returned to the host before more work is admitted. It holds capacity back for boot bursts, and it tightens admission when a host fails or telemetry goes stale.

Why memory, and not CPU?

Because that is where the room is. When CPU is oversubscribed, things run slower. When memory runs out, something gets killed. So memory tends to be the resource a fleet is most careful with, and the one with the most idle capacity sitting in it. It is also the harder problem, which is why Binpacked puts a host agent on every worker.

How do we find out whether it is a good fit?

Ask us. We will collect the data from your fleet as it runs today: what is on each worker, how much memory guests are allocated, and how much of it they touch. Nothing changes while we do. Then we tell you whether Binpacked is a good fit, and we say so if it is not. Get in touch to start.

How many more VMs per worker?

It depends on two things: how much of a worker’s RAM is allocated to guests, and how much of that the guests actually touch. The calculator does the arithmetic with your inputs. We would rather collect the real numbers from your fleet than quote you an average.

What about warm sandboxes?

Planned, not shipped. The idea is that for warm sandboxes the host agent could recommend when to keep a guest resident, when to reclaim its memory, and when to ask the runtime to pause it. If that is your workload, we would like to hear how you run it.

Is any of this new?

No. Hypervisors have overcommitted guest memory for more than twenty years, and the largest fleets pack and overcommit as a matter of course. VMware described its approach in a 2002 paper, and Google described Borg’s in 2015. What is hard is doing it on your own fleet without missing your targets. That is the part Binpacked is for.

See if it fits.

We will collect the data from your fleet as it runs today and tell you whether Binpacked is a good fit. If it is not, we will say so.

hello@binpacked.com