A phone with work to do has two bad options.

01Two bad options

Work arrives on a phone faster than the phone wants to deal with it. It has two ways out.

The first is to compute locally, which means clocking the CPU up. Silicon charges steeply for that. Dynamic power in a chip scales with the cube of clock frequency, so the relationship runs the other way when you read it as throughput: quadruple the power and you get about sixty percent more work done; to actually double the work you need eight times the power. The first watt is cheap. The eighth is nearly wasted. In the simulator this is the whole local path — power in, a cube root out, a small dent in the queue.

The second is to transmit, sending the work over the uplink to a server sitting at the base station. Here the penalty is logarithmic rather than cubic, which is Shannon’s law: doubling transmit power does not double the data rate, it adds a fixed amount to it, and how much it adds depends entirely on how good the link happens to be at that instant. On a good link a little power moves a lot of data. On a bad one you can push the dial to the ceiling and barely dent the queue.

So the trade is not between a good option and a bad one. It is between two different shapes of diminishing return, and which shape is currently cheaper depends on the radio channel — which nobody controls and which drifts continuously. The channel is modelled as a first-order autoregressive process with correlation 0.95: most of this millisecond’s value is inherited from the last one, with a fresh random nudge on top. You can never plan around it. You can only react to it, every slot, forever.

02What we simulated

One cell. Three mobile devices, one transmit antenna each, all sitting 100 units from a base station that has four receive antennas and hosts the edge server. Each device keeps a buffer of unprocessed work. Work arrives as a Poisson process — average one unit per slot for device one, two for device two, three for device three — and leaves the buffer two ways: bits computed locally, and bits transmitted to the server.

A slot is one millisecond, and one slot runs like this:

1. Each device reads its current channel vector — the complex gain of its path to each of the four receive antennas, after path loss.
2. Each device sets its two dials: transmit power and local CPU power, each anywhere in the range 0 to 2.
3. The base station stacks the three channel vectors into a 4 × 3 matrix and takes the Moore–Penrose pseudo-inverse. That is zero-forcing: the receiver algebraically cancels the other users’ signals, leaving each device with an effective noise level set purely by channel geometry. The reciprocal of that noise is what the code calls SINR and feeds back to the device.
4. Bits leave the buffer along both paths at once — log2(1 + P_transmit · SINR) over the air, and a cube root of local power converted through a fixed cycles-per-bit constant.
5. A random Poisson batch of new work arrives.
6. Whatever is left is backlog, and the channel advances one autoregressive step.

Three things we assumed away, all of them load-bearing.

The edge server is infinite. Unlimited CPU, unlimited storage, everything offloaded processed instantly and for free. The optimisation models the device and the uplink, not the server.

Zero-forcing is perfect. The function that computes SINR takes the users’ transmit powers as an argument and never reads it. In the simulator, one phone’s transmit power therefore never degrades another phone’s link. Combined with the infinite server, that means no device’s action enters any other device’s transition or reward at all. The three devices are coupled on paper and independent in the code. “Decentralised” here is an honest description of what each agent can see. It is not a claim that we tested how agents interfere.

Delay is a proxy. There is no Little’s-law computation and no delay variable anywhere in the codebase. What the report plots as “Average Delay” is mean buffer occupancy — backlog. A long queue means someone is waiting, which is the right direction, but it is not a measured latency and this page never calls it one.

03Turning it into a learning problem

None of the above is yet a learning problem. It becomes one when you stop trying to derive the right power split and instead let each device discover it by being scored.

The vocabulary is small and worth pinning down, because all of it is about to get used. An agent is the thing making decisions — here, one per device, and nothing more exotic than a small neural network. A policy is the mapping the agent is trying to learn: from what it can see to what it should do. A reward is a single number handed to the agent after every decision, saying how that moment went. An episode is one run of the system from a reset state to a stopping point; here, 99 consecutive slots. The agent does not get told what the right action was. It only ever gets told what the last action was worth.

What the agent seesis three numbers. How much work is queued right now. How good the link was, as the SINR the base station fed back. And the strength of its own radio path, the squared channel norm normalised by noise power. That is the entire observation — no other device’s queue, no other device’s power, no arrival-rate parameter, no clock.

One detail worth stating because it is a real property of the setup rather than a bug: the device acts on the previous slot’s SINR. It sets its dials from a state whose SINR belongs to the last slot, even though the channel_gain component beside it has already advanced to this one — a mismatched pair, not a blind one.

What the agent does is set two numbers, each between 0 and 2: transmit power and local CPU power. Both draw on the same battery.

What the agent is scored on is one number per slot, and it is always negative. This is a problem where you can only lose; the job is to lose less.

reward = − t_factor · (P_transmit + P_local) · 10 − (1 − t_factor) · backlog

P_transmit + P_local is the total power drawn this slot, taken straight from the action. backlogis the queue length after this slot’s work has drained and this slot’s arrivals have landed. t_factor is a single trade-off knob between 0 and 1 that sets both weights at once, and it is 0.2 in every script that shipped.

That knob is the only place in the entire system where a human’s preference enters. Setting it to 0.2 gives the power term a weight of 0.2 and the backlog term a weight of 0.8 — we were saying that a unit of queue matters four times as much as a unit of power. That was the intent, and it is the intent the report describes.

The shipped arithmetic is not quite that. A hard-coded ×10 sits on the power term, undocumented and unexplained anywhere in the source or the seventeen design notes, so the coefficients the agent actually optimises against are 2.0 on power and 0.8 on backlog. Per unit, power carries the larger coefficient. Whether the tilt survived in practice depends on the ranges: backlog runs in the tens across most of training while total power is capped at 4, so my expectation is that the queue term still dominates the magnitude. That is an expectation, not a measurement. We never ran it the other way, and nothing in the results isolates the two terms.

Nothing in that expression tells an agent what to do. It never says “offload when the channel is good.” It says only that power costs you something and a queue costs you something. What it does predict is a cross-device pattern: a device that takes this reward seriously should spend power to keep its queue short, and the device with the heaviest arrival stream should spend the most. Section 05 has the power numbers, and section 06 explains why two thirds of them cannot be used to check that prediction.

Learning, concretely, is bookkeeping across an enormous number of these moments. After every slot the agent files a five-part record — the state it saw, the dials it set, the score it got, whether the episode ended, and the state it landed in — into a replay buffer, a store of past experience holding up to 250,000 records. Then it pulls 64 of them back out at random, old ones included, and adjusts its network on that batch.

What it adjusts toward is the part worth being precise about, because it is the part people usually get wrong when explaining this. The agent does not wait to see how the episode turned out and then compare. It keeps a running estimate of how bad the future looks from any given state. One slot later, holding one real score, it has a slightly better-informed estimate of the same quantity — and the gap between the old estimate and the new one is the entire learning signal. A guess corrected by one step of reality. That is why an agent can learn in the middle of an episode rather than at the end of it, and it is why both learners here need a second copy of their own network, which is the subject of the next section.

Each agent got 111 episodes of 99 slots: 10,989 decisions per device, about 33,000 training updates across the three. That is a small number for reinforcement learning, and it matters later.

04Two ways to learn the same thing

We ran two learners over the identical environment. They differ in one structural way, and everything else follows from it.

DQN cannot emit a real number. It learns a value function: for a given state, an estimate of the total future score obtainable by taking each available action. To use it, you compute all those estimates and take the largest. That requires the action list to be finite, so the two power dials were quantised onto five notches each — 0, 0.5, 1.0, 1.5, 2.0 — and the pair flattened into a single index over 25 joint actions. The network takes the three-number state and outputs 25 numbers.

DDPG does emit real numbers, by splitting the job across two networks. An actor takes the state and outputs a power pair directly, anywhere in the continuous box from 0 to 2. A critic takes the state andthat pair and scores it, exactly as DQN’s value function does but for one specific action rather than all of them. The actor is then trained by asking the critic which direction its score is increasing in, and stepping that way. Neither network ever enumerates an action space; the critic’s gradient is what stands in for the argmax.

Two pieces of machinery are shared, and both exist to fix a specific pathology.

The first is the replay buffer already described. Consecutive slots in this system are almost the same slot — the channel is 95% inherited, the queue moves by a fraction — so training on experience in the order it arrives means feeding a network a stream of near-copies. It overfits the last few seconds and forgets everything else. Sampling 64 records at random from a long history breaks that correlation.

The second is the target network, and this is the one to hold on to. The learning signal above compares the agent’s current estimate against its own estimate one step later. Both come from the same network. Improve the network and you move the thing you were measuring against, which is a feedback loop that can chase itself out of the sensible range entirely. The fix is to keep a second, slowly-updated copy of the network and compute the comparison against that. Here every target network trails its online counterpart by a soft update: after each training step, the target absorbs 0.1% of the online weights and keeps 99.9% of what it had. A moving target, moved slowly enough to be treated as fixed.

That is the mechanism this project got wrong, and the audit is mostly about it.

05What the corrected runs show

These are not the 2024 numbers. The original submission was a single run per algorithm, seeded from the wall clock, on code with two defects the next section describes. What follows is the fixed code, run five times per algorithm with controlled seeds; every curve is the mean across those seeds, and every table cell carries a bootstrap confidence interval in the repository. The 2024 figures, and the story of how they changed, are in the audit below.

The saved arrays are per-episode averages, 111 episodes by 3 users, and the numbers below are the mean of the last twenty episodes — the closest thing to an end-of-training figure the data supports. One caveat for the middle column: the saved power array is not the raw P_transmit + P_local the reward is computed from; it is that sum minus over_power, the refund the simulator credits back when the local CPU was set to drain more work than the buffer held.

The movement across training is the more telling half. Mean reward over the first twenty episodes against the last twenty: DDPG started at −28.6, −65.3 and −108.1 for the three users and ended at −4.8, −10.8 and −12.4. DQN started at −6.1, −9.9 and −29.7 and ended at −4.2, −7.0 and −15.5.

Both learners improve now — and that is the first thing the fix changed. DDPG begins far worse and climbs a long way, steepest on the heaviest user. DQN, which in the original run had barely moved from start to finish, now clearly learns too: on the busiest device it roughly halves its backlog cost over training. The 2024 report had read DQN’s flatness as stability from a small discrete action space. It was not stability. It was a comparator that never really trained — the audit has the reason.

The head-to-head is no longer a clean win for either side. On the two lighter devices DQN finishes ahead — five discrete power levels are enough when the queue is easy to hold down. On the heaviest device DDPG wins, and wins with a much tighter confidence interval: continuous control lets it spend power smoothly to keep the queue short, which is exactly what a reward that weights delay four times heavier than power asks for. So the advantage of continuous control is real but specific — it shows up on the hardest user, not as the blanket victory the original write-up claimed.

The two metrics behind the reward, for device 1

Reward is a weighted sum of power drawn and backlog carried, so the only way to see what an agent traded is to plot both. These are device 1 in each algorithm.

The other two devices

In the original code these panels were meaningless — devices 2 and 3 trained against device 1’s network. With the wiring fixed they are now genuine, and they carry the finding: DQN holds its own on device 2, while device 3 is where DDPG separates.

06The audit — and the fix

I went back to this code two years after submission to write it up properly, and read it harder than I had read it while we were building it. Three defects came out of that. I confirmed the first two by running the code rather than by reasoning about it, because they are exactly the kind of thing that reads as fine on the page. The third is visible in two lines of source.

This section diagnoses the original 2024 code. All three defects have since been fixed — the corrected results in the previous section are what the fixes produced, and a regression test now guards the one that mattered most. Each defect below ends with what the fix was. The original submission is preserved verbatim under a git tag, so the before and after are both reproducible.

The target networks are cross-wired between agents

Every user gets its own agent, its own networks, its own target networks and its own replay buffer. They share only the TensorFlow session and the environment object. That was the design and it is what the code appears to describe.

It is not what the code does. The DDPG critic slices TensorFlow’s global list of trainable variables using a per-agent variable countas if it were an absolute index into that list. For the first agent constructed into a fresh graph, the count and the index coincide and everything is correct. For the second and third agent they do not, and the slice lands back at the top of the list — on user 1’s weights. The DQN Q-network is blunter still — it takes the whole global list with no offset, so each successive agent’s parameter set swallows every agent built before it (measured lengths 6, 18, 30 across the three). The shapes match where it matters, so nothing raises, nothing warns, and the training loop runs to completion.

Two probes, both on the DQN:

Printing the variable each target network actually reads from returns the same tensor for all three agents: user 1’s first fully-connected weight matrix.

Then the behavioural version, which is the one that settles it. Stamp the three online networks with the constants 1, 2 and 3. Zero all three target networks. Run exactly one soft update at τ = 0.001. Correct wiring gives 0.001, 0.002, 0.003 — each target absorbing a thousandth of its own online network.

expected   0.001   0.002   0.003
measured   0.001   0.001   0.001

Users 2 and 3 spent the entire run being pulled toward user 1’s network.

The scope is worth being precise about, because it is not uniform. The DDPG actoris wired correctly — it captures its variables as a delta around its own construction rather than as an absolute offset, which is the pattern the other two classes should have copied. So the actor is clean. But the actor’s only learning signal is the gradient the critic hands it, and the critic is broken for users 2 and 3. Corrupt input to a correct mechanism is still corrupt output. DDPG users 2 and 3 are compromised end to end.

User 1 was correctly wired in both algorithms — in the original code, the single reason any curve was trustworthy. The fix: capture each network’s variables as a delta around its own construction (the pattern the actor already used) instead of an absolute offset. The same behavioural probe now returns 0.001, 0.002, 0.003, and a regression test asserts it on every build so the bug cannot return silently. It turned out to be inherited verbatim from the paper authors’ own released code, not something we introduced.

The exploration rate never decayed

DQN explores with ε-greedy: with probability ε take a uniformly random action, otherwise take the one the network currently rates highest. ε starts at 1.0 — pure random — and decays toward zero as the agent learns, so the run shifts gradually from gathering experience to using it.

The decay factor is 0.999976 per decision. That is a schedule sized for the hard-coded step budget of 100,000 that sits next to it in the file. Our run is 111 episodes of 99 steps: 10,989 decisions. Over 10,989 decisions, ε ends at 0.768.

Stacked on top of that is a warm-up constant of 5,000 steps, during which actions are forced uniformly random regardless of what the network thinks. That is 45.5% of the entire run before the policy gets a vote at all, and ε is still 0.887 when it ends.

Put together: roughly nine percent of the DQN’s actions across the whole run came from its policy. The rest were coin flips. The replay log agrees. It records all three agents’ transitions — 32,967 lines — and across them 24 of the 25 joint actions appear between 1,118 and 1,320 times each, against a uniform expectation of 1,319. The one exception is action 0, no power on either dial, which appears 4,205 times, and whose share of the agents’ choices climbs steadily through training with the jump landing exactly at the 5,000-step warm-up boundary. That climb is the only visible policy in the whole run.

This is also the reading to apply to DQN’s power column. Uniform random over the 25-action grid draws 2.0 on average, so device 1’s plotted 1.100 is not a learned frugality: it sits below the random draw mainly because the saved array subtracts the over-provisioning refund, which is large when the buffer is near empty.

This is what makes the 2024 reading of the flat DQN curve wrong. The report credited the flatness to fast convergence in a small discrete action space — the agent finding a good policy early and holding it. The available explanation is simpler and does not require the agent to have learned anything: a policy that never stops exploring never changes. You cannot distinguish “converged” from “still random” by looking at a flat line, and here the arithmetic on the ε schedule tells you which one it was.

It also means DQN was never a tuned baseline. It was a mostly-random controller with a small learned tilt toward switching both dials off, and DDPG was compared against that.

The fix: replace the fixed decay constant with a linear schedule scaled to the actual training budget — anneal from 1.0 to a 0.05 floor over the first 80% of the run, after a short warm-up. ε now reaches its floor around episode 90 instead of ending at 0.77, and the corrected DQN in the previous section is the result: a comparator that actually trains. Like the wiring bug, the original schedule was inherited from upstream, where it was correct for a run 36× longer than ours.

The two evaluation runs did not start from the same place

The environment’s reset takes an isTrain flag. With it false, buffers and SINR start at zero; with it true, the starting buffer is randomised. The DDPG evaluation driver passes false. The DQN evaluation driver calls reset with no argument at all, and the default is true.

So the two evaluation runs began from different buffer distributions. It is a one-word difference between two files that were plainly written by copying one from the other, and it is the smallest of the three defects. Alongside it, the DQN evaluation driver constructs a fresh agent with its exploration rate back at its starting value, which means the DQN “evaluation” re-runs a near-random agent rather than measuring a policy.

It did not corrupt anything on this page, because neither driver produced anything on this page: both shipped result files are training runs. That is the larger point. The project has no evaluation protocol at all — the two drivers disagree with each other, and the quantity they exist to measure was never measured.

The fix: both evaluation drivers now reset the same way and load the corrected, seeded checkpoints, so a greedy evaluation is finally possible — though the numbers on this page are still the training curves, which is what the multi-seed campaign produced.

And there is nothing to compare against

Not a defect so much as an absence, but it constrains the conclusions harder than any of the three. The repository contains no baselines. No uniform-random controller, no fixed-power controller, no greedy all-local or all-offload policy, no per-slot analytic optimum. So there is no evidence anywhere in this project that either learner beats a device that simply transmits at 1.0 every slot and never thinks about it.

Two dead helper functions invert the offload path — power required to send a given number of bits, once from the channel norm and once from SINR. Nothing calls either. The matching local-path inverse is live, used for the over-provisioning refund. They are the residue of a greedy baseline that was removed; the environment still carries a comment describing itself as specially designed for greedy agent training.

None of these three are exotic. Two of them are a variable-scoping mistake and an arithmetic mismatch between a decay constant and a loop bound. They survived because nothing in the project ever checked them: no test asserted that two agents held different weights, and nobody divided 100,000 by 10,989.

07What survives

Being precise about this matters more than being generous in either direction.

The simulator stands. The physics is implemented as documented: Shannon capacity for the uplink, a cube-root frequency-to-power model for local compute, AR(1) Rayleigh fading with a path-loss term, and a genuine pseudo-inverse zero-forcing receiver over the 4 × 3 channel matrix. The defects are all in the learning code. Nothing about them touches the environment. Its limitations are the ones declared above — infinite server, perfect zero-forcing, backlog as a delay proxy — plus one more: the buffer-overflow penalty is commented out in every terminal class, so the nominal capacity of 100 is never enforced and the saved overflow array is identically zero in both runs.

All three users are now correctly wired, in both algorithms, and the run is repeated over five seeds — so the three-user comparison, which the original code could not support, is now a real measurement with confidence intervals.

Both learners train. DDPG improves steeply from a poor start; DQN, which in the original run looked flat, now clearly learns once its exploration is scaled to the budget. On the corrected numbers, neither dominates: DQN finishes ahead on the two lighter devices, and DDPG wins on the heaviest — and with a tighter confidence interval, which is its own result about stability under load.

What does not survive:

The headline.The 2024 write-up’s claim that DDPG “significantly outperforms” DQN was an artifact of a crippled comparator. Once DQN explores properly and every agent trains against its own network, the blanket win becomes a workload-dependent split. Continuous control helps — specifically on the hardest user — but it is not the across-the-board victory the report described.

The mechanism the report credited.The claim that DQN’s original flatness demonstrated fast convergence in a discrete action space is not supported. The ε arithmetic supplied a simpler explanation that required no learning at all — and once the schedule is fixed, the flatness is gone.

Any absolute claim of competence, for either learner. The repository still has no baseline — no random, fixed-power or greedy controller — so “DQN beats DDPG on device 1” is a statement about the two learners relative to each other, not evidence that either beats a device that simply transmits at 1.0 every slot and never thinks about it. Adding those baselines is the next real step.

And one thing we could never have claimed either way: what the policy actually did.The split between bits offloaded and bits computed locally is calculated on every single slot and written to disk on none of them. No array, log or figure anywhere in the project records the offload fraction against link quality. So the most natural sentence to write about a system like this — that the agent learned to transmit when the channel was good and fall back on the CPU when it wasn’t — is one I have no evidence for. It may well be true. Nothing here shows it.

08What I would do differently

Roughly in order of how much each would change the conclusions. The first three are done — they are what produced the corrected results above; the rest are the honest next steps.

Done — a test that would have caught the wiring. Construct three agents in one graph, assert that no two of them share a target variable, and assert the behavioural version: stamp the online networks with distinct constants, zero the targets, run one soft update, check you get three distinct numbers. That is about fifteen lines and it fails loudly on the shipped code. The general lesson is narrower than “write tests” — silent shape-compatible aliasing between supposedly independent models is a failure mode with no symptom, so it needs a test that asserts independence directly rather than a test that checks the loss goes down.

Done — size the exploration schedule to the run you are actually doing. A decay constant is meaningless without the step count next to it. And the broader version of that mistake is the one worth naming: the comparator has to be tuned as hard as the method you are promoting, or the comparison measures your attention rather than the algorithms.

Done — fixed, recorded seeds with confidence bands on every curve. The original seed was derived from the wall clock, so not one figure could be regenerated. It now runs five fixed seeds per algorithm with bootstrap confidence intervals; a single run of 111 episodes takes about four and a half minutes, so there was never a resource reason for n=1. (Five, not the ten first planned — enough to separate the curves that separate; more would tighten the bands.)

Real baselines, built before the learners. Uniform random, all-local, all-offload, and the per-slot convex optimum by one-dimensional bisection — the last of which is computable here, since the reward is well behaved in each power dial. Without a floor and a ceiling, a learning curve is a shape with no scale.

Log the offload split.Both quantities already exist in the step function. Adding them to the saved arrays is a one-line change and it is the difference between “reward improved” and a policy you can actually look at and argue with.

Initialise the target networks with a hard copy. The function named init_target_network runs the same 0.1% soft blend as every other step, so the targets begin at their own random initialisation and absorb a thousandth of the online weights on the first call. It is a smaller error than the cross-wiring, but it is the same class of error: a name that describes an intent the body does not implement.

TD3 and SAC, not more DDPG.DDPG’s characteristic failure is that the critic overestimates action values and the actor then optimises straight into the overestimate. TD3 addresses that with two critics and a pessimistic minimum over them, by updating the actor less often than the critics, and by smoothing the target action with noise so the critic cannot exploit a single sharp peak. SAC changes the objective instead: it adds the policy’s own entropy to what is being maximised, so exploration is something the agent learns to modulate rather than something injected from outside at a fixed rate — which is precisely the knob this project got wrong on the DQN side and never scheduled at all on the DDPG side, where the exploration noise ran at constant amplitude for the entire run.

Not MADDPG, and not QMIX.These are the obvious names to reach for on a page with the words “multi-agent” and “decentralised” on it, and they would do nothing here. Both exist to handle non-stationarity caused by other agents’ policies changing — agent A’s environment shifting because agent B learned something. That effect does not exist in this simulator. The SINR function ignores the transmit powers, and the server is infinite, so no device’s action appears anywhere in another device’s transition or reward. What we have is three independent decision processes whose observations happen to be statistically correlated through a shared channel matrix. Centralised training with decentralised execution has nothing to centralise. If I wanted the multi-agent problem to be real, the first change would be to the environment, not the algorithm: an MMSE receiver instead of perfect zero-forcing, so one device’s power genuinely raises another’s noise floor, and a finite server so offloaded work contends for something. Then the coupling exists and modelling it means something.

09Code, and the people

The full technical README documents the environment, both algorithms, every constant, the configuration reference, and a longer version of the defect list than this page has room for. It is the manual; this page is the argument.

The system was built from the setup in Chen & Wang, Decentralized computation offloading for multi-user mobile edge computing: a deep reinforcement learning approach, EURASIP Journal on Wireless Communications and Networking, 2020, article 188 — same environment model, same reward structure, same DDPG-versus-DQN framing. We never compared our numbers against theirs.

Team 5, DLFA programme, Centre for Continuing Education, IISc Bangalore, November 2024: Anaz Jaleel, Raghu Koratagere, Santosh Bagali, Shivajith Mutteal. The simulator and both learners were built by the four of us. The audit in the last three sections is mine and postdates the submission by two years.