[{"content":"","date":"14 March 2026","externalUrl":null,"permalink":"/posts/","section":"Articles","summary":"","title":"Articles","type":"posts"},{"content":"","date":"14 March 2026","externalUrl":null,"permalink":"/tags/c/","section":"Tags","summary":"","title":"C","type":"tags"},{"content":"","date":"14 March 2026","externalUrl":null,"permalink":"/","section":"Daniel Miess","summary":"","title":"Daniel Miess","type":"page"},{"content":"","date":"14 March 2026","externalUrl":null,"permalink":"/tags/embedded/","section":"Tags","summary":"","title":"Embedded","type":"tags"},{"content":"","date":"14 March 2026","externalUrl":null,"permalink":"/tags/linux/","section":"Tags","summary":"","title":"Linux","type":"tags"},{"content":"","date":"14 March 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"Every couple of months someone asks me why we are writing C for a brand new embedded platform in 2026. The question is fair. Rust has memory safety. Go has goroutines and a standard library that makes networking trivial. Modern C++ has smart pointers and RAII. C has malloc and good luck.\nI get the skepticism. But after working in C across multiple embedded Linux platforms — cellular gateways, display drivers, telematics devices — I keep coming back to it for new projects, not out of habit, but because the practical realities of embedded development still favor it in ways that are hard to appreciate until you have shipped firmware on constrained hardware.\nThis is not a language war piece. I have used C++ extensively and liked it. I think Rust is genuinely interesting. But the decision about what language to write production firmware in is not an abstract one, and the factors that matter most are not the ones that get discussed in blog posts.\nThe Ecosystem Is the Language # When people compare languages, they compare syntax and features. When I pick a language for an embedded project, I am mostly thinking about the ecosystem.\nEvery vendor SDK I have ever worked with is in C. Qualcomm, Sierra Wireless, NXP — the reference code, the API headers, the example applications, all C. The Linux kernel interfaces are C. ModemManager and NetworkManager expose D-Bus APIs, but their plugin interfaces and internal APIs are GObject-based C. When you need to write a custom ModemManager plugin to support a new modem chipset, you are writing C whether you planned to or not.\nYou can call C from other languages. FFI exists. But FFI is a boundary, and every boundary is a place where bugs hide. Wrapping a vendor SDK that passes around raw pointers and expects you to manage buffer lifetimes in a language whose whole selling point is safe memory management creates friction. You spend time writing bindings instead of writing the actual application. For a small team shipping firmware on a deadline, that overhead matters.\nThe practical reality is that in embedded Linux, C is not just a language. It is the lingua franca that everything else is built on. Choosing C means you can use vendor code directly, contribute patches upstream without translation, and read kernel source to debug problems without a language barrier between you and the system.\nDebugging on Target # This is the one that matters most to me day to day, and it gets overlooked in language comparisons.\nWhen a device in the field does something unexpected, I need to reproduce it and debug it. On an ARM64 embedded board, that means cross-compiling, deploying to the device, attaching GDB over SSH, and stepping through code. This workflow with C is as good as it gets. GDB understands C natively. The mapping between source lines and instructions is direct. Stack traces are readable. Memory layouts are predictable. When I inspect a struct, I see the struct. When I set a breakpoint, I know exactly what code is going to execute.\nWith garbage-collected languages, the debugging story on embedded targets gets murkier. The runtime is doing things you did not ask for — garbage collection pauses, goroutine scheduling, heap compaction. These are fine on a server. On an embedded device where you are trying to figure out why a modem AT command timed out after exactly 3.2 seconds, runtime non-determinism is a problem. You need to know that between point A and point B in your code, nothing else happened. C gives you that.\nRust\u0026rsquo;s debugging story is better than Go\u0026rsquo;s for embedded — no garbage collector, predictable layout — but GDB support for Rust is still catching up. Complex Rust types like enums, iterators, and trait objects can produce debug info that GDB does not display cleanly. It is improving, but \u0026ldquo;improving\u0026rdquo; is not the same as \u0026ldquo;works reliably today on the ARM target board sitting on my desk.\u0026rdquo;\nI have lost enough time to tooling friction that I weight this heavily. The language you debug in matters more than the language you write in, because debugging is where you spend time when things go wrong, and things always go wrong.\nCross-Compilation Is a Solved Problem # Embedded development means cross-compilation. I write code on an x86 workstation and it runs on an ARM64 device. With C, the cross-compilation toolchain is mature to the point of being boring. aarch64-linux-gnu-gcc works. It has worked for years. The Yocto build system, which is the standard for building embedded Linux images, is built entirely around C and C++ toolchains. CMake, Make, autotools — the build system ecosystem assumes C.\nCross-compiling Rust for embedded ARM targets works, but the experience is not as smooth. You need the right target triple, the right sysroot, and if you depend on crates that have C dependencies (which many do), you are back to dealing with a C cross-compilation toolchain anyway, plus the Rust one on top. It is an extra layer.\nGo cross-compiles easily for the simple case, but CGo — which you need the moment you want to call into a C library — introduces the same cross-compilation complexity as C, with the added complication that the Go and C toolchains need to agree on linking.\nNone of these are unsolvable problems. But in embedded development, the build system is already one of the most frustrating parts of the job. Adding complexity to it for a language choice has a real cost.\nBinary Size and Runtime Overhead # On a server, nobody cares if your binary is 2MB or 20MB. On an embedded device with limited flash storage and a firmware image that ships over cellular connections, it matters.\nA C program links against libc, which is already on the device, and produces a small binary. A Go program bundles its entire runtime, including the garbage collector and goroutine scheduler. For a simple application, the Go binary can be 5-10x larger than the C equivalent. Rust is closer to C in binary size, but still tends to be somewhat larger due to monomorphization and the standard library.\nRuntime memory overhead follows a similar pattern. A C program uses what you allocate. A Go program needs memory for its runtime, its garbage collector, and its goroutine stacks. On a device where total RAM is measured in hundreds of megabytes and shared among multiple processes, the runtime overhead of higher-level languages eats into your budget.\nI have worked on devices where we carefully tracked memory usage per process because we had to. In that environment, a language that uses memory you did not explicitly ask for is a liability.\nThe Hiring Question # This one is less technical but equally real. When I need to hire an embedded developer, I need someone who understands pointers, memory layouts, hardware registers, and the Linux kernel interfaces. That person almost certainly knows C. They might also know C++ or Rust, but C is the baseline.\nThe pool of embedded developers who are productive in Rust is growing, but it is still small. Finding someone who can write safe, idiomatic Rust and understands the embedded domain — kernel interfaces, device trees, cross-compilation, hardware debugging — is hard. Finding someone who can do that in C is not easy either, but the pool is significantly larger.\nFor a team that needs to ship firmware and will need to hire people to maintain it, language choice is also a staffing decision. Picking a language that a small fraction of your candidate pool knows is a risk.\nWhere C Is the Wrong Choice # I do not think C is the right answer everywhere, even in embedded.\nNetwork-facing parsers and protocol implementations are where C\u0026rsquo;s weaknesses hurt the most. Parsing untrusted input with manual buffer management is where buffer overflows live. If I were writing a new TLS implementation or a protocol parser that handles data from the network, I would want Rust\u0026rsquo;s guarantees. The cost of a memory safety bug in code that faces the internet is too high.\nHigher-level application logic that does not touch hardware or kernel interfaces is a reasonable place for Go or Rust. If you have a component that reads from a message queue, transforms some data, and sends it somewhere — and it does not need to interact with C libraries or vendor SDKs — then the ergonomic benefits of a higher-level language might be worth it.\nNew projects with no vendor SDK dependency and a team that already knows Rust are a legitimate case for Rust on embedded. The language\u0026rsquo;s safety properties are real, and if you are not fighting the ecosystem to use it, the trade-off shifts in its favor.\nThe key word is \u0026ldquo;trade-off.\u0026rdquo; There is no free lunch. Rust\u0026rsquo;s safety comes with ecosystem friction and a steeper learning curve. Go\u0026rsquo;s productivity comes with runtime overhead and a garbage collector. C\u0026rsquo;s ecosystem fit comes with the responsibility of managing your own memory. The right choice depends on the specific constraints of the project, not on which language won the latest internet argument.\nWhat I Actually Do # On the platform I work on now — an ARM64 Linux telematics device — the firmware is C. The decision was made before I joined, but I agree with it. We interface with ModemManager, NetworkManager, and custom kernel modules. We talk to hardware over memory-mapped registers. We cross-compile with Yocto. We debug on target with GDB over SSH.\nEvery one of those activities is easier in C than it would be in any alternative. Not because C is a better language in the abstract, but because the entire toolchain, ecosystem, and workflow assumes C. Fighting that assumption would cost us time we do not have, for benefits that would not show up in the reliability of the shipped product.\nC in 2026 is not exciting. It does not have a type system that catches use-after-free at compile time. It will not generate conference talks about how you rewrote everything and it was better. But it works, the tools are mature, the ecosystem is deep, and when something goes wrong at 3am on a device in a truck somewhere, I can attach a debugger and see exactly what is happening.\nFor the kind of embedded work I do, that is what matters.\n","date":"14 March 2026","externalUrl":null,"permalink":"/posts/the-case-for-c-in-2026/","section":"Articles","summary":"Rust is promising, Go is productive, and C is what I still reach for when writing firmware for ARM Linux devices. This is not a language war — it is a practical look at why.","title":"The Case for C on New Embedded Projects in 2026","type":"posts"},{"content":"","date":"10 March 2026","externalUrl":null,"permalink":"/tags/ai/","section":"Tags","summary":"","title":"Ai","type":"tags"},{"content":"","date":"10 March 2026","externalUrl":null,"permalink":"/tags/c++/","section":"Tags","summary":"","title":"C++","type":"tags"},{"content":"","date":"10 March 2026","externalUrl":null,"permalink":"/tags/cellular/","section":"Tags","summary":"","title":"Cellular","type":"tags"},{"content":"A year ago, if you told me an AI tool would become one of the most useful things in my embedded development workflow, I would have been skeptical. Embedded work is registers, timing, memory constraints, and hardware quirks. It felt like the last place AI would be useful.\nTurns out I was wrong.\nThese tools have changed how I work day to day, and I want to talk about what that actually looks like when you are writing C for ARM Linux devices and debugging hardware in the field.\nWhat I Use # I work on an ARM64 Linux-based telematics device — C code, modem control, cellular connectivity, the whole stack from hardware abstraction up through application-level data collection. On the AI side, I primarily use Claude Code (Anthropic\u0026rsquo;s CLI tool) and Gemini. Nothing fancy. They just live in my terminal alongside everything else.\nThese are not side projects. This is production firmware running on large fleets. The stakes are real, which is part of why I find it worth writing about.\nThe Most Underrated Part: Connecting Your Information Sources # Before I get into specific use cases, I want to talk about something that does not get enough attention. The single most important thing you can do to get value out of AI tooling is to connect it to the information sources you already use.\nIn embedded work, the knowledge you need to do your job is scattered everywhere. The bug you are investigating has context in a JIRA ticket. The code change that might have caused it is in a GitLab merge request. The hardware behavior is documented in a spec sheet or datasheet. The device logs are in one place, the fleet data is in another.\nThe real power of AI tooling is not that it writes code for you. It is that it can pull context from all of these sources and synthesize it. When I can point Claude Code at a JIRA ticket, have it pull the relevant merge request, cross-reference that with device logs, and then look at the code — that is when it goes from a neat trick to a genuine workflow change. Instead of spending 30 minutes just gathering context before I can even start thinking about a problem, the AI does that legwork.\nGetting these integrations set up properly is not glamorous work. It is configuring MCP servers, setting up API tokens, making sure the tool can actually reach your JIRA instance and your GitLab repos and your BigQuery tables. But it is the foundation that makes everything else work. If your AI tool cannot see the same information you see, it is working with one hand tied behind its back.\nDebugging and Log Analysis # This is where AI has had the biggest impact on my productivity, hands down.\nEmbedded devices generate a ton of log data — syslogs, multilogs, modem traces, network manager output. When a device in the field starts acting up, I used to spend a long time manually going through logs, correlating timestamps, trying to build a picture of what happened.\nNow I feed those logs into Claude Code. What took me an hour of reading often takes minutes. It is good at the thing I find most tedious: correlating events across different subsystems. A modem disconnect event that lines up with a network manager state transition that lines up with an application retry — the AI spots those connections fast.\nMore than that, I can have a back-and-forth about it. \u0026ldquo;Why would ModemManager transition to this state here?\u0026rdquo; or \u0026ldquo;What would cause this sequence?\u0026rdquo; Questions that used to mean going and reading source code or documentation. Now I get a useful starting point in seconds and can verify it against the code.\nI want to be clear: I still verify everything. This is embedded software. Incorrect assumptions mean bricked devices in the field. But the debugging cycle has gotten dramatically shorter.\nWriting and Modifying C Code # This is where I expected AI to fall flat, and where it has surprised me the most.\nEmbedded C does not leave much room for error. Fixed-size buffers, hardware registers, strict memory constraints, code running on devices you cannot easily get your hands on once they are deployed. I figured AI-generated C would be sloppy and full of the kind of subtle issues that bite you three months later.\nIn practice, the quality is much higher than I expected. Not perfect — I catch issues regularly, and I review everything — but the baseline is solid. Where it really shines:\nBoilerplate. Setting up new modules, writing init and cleanup functions, defining structures with their helper functions. This stuff has to be correct but is not intellectually interesting. AI handles it well and saves me real time.\nRefactoring. Restructuring code, changing interfaces, updating a data structure that gets used in 40 places. AI makes consistent changes across a codebase without the kind of \u0026ldquo;I updated it in 39 of 40 places\u0026rdquo; bugs that happen when I do it manually.\nPattern translation. \u0026ldquo;Take this polling-based implementation and make it event-driven with callbacks.\u0026rdquo; I know how to do that transformation, but it is tedious and error-prone by hand. AI does it fast and consistently.\nThe important thing is that AI does not replace the thinking. I still design the architecture, make the trade-offs, pick the approach. It just makes implementing those decisions a lot faster.\nData Analysis # My work involves looking at telemetry data from large device fleets — BigQuery SQL against tables with millions of rows, tracking firmware rollout health, diagnosing fleet-wide issues.\nAI has made me noticeably faster here. I describe what I want in plain language — \u0026ldquo;top 10 devices by connection failure rate this week, broken down by firmware version\u0026rdquo; — and get a working query to start from. For someone who writes SQL regularly but would not call themselves a SQL expert, that is a big deal.\nIt also helps me ask better questions. When I am exploring data, it suggests angles I had not thought of. \u0026ldquo;Does this correlate with the carrier?\u0026rdquo; or \u0026ldquo;This looks time-of-day dependent, here is a query to check.\u0026rdquo; Having that kind of back-and-forth when you are trying to understand a dataset is genuinely useful.\nWhere It Falls Short # I do not want to oversell this. There are real limitations.\nHardware-specific knowledge is still thin. When I am dealing with a specific modem chipset\u0026rsquo;s AT command set or debugging a timing-sensitive interaction between two components, the AI often does not have the specific domain knowledge. It can help me organize my thinking, but the actual work still requires reading datasheets and understanding the hardware.\nReal-time and safety-critical work is where I stay conservative. Code where timing matters at the microsecond level, or where failures have physical consequences — I am much more careful about AI-generated code there. The tools are getting better, but the stakes are too high.\nBuild tooling is hit or miss. Yocto recipes, cross-compilation problems, linker scripts — the AI sometimes produces plausible-looking but wrong answers. My guess is the training data just does not have enough of these specific configurations.\nWhere This Is Headed # What I find most interesting is how unremarkable it has become. I do not think about \u0026ldquo;AI-assisted development\u0026rdquo; as a separate thing. It is just how I work. Claude Code is in one terminal pane, my editor is in another, and I move between them the same way I move between code and documentation.\nIf you work in embedded and have not tried these tools, start with log analysis. Take a debug session that would normally eat an hour and throw the logs at an AI. When it finds the root cause in a couple minutes, you will get it.\nEmbedded has always been conservative about new tools, and for good reason. Our mistakes are expensive. But AI tooling has gotten past the point of being a novelty. The developers who figure out how to use it effectively — and critically, who put in the work to connect it to their actual information sources — are going to have a real edge.\nThe tools will keep getting better. But even right now, today, they are making me faster at my job. That is not a prediction. It is just what I see every day.\n","date":"10 March 2026","externalUrl":null,"permalink":"/posts/ai-in-embedded-development/","section":"Articles","summary":"AI tools like Claude Code and Gemini have become a core part of how I write C, debug devices, and analyze fleet data. Here is what actually works in embedded development — and what surprised me.","title":"How AI Is Changing My Work as an Embedded Developer","type":"posts"},{"content":"Early in my time working on embedded cellular gateways, I inherited a firewall implementation that was a single large shell script. It worked — in the sense that it produced a functioning firewall — but it was slow, brittle, and every time we needed to change it, something broke. Eventually we rewrote the whole thing in C++. The runtime dropped by an order of magnitude, but honestly, the performance was not even the best part.\nThis is the story of that rewrite and what I learned from it.\nWhat We Started With # The existing firewall was a single shell script. Not a small one. It would start by making calls over message queues to a custom central configuration server to retrieve the device\u0026rsquo;s current settings — what interfaces were up, what services were enabled, what the user had configured through the web UI. Then, based on all that state, it would construct and execute hundreds of individual iptables calls.\nEvery rule was its own iptables -A or iptables -I command. Chain creation, policy setting, NAT rules, filter rules, forwarding rules — all individual shell commands, executed one at a time, sequentially.\nThe performance problem was straightforward. Each iptables call is a separate process invocation. Each one has to load the existing ruleset from the kernel, add or modify a rule, and write it back. When you are doing this hundreds of times in a row, you are loading and writing the entire ruleset hundreds of times. On the embedded hardware we were targeting, this added up to a noticeable delay during boot and whenever the firewall needed to be reconfigured.\nBut performance was only part of the problem.\nWhy Shell Was the Wrong Tool # The deeper issue was maintainability. The script had grown organically over time as features were added, and it was genuinely difficult to work with.\nShell scripting gives you string manipulation, conditionals, and the ability to call external programs. That is about it. When your logic is \u0026ldquo;if the device has a VPN tunnel active on interface tun0, and the user has enabled port forwarding for TCP port 443 to internal host 192.168.1.10, and the device is in router mode rather than bridge mode, then add these specific NAT and filter rules\u0026rdquo; — expressing that in shell becomes an exercise in nested if statements, string concatenation, and hoping you got the quoting right.\nError handling was basically nonexistent. If one iptables call in the middle of the script failed, the script kept going. You would end up with a partially configured firewall and no clear indication of what went wrong. Debugging meant adding echo statements and staring at log output.\nTesting was also painful. You could not unit test any of this. The only way to verify a change was to deploy it to a device and see what happened. Given that a misconfigured firewall on a cellular gateway can make the device unreachable, this was not a comfortable development cycle.\nWe kept running into bugs. Not because the developers were careless, but because the tool was fighting us. Shell is great for small automation tasks. It is not great for implementing complex, stateful logic with many interacting conditions.\nThe Rewrite # The C++ replacement took a fundamentally different approach. Instead of making hundreds of individual iptables calls, it built up the entire firewall configuration as a string — in the format that iptables-restore expects — and then fed it to iptables-restore in a single operation.\niptables-restore is designed to load a complete ruleset atomically. You give it a text file describing all your chains, policies, and rules, and it loads the whole thing into the kernel at once. One process invocation, one kernel interaction, done. Compared to hundreds of sequential iptables calls, the performance difference was dramatic.\nThe C++ application would:\nQuery the central configuration server for the current device state (same message queue interface as before, but now called from C++ rather than shell) Based on that state, construct the complete iptables-restore formatted configuration using string manipulation Feed the result to iptables-restore The string building was not particularly elegant — it was a lot of appending lines to a string buffer based on conditionals — but it was straightforward C++ that could be debugged with normal tools, stepped through in a debugger, and reasoned about using normal programming patterns.\nWhat Got Better # Performance was the most visible improvement. The firewall configuration that used to take seconds now completed in a fraction of a second. On an embedded device where boot time matters and users notice delays when they change settings through the web UI, this was significant. The exact numbers are hazy at this point, but it was roughly an order of magnitude faster.\nReliability improved immediately. With iptables-restore, the ruleset is loaded atomically. Either the whole thing loads or none of it does. No more partially configured firewalls from a script that failed halfway through. If the configuration string had a syntax error, iptables-restore would reject it entirely and the previous ruleset would remain in place. That is a much better failure mode than \u0026ldquo;half your rules are applied and good luck figuring out which ones.\u0026rdquo;\nMaintainability was the biggest long-term win. The firewall logic was now expressed in a real programming language with proper control flow, data structures, and error handling. New features that would have been a nightmare to implement in shell — things like dynamic rules that change based on VPN tunnel state, or complex port forwarding configurations — became straightforward.\nWe could also write tests. Not full integration tests against a real iptables stack, but we could test the string generation logic. Given this configuration state, does the code produce the correct iptables-restore formatted output? That kind of test was impossible with the shell script approach.\nDebugging went from \u0026ldquo;add echo statements and pray\u0026rdquo; to \u0026ldquo;set a breakpoint and look at the state.\u0026rdquo; When a customer reported a firewall issue, we could reproduce their configuration, step through the code, and see exactly what rules were being generated and why.\nWhat I Would Do Differently # If I were doing this today, I would look more seriously at using nftables instead of iptables. The nftables framework is the successor to iptables in the Linux kernel, and it has a cleaner interface, better performance characteristics, and native support for atomic rule replacement without needing a separate restore tool. At the time of the rewrite, nftables was not mature enough for our needs, but that has changed.\nI would also think harder about the configuration data model. We kept the same message queue interface to the configuration server, which meant we were still working with the same data representation as the shell script. If I were starting fresh, I would design the internal data model around the firewall\u0026rsquo;s actual needs rather than adapting to an interface that was designed for a shell script consumer.\nThe string building approach, while a big improvement over shell, is still a bit fragile. Building iptables-restore formatted text through string concatenation means typos in format strings can produce subtle bugs. A builder pattern or a proper abstraction over the rule format would have been worth the investment.\nWhen to Make This Kind of Move # Not every shell script should be rewritten in C++. That would be a waste of time in most cases. But there are signs that a shell script has outgrown its format:\nIt is making hundreds of calls to the same external tool. If your script is just a loop around an external command, you are paying process invocation overhead for no reason. Look for a batch mode (like iptables-restore) or a library interface. The logic has too many interacting conditions. When your if statements are nested four deep and depend on multiple pieces of state, you have a program, not a script. Programs deserve real programming languages. You cannot test it. If the only way to verify a change is to run it on a live system, the feedback loop is too slow and the risk is too high. You keep finding bugs in the same area. Recurring bugs are a sign that the code is too hard to reason about correctly. Better tooling — types, a debugger, structured error handling — helps. Shell is the right tool for plenty of jobs. But when you find yourself fighting it, that is not a skill issue. It is a signal that the problem has grown beyond what shell was designed to handle.\nThe firewall rewrite taught me that the best part of a performance optimization is often not the performance. It is all the other things that get better when you move to a tool that fits the problem.\n","date":"10 March 2026","externalUrl":null,"permalink":"/posts/iptables-shell-to-cpp/","section":"Articles","summary":"How a single shell script making hundreds of iptables calls became a C++ application using iptables-restore, and why the rewrite was about more than just performance.","title":"iptables to C++: When Shell Scripts Stop Scaling","type":"posts"},{"content":"","date":"10 March 2026","externalUrl":null,"permalink":"/tags/networking/","section":"Tags","summary":"","title":"Networking","type":"tags"},{"content":"","date":"10 March 2026","externalUrl":null,"permalink":"/tags/tooling/","section":"Tags","summary":"","title":"Tooling","type":"tags"},{"content":"If you have ever opened a QXDM capture from a misbehaving modem, you know the feeling. Thousands of messages scroll past. NAS, RRC, MAC, PHY — layers upon layers of protocol activity, most of it routine, all of it potentially relevant. Somewhere in there is the answer to why the device dropped its data session at 3am, but finding it is going to take a while.\nI have spent a lot of time staring at QXDM logs over the years, working on cellular telematics devices with Qualcomm modems. It is one of those skills that takes years to develop and never quite feels routine. Every capture is a puzzle, and the pieces are scattered across protocol layers with names that read like alphabet soup.\nRecently I started throwing these captures at AI tools, mostly out of curiosity. The results have been good enough that it has changed how I approach modem debugging.\nThe Problem with QXDM Analysis # For anyone unfamiliar, QXDM is Qualcomm\u0026rsquo;s diagnostic tool for capturing and viewing messages between the modem firmware and the cellular network. It logs everything — the full conversation between your device and the tower, from the initial attach procedure through bearer setup, data sessions, handovers, and disconnects.\nThe challenge is not getting the data. QXDM gives you all of it, sometimes too much. The challenge breaks down into three parts:\nSheer volume. A ten-minute capture from an active device can contain tens of thousands of messages. Most of them are routine — measurement reports, keep-alives, status updates. The interesting stuff, the messages that explain why something went wrong, might be a handful of entries buried in the middle.\nDomain knowledge. Even once you find a relevant message, understanding what it means requires fairly deep knowledge of 3GPP specs, Qualcomm-specific implementations, and how various information elements interact. What does this particular reject cause code mean? Why did the network send this configuration? Is this RRC reconfiguration normal for this carrier? These are not questions you can answer without experience or a lot of spec reading.\nCross-layer correlation. Cellular problems rarely live in a single protocol layer. A data session failure might start with an RRC reconfiguration at the radio layer, propagate through a NAS reject at the network layer, and surface as a PDN connectivity failure at the application layer. Understanding the full picture means mentally stitching together messages across layers and time — which is tedious and error-prone even when you know what you are looking for.\nWhat AI Actually Does Well Here # I want to be specific about where AI tools have helped, because the value is not evenly distributed across these problems.\nFiltering Signal from Noise # This is the biggest win. I export a QXDM capture to a text format and feed it to Claude Code. Then I say something like \u0026ldquo;find the sequence of events leading up to the data session drop at timestamp X.\u0026rdquo; The AI is very good at scanning through thousands of messages, ignoring the routine stuff, and pulling out the relevant sequence.\nWhat used to be 20 minutes of scrolling and filtering becomes a few seconds. The AI identifies the key messages, presents them in order, and usually gets the selection right. I still sanity-check it against the raw capture, but it narrows the search space to the point where I am reviewing a handful of messages instead of thousands.\nExplaining What Messages Mean # This is where the domain knowledge gap gets smaller. I can point at a NAS message and ask \u0026ldquo;what does this reject cause code mean and what typically causes it?\u0026rdquo; or look at an RRC reconfiguration and ask \u0026ldquo;is this a normal handover configuration for LTE Band 4?\u0026rdquo; and get a useful answer.\nIt is not perfect. There are Qualcomm-specific quirks and carrier-specific configurations where the AI clearly does not have enough training data. But for the standard 3GPP stuff — cause codes, message types, information elements, state transitions — it knows enough to save me from digging through specs every time.\nThe practical value here is huge for teams where not everyone has ten years of cellular experience. A less experienced engineer with an AI tool can now make meaningful progress on QXDM analysis instead of being completely blocked until a senior person is free to help.\nCall Flow Reconstruction # When I need to understand a complete attach sequence or troubleshoot a handover failure, I ask the AI to reconstruct the call flow from the capture. It pulls out the relevant messages in order, labels the direction (UE to network vs network to UE), and summarizes what is happening at each step.\nThis is work I can do myself, but it takes time. The AI does it fast and formats it in a way that is easy to follow. I then use that reconstructed flow to spot where things diverged from what I expected. \u0026ldquo;The attach procedure looks normal until step 7, where the network responded with X instead of Y\u0026rdquo; — that kind of analysis goes from a 30-minute exercise to a 2-minute conversation.\nComparing Good and Bad Captures # This is maybe the most useful thing I have found. When I have a device that works and one that does not, both on the same network, I export both QXDM captures and ask the AI to compare them. \u0026ldquo;What is different about the attach procedure between these two captures?\u0026rdquo;\nManually diffing two QXDM captures is painful. The timestamps do not align, the message ordering might differ slightly, and there is so much common noise that the real differences are hard to spot. The AI handles this well — it abstracts away the routine messages and highlights the meaningful differences. \u0026ldquo;In the working capture, the network assigned QoS profile X; in the failing capture, it assigned Y. That is your divergence point.\u0026rdquo;\nThis has cut down investigation time on some issues from hours to minutes.\nWhere It Still Struggles # Qualcomm-specific internal messages. QXDM captures include Qualcomm proprietary diagnostic messages that are not part of the 3GPP standard. The AI generally does not know what these mean, and when it tries to interpret them, it is often wrong. I have learned to explicitly tell it to skip those and focus on the standard protocol messages.\nCarrier-specific behavior. Different carriers configure their networks differently, and what is normal on one carrier might be unusual on another. The AI does not always know these differences. When I ask \u0026ldquo;is this behavior normal?\u0026rdquo; the answer I get is whether it is normal per spec, not whether it is normal for a specific carrier deployment.\nVery large captures. There is a practical limit on how much data you can feed to the AI in one session. A multi-hour QXDM capture exported to text can be enormous. I usually have to pre-filter or split the capture before the AI can process it, which means I need to have some idea of what time range to look at. This is not a deal-breaker, but it means the AI is not quite a \u0026ldquo;just throw the whole thing at it and get answers\u0026rdquo; solution for very long captures.\nSubtle timing issues. When a problem is related to timing — a message arrived too late, a timer expired, a race condition between layers — the AI sometimes misses it. It is good at the logical sequence of events but less good at spotting when the timing of those events is the problem.\nHow I Actually Use It # My workflow for a QXDM analysis session now looks like this:\nReproduce the issue or get a capture from the field Export the relevant time window to text Feed it to Claude Code with context about what the problem is Ask for the event sequence leading up to the failure Follow up with specific questions about messages or behaviors I do not understand If I have a working device for comparison, feed both captures and ask for differences Steps 4 through 6 are iterative. I go back and forth, asking follow-up questions, requesting deeper analysis of specific messages, and occasionally correcting the AI when it misinterprets something. The whole thing is a conversation, not a one-shot query.\nThe result is that I get to the root cause faster, I learn things about the protocol stack along the way, and I can communicate findings more clearly to the team because the AI helps me articulate what happened in plain language.\nThe Bigger Point # QXDM analysis is one of those skills that has traditionally been a bottleneck. Only a few people on any team really know how to do it well, and those people are always overloaded. AI does not eliminate the need for expertise — you still need to know what questions to ask and when the AI is wrong. But it brings the floor up. It lets less experienced engineers make progress on problems that used to require senior attention.\nFor me personally, it has made cellular debugging less of a grind. The tedious parts — the scrolling, the filtering, the spec lookups — are the parts the AI handles best. That leaves me more time for the parts that actually require thinking.\nIf you work with QXDM captures and have not tried feeding them to an AI, take a capture you have already analyzed manually and see if the AI reaches the same conclusion. You might be surprised how well it does.\n","date":"10 March 2026","externalUrl":null,"permalink":"/posts/ai-qxdm-analysis/","section":"Articles","summary":"QXDM captures contain everything you need to diagnose cellular issues — buried under thousands of messages you do not care about. AI changes the math on how long it takes to find what matters.","title":"Using AI to Make Sense of QXDM Captures","type":"posts"},{"content":"","date":"10 March 2026","externalUrl":null,"permalink":"/tags/vpn/","section":"Tags","summary":"","title":"Vpn","type":"tags"},{"content":"I spent several years building and maintaining a StrongSwan-based VPN solution on embedded cellular gateways. It worked. Customers used it. But it was a constant source of complexity — in the codebase, in debugging, in customer support calls, and in the sheer amount of configuration needed to get a tunnel up and running.\nWhen WireGuard came along, I was skeptical in the way you get skeptical after years of dealing with VPN implementations. \u0026ldquo;Simple and fast\u0026rdquo; is what every VPN claims to be. But after working with WireGuard on embedded devices, I am convinced it is genuinely different, and it is particularly well-suited for the constraints embedded systems deal with.\nThe IPsec Reality on Embedded # To understand why WireGuard matters for embedded, it helps to understand what IPsec actually looks like when you deploy it on a small device.\nStrongSwan is a solid IPsec implementation. It is well-maintained, standards-compliant, and supports a huge range of configurations. That last part is both its strength and its problem on embedded devices.\nAn IPsec VPN involves IKE negotiation (with its own state machine and multiple exchanges), Security Associations, SPIs, rekeying, dead peer detection, NAT traversal, and a dizzying number of cipher suite combinations. The configuration for even a basic site-to-site tunnel involves specifying connection parameters, authentication methods, traffic selectors, lifetime values, and DPD settings. On the embedded gateway I worked on, we had a C++ application that interfaced with StrongSwan through the Davici API, translating user-friendly web UI settings into the correct StrongSwan configuration. That translation layer alone was a meaningful piece of code to maintain.\nDebugging was painful. When a tunnel would not come up — which happened regularly due to mismatched configurations between endpoints — you were looking at IKE logs full of proposal negotiations, SA state transitions, and retransmissions. Figuring out whether the problem was an authentication issue, a proposal mismatch, or a NAT traversal failure required experience and patience.\nThen there were the edge cases. Rekeying under packet loss. DPD timers interacting poorly with cellular networks that have variable latency. NAT devices in the path that handle ESP packets differently. Each of these generated support cases and required careful handling in code.\nNone of this means StrongSwan is bad. It implements a complex protocol correctly. The problem is that the protocol itself carries a lot of weight, and on an embedded device with limited resources and a need for reliability, that weight matters.\nWhat Makes WireGuard Different # WireGuard takes a fundamentally different approach, and almost every design decision turns out to be the right one for embedded.\n~4,000 Lines of Kernel Code # The entire WireGuard implementation is roughly 4,000 lines of code in the Linux kernel. For context, the IPsec stack in the kernel — not counting userspace tools like StrongSwan — is an order of magnitude larger.\nOn an embedded device, this matters for several reasons. A smaller codebase means fewer bugs, a smaller attack surface, and easier auditing. When you are shipping firmware to devices that might be deployed for years in hard-to-reach locations, the probability of security vulnerabilities in your VPN stack is directly related to the amount of code in that stack.\nIt also means the kernel module is small enough to actually read and understand. I have read the WireGuard source. I have not read the IPsec kernel implementation. One of those is feasible for an embedded developer who wants to understand what their device is doing. The other is not.\nConfiguration That Fits on a Napkin # A WireGuard configuration for a site-to-site tunnel is a few lines:\n[Interface] PrivateKey = \u0026lt;key\u0026gt; Address = 10.0.0.2/24 [Peer] PublicKey = \u0026lt;key\u0026gt; Endpoint = 203.0.113.1:51820 AllowedIPs = 10.0.0.0/24 PersistentKeepalive = 25 That is the whole thing. There is no proposal negotiation because WireGuard uses a fixed cryptographic construction (Noise protocol framework with Curve25519, ChaCha20, Poly1305, BLAKE2s). There is no cipher suite selection because there are no options to select. There are no lifetime values to tune because WireGuard handles rekeying internally on a fixed schedule.\nFor embedded devices, this simplicity translates directly into reliability. There is no configuration mismatch to debug because there is barely any configuration to mismatch. The most common IPsec support issue — \u0026ldquo;why won\u0026rsquo;t my tunnel come up\u0026rdquo; caused by proposal or authentication disagreements — essentially does not exist with WireGuard.\nPerformance on Constrained Hardware # WireGuard runs in kernel space and uses modern cryptographic primitives that are fast on the kinds of ARM processors found in embedded devices. ChaCha20 in particular was designed to perform well on processors without hardware AES acceleration, which is common on lower-cost embedded hardware.\nIn practice, the throughput and latency differences between WireGuard and IPsec on embedded hardware are noticeable. The handshake is a single round trip (1-RTT), compared to IKEv2 which requires multiple exchanges. On a cellular connection with 50-100ms latency, the difference between a 1-RTT handshake and a multi-round IKE negotiation is significant, especially when roaming or reconnecting after a network change.\nRoaming Just Works # This is the feature that matters most on cellular-connected embedded devices. WireGuard is designed to handle endpoint changes gracefully. If a device moves between cell towers, gets a new IP address, or switches between network interfaces, WireGuard handles it transparently. The tunnel stays up because WireGuard associates peers with their cryptographic identity, not their network address.\nWith IPsec, a source IP change typically means renegotiating the SA. Depending on DPD timers and configuration, this can take seconds to minutes. On a cellular gateway that might change IP addresses regularly, this creates gaps in connectivity that can be a real problem for applications relying on the tunnel.\nWireGuard just updates the endpoint when it sees a valid packet from a known peer at a new address. No renegotiation, no delay, no dropped traffic beyond what the network change itself causes.\nWhat You Give Up # WireGuard is not a drop-in replacement for IPsec in every scenario. There are trade-offs.\nNo standards compliance. IPsec is an IETF standard. WireGuard is not. If your customers or deployment requires compliance with specific standards or interoperability with third-party IPsec devices, WireGuard is not an option.\nNo certificate-based authentication. WireGuard uses static public keys. There is no PKI, no certificate chains, no revocation. For large deployments, managing and distributing keys requires building your own infrastructure around it.\nLimited configurability. The lack of options is usually a feature, but sometimes you genuinely need to control cipher selection, lifetime parameters, or traffic selectors. WireGuard does not give you those knobs.\nFewer enterprise integrations. IPsec works with RADIUS, EAP, and various enterprise authentication backends. WireGuard has none of that built in.\nFor embedded IoT and gateway devices connecting back to your own infrastructure, these trade-offs are usually acceptable. You control both ends, you do not need standards compliance for interop, and you can manage keys through your device provisioning system. But for products that need to connect to arbitrary customer VPN concentrators, IPsec is still probably necessary.\nThe Embedded Sweet Spot # Where WireGuard really shines is the pattern that is increasingly common in IoT and telematics: a fleet of devices in the field that need a secure tunnel back to a central server or cloud endpoint.\nIn this scenario, you control both sides. You can bake the WireGuard configuration into the firmware. Key distribution happens during device provisioning. The central endpoint runs a WireGuard server that accepts connections from the fleet. Every device has a persistent, low-overhead, encrypted tunnel that survives network changes and reconnects instantly.\nCompare this to deploying IPsec across a fleet: configuring StrongSwan on every device, managing certificates or PSKs, tuning DPD timers for cellular networks, handling the inevitable support cases from tunnel establishment failures. The operational difference is significant.\nLooking Forward # WireGuard has been in the Linux kernel mainline since 5.6. It is no longer experimental or niche. For new embedded Linux projects that need VPN connectivity, I think WireGuard should be the default choice unless you have a specific reason to need IPsec.\nHaving worked with both extensively, the difference in operational complexity alone is worth the switch. Add in the performance improvements, the roaming behavior, and the reduced attack surface, and it is hard to make a case for IPsec in scenarios where WireGuard is applicable.\nThe best VPN for an embedded device is the one that works reliably, requires minimal configuration, and stays out of your way. After years of fighting with IPsec to achieve those goals, WireGuard just does it.\n","date":"10 March 2026","externalUrl":null,"permalink":"/posts/wireguard-for-embedded/","section":"Articles","summary":"After years of wrestling with IPsec and StrongSwan on embedded gateways, WireGuard feels like the VPN that was designed for constrained devices. Here is why.","title":"Why WireGuard Is the VPN for Embedded Devices","type":"posts"},{"content":" Who I Am # I am a software engineer with a decade of experience in embedded Linux systems, specializing in cellular connectivity, networking, and low-level C development. I enjoy working on problems where software meets hardware and where reliability matters.\nExperience # Senior Software Development Engineer — Geotab, Oakville, ON (2024–Present)\nDeveloping firmware in C for the Trinity ARM64 embedded telematics platform running Linux Owned the cellular connectivity stack, including custom ModemManager plugin development for Sequans modems, connection stability improvements, and signal metrics reporting Led the implementation of the \u0026ldquo;Last Gasp\u0026rdquo; power loss handling system — detecting switchover to internal battery, executing safe shutdown sequences, and capturing final GPS position before power loss Built D-Bus interfaces between the Go application layer and the Connection Manager daemon, exposing WiFi configuration and ignition state across service boundaries Developed geographic enforcement features for location-based device behavior Performed field troubleshooting and fleet-wide debugging using BigQuery data analysis across deployed device populations Senior Software Development Engineer — AMD, Markham, ON (Jun. 2022–2024)\nDeveloped features and addressed bugs for Windows laptop display drivers using C and C++ Took the lead in enabling hardware features in driver focused on reducing power consumption and assisted team responsible for collecting power usage metrics Made several contributions to the open source AMD display driver in the Linux kernel to enable feature parity between Linux and Windows drivers Staff Firmware Engineer — Sierra Wireless (Acquired by Semtech), Richmond, B.C. (Dec. 2021–May 2022) Senior Firmware Engineer — (Oct. 2019–Nov. 2021) Junior Firmware Engineer — (Sept. 2017–Sept. 2019)\nDesigned C++ Linux applications for communicating with cellular LTE radios to establish and maintain data sessions Rewrote iptables firewall configuration from a collection of shell scripts into a faster and more maintainable C++ application, bringing down runtime by an order of magnitude Developed and maintained a StrongSwan VPN solution consisting of a C++ application that leveraged the Davici API with custom firewall and routing rules Wrote a C++ application that can be sideloaded onto a cellular gateway to report customizable telemetry data to a user\u0026rsquo;s Microsoft Azure IoT Hub Took responsibility for addressing CVEs reported against 3rd party applications and libraries included with firmware package as well as remediating CWEs in team\u0026rsquo;s source code Created a Co-op hiring and training program and directly supervised and mentored co-op students Drove the creation of an Emerging Professionals Employee Resource Group and took on the role of first global lead Firmware Developer Co-op — Sierra Wireless, Richmond, B.C. (Jan.–Dec. 2016)\nDeveloped policy routing and Ethernet WAN features by interfacing application code with hooks from the Linux kernel Completed significant refactoring of radio module interface, structuring code using object-oriented principles Implemented Yocto based workflow for generation of manufacturing/recovery image and toolchain creation Embedded Software Developer Co-op — Avigilon (Acquired by Motorola Solutions), Vancouver, B.C. (Jan.–Aug. 2015)\nUsed Yocto to create OS image for use in OpenStack virtualized environment along with automated nightly build process Designed and implemented visual media transcoding task for use in asynchronous execution pipeline Developed platform scripts to initialize embedded Linux camera environment Education # Bachelor of Applied Science — Simon Fraser University, Burnaby, B.C. (2017) Concentration: Electronics Engineering\nCisco Certified Entry Networking Technician (CCENT) — BCIT, Burnaby, B.C. (2019)\nSkills # Languages: Bash, C, C++, Go, Lua, Python, SQL\nApplications: Atlassian tools, BigQuery, CMake, Docker, Git, iptables, Microsoft Azure, StrongSwan, Subversion, Wireshark, Yocto\nTechnologies: D-Bus, DHCP, DisplayPort, DNS, HDMI, IPsec, LTE, ModemManager, NetworkManager, NMEA, RS232, SNMP, TCP/IP, UDP, WireGuard\nContact # Feel free to reach out at daniel@miess.ca.\n","externalUrl":null,"permalink":"/about/","section":"Daniel Miess","summary":"About Daniel Miess","title":"About","type":"page"}]