Streaming Real-Time Market Data into Excel: Why RTD Beats an API for Analysts

An analyst doesn’t so much use Excel as live in it.  The spreadsheet is where the models, the risk, and the intuition already sit.  So when “get live market data into your model” turns into “learn our REST API, manage an auth token, write a polling loop, and reshape JSON,” the work has quietly moved out of the environment where you’re fluent and into one where you aren’t.

I ran a live trading dashboard in Excel for years, and before RTD it was a Rube Goldberg machine.  Positions and account values came in over DDE, which required (1) a separate “feeder” Excel process with its own workbook just to receive the DDE, and (2) a Java socket bridge connecting the DDE to the broker’s trading platform.  Real-time market data came through yet another Excel add-in.  I got the polling cycle from feeder to dashboard running about as well as it could run.  It would still stumble or break, and the poll-and-populate cycle was most disruptive when I could least afford it: during peak market periods, like the minutes just before the close.  Three processes and a Java bridge, just to keep one workbook current.

There’s an older, better-fit answer hiding in plain sight: RTD (Real-Time Data), a Microsoft protocol that has been sitting in Excel since 2002.  Twenty-four years!  Unlike a =WEBSERVICE() call you have to nudge to recalculate, an RTD server hands Excel updates as fast as they change and as fast as Excel can receive them.  The implementation matches the mental model: a cell is a subscription, not a snapshot.  The spreadsheet stays live without the analyst writing anything more than a native Excel formula.

What does polling actually cost?  A timer-driven refresh refetches mostly unchanged data, so it’s expensive, and it still lags the moves you care about, so it’s slow.  RTD only updates what changes when it changes.  But the bigger difference is one you feel rather than measure: hardened RTD just works – transparently, reliably.  You never hesitate before editing a cell, wondering whether this keystroke will upset the whole contraption and freeze the workbook, flip it into manual-calculation mode, or silently wedge the feeder.  I never tracked latency statistics – I don’t do ultra-high-frequency work anymore – and the old system did run at the refresh rates we needed.  It was just heavy and fragile, and the replacement isn’t.

There’s also a hidden tax in the API path that rarely gets billed honestly: reshaping.  JSON is a tree and a spreadsheet is a grid, and every analyst who consumes a web API ends up owning that transform.  RTD lands data already cell-shaped.

The general principle shows up in everything I build: meet people in the environment where they’re already fluent, and tuck the plumbing behind a wall they never have to think about breaking open.  The analyst’s environment is a grid of live cells.  Give them realtime data, not snapshots.

Coding Agents Grow Up

For years, programming something new followed a predictable, exhausting rhythm: write some code, hit a wall, and then disappear into a forest of documentation and StackOverflow tabs to find the trick to get it working. In 2025, that era ended for me.

Today, AI code assistants take so much drudgery out of development and debugging that the work has become mostly gratifying and rarely frustrating — the opposite of how programming was in the Before Times.

Between my work and my side interests, I often feel like I live in Visual Studio Code (VSCode — a popular open-source development environment). Early in 2025 I subscribed to GitHub Copilot, which integrates AI coding assistants into VSCode. At $10/month it’s a phenomenal bargain that offers software developers an easy way to loop in the latest models from OpenAI, Google, and Anthropic. Now when I’m trying something new (like this little project I did for fun) I can mostly stay in VSCode and work with an AI assistant that has mastered all of the documentation.

As mentioned elsewhere, my most significant side project in 2025 was helping Ukrainians develop an open-source ballistic calculator (pyballistic) in Python and Cython. I polished that off at the end of September.  By that point I had begun to spend more time with Github Copilot, and the capabilities of its latest models gave me enough confidence and support to tackle what would have previously been an absurdly ambitious project for my day job: an Excel Real-Time Data (RTD) server for the Interactive Brokers API.  (This RTD server feeds live market data, positions, and orders directly into Excel using native Excel formulas.) After two months of working seven days a week on that, I had a beautiful piece of software so solid (and validated every build by over 800 unit tests) that I had begun to use it in live trading operations.

Early Childhood Development

Watching these models mature over the last year has been like watching a child grow up.

Tell a child, “Clean your room.” First they’ll spend more time arguing than it would take to just do it. When they finally declare the task “done,” you might find a few toys picked up but most of the mess still there. Emphasize that “clean your room” means everything and you might find the floor clean but everything shoved under the bed.

Claude v3 was notorious for hacking shortcuts. Ask it to fix a failing test and it might just replace the test logic with a “return true;” statement. Claude v3.5 wouldn’t be so brazen, but it was still prone to hack the example rather than the task. GPT-4 and Gemini v2 would enthusiastically announce completion without checking their work. Like the child who picks up two toys and concludes that his room must be clean, even though the mess is visible from outside the door.

The teens came quickly: Claude v3.7 and its contemporaries would often spend more effort arguing that its failure was actually success than it would have taken to do the work correctly.

More recent models have become more likely to keep checking and working until they succeed. Performance of the latest models is still wildly variable: a model that astonishes me with its apparent skill one day may choke on something relatively simple the next. But they are getting more consistent. And they are definitely getting more intelligent.

What is intelligence?  It becomes easy to see when you’re doing hard work with different models. One of the neat things about Copilot is that you can choose to watch the model at work. They all think “out loud,” meaning you can read their chain of thought to understand how and why they do things. When it’s not having an off day, Claude Opus is intelligent.  Given a problem:

  • It can more reliably identify what matters.
  • It has a better sense of what to look at and what to ignore.
  • It produces better assessments of what’s possible and makes better plans to get there.
  • It knows when to persist and when to change directions.

These are some of the things that separate a junior developer from a more experienced one. They are also qualities that characterize more intelligent people.

Let me show you. Have you ever wondered what it’s like debugging software? Well, debugging is one thing the newer models can usually do as well as a good human programmer. In fact, they can do it better because they can run the process faster and interact with the code more directly. Below I pasted a transcript of Claude working to find and fix a tricky bug in my RTD server. This could just as well have been a transcript of my thoughts if I had to debug it. But whereas this would have been a draining hour+ distraction for me, the Claude instance cranked this out in minutes.

Continue reading “Coding Agents Grow Up”

2025: The End of the Human Polymath

Born in 1976, I was just early enough to taste pre-internet life. In grade school, on a dial-up 1200baud modem and IBM PC, I was ahead of the curve in actually connecting to primordial pieces of the internet, though they didn’t have a lot of utility outside of academic collaboration. I grew up with a physical encyclopedia at home – the 22 volumes of the 1987 World Book Encyclopedia took up more than 3 feet of shelf space. I wondered how they decided what to include in those books, because I was mostly frustrated to find my subjects of interest barely grazed, if covered at all. In the mid-1990s Microsoft published a DVD to make the printed encyclopedia obsolete: Encarta, which somehow offered even less information but more data because it had “multimedia” – the buzzword for sound, video, and primitively interactive content.1

So what did a curious young mind do back then? There were so many more questions than answers. For a typical How or Why question, your local library might have a book containing an answer, but you’d have to physically visit the library, search their card catalog for books covering the subject, and then physically find potential matches on the shelves and thumb through each to see if it actually provided the details sought. You couldn’t reach out to experts because even if you could find their names you couldn’t easily find contact information. So you were stuck with whatever local adults happened to know. My Dad was very smart, and he had smart work colleagues who could go quite far in some areas of math and physics. What about teachers? Public school teachers were – despite their avowed profession – astonishingly underinformed. (That realization led me to despise them: I did not get along with public school teachers after 3rd or 4th grade when I discovered that, to any random question, I was more likely to have the right answer then they.)

It was a struggle to build a deeper-than-average understanding of the world. I put in a lot of work seeking answers to practical questions, and that gave me a lot of practical knowledge. I certainly had gaps: Pop culture, sports – many obsessions of the average person did not interest me, so I was never going to be a Jeopardy champion. But in the realm of practical and technical knowledge I was exceptional. I read slowly, but have an insatiable thirst for understanding how and why things work. Plenty of people idly wonder. I don’t just wonder: I search. When I had a random question and couldn’t quickly find the answer I would write it down, and eventually I would find an answer and absorb everything around it. Maybe the hunt is why the answers stick in my head.

Now, what search engines started, LLMs have so thoroughly finished that future generations are bound to forget that there was a time when knowing things was not only difficult but also useful.

“Know-it-all” was often thrown around as a pejorative. But, back in the dark ages of the late 20th century, extensive practical knowledge had real utility. It could make the difference between staring blankly at a problem (or not even recognizing the presence of a solvable problem) and jump-starting solutions by drawing on a deep well of understanding how other things work and how they could relate. A know-it-all2 is more likely to:

  • Recognize the absence or presence of a significant problem. (What is that sound, and should I get it looked at?)
  • Flag misleading or false assertions. (Could competitive chess players really burn thousands of calories thinking during a match?)
  • Explain what matters, when, and why. (When do you really need to change engine oil, and should you pay extra for synthetic?)

Even when search engines came along, the human polymath still had value. Answers were more accessible, but you still had to know the right questions. You had to know if a thing was a thing, what terms might apply, and what a correct answer should look like.

Today, it’s over. We have reached the singularity of convenience. This year, as they ironed out the chatbot propensity to hallucinate, the value of the human know-it-all evaporated. Yes, I still catch the bots making factual errors, but if you keep them talking they eventually notice the errors themselves.

I took pride in being the guy to ask, the guy with the notoriously uncanny breadth and depth of knowledge, the guy who – even if he didn’t have the answer off the top of his head – would likely find it faster than anyone else. “Have a practical question? Just ask me. Worst case: I don’t know. More likely: I’ll point you in the right direction.” Now? I tell people to ask the bots. There is no way I can give as quick and thorough an answer on as broad a set of topics as they can.


  1. What was I looking for? Something like a cross between Wikipedia and The Way Things Work. Here’s how I described it in a 1998 journal entry: The Practical Encyclopedia of Technology.  It would contain in applicable form all of mankind’s technological achievements—information I haven’t been able to find elsewhere, like how transmission mechanisms are actually implemented on vehicles, the composition and construction of TFTs, how ball bearings are manufactured.  Every article on a specific piece of technology would be of the following form:
    – Brief theory;
    – References to components (e.g., transmissions would reference ball bearings, metal casting, gears, lubricants);
    – Problems encountered in implementation;
    – Canonical solutions to problems, in sufficient detail to actually implement on that information alone;
    – Other solutions that have been tried, and why they haven’t caught on;
    – References to sources for theory on the subject;
    – Patent Office classification fields of the technology, etc.
    ↩︎
  2. The age of the literal know-it-all – someone who knows everything that is known in a society – ended centuries ago. At least in the developed Western world, that has been impossible since the early 1800s. The title may be hyperbole, but The Last Man Who Knew Everything describes a plausible contender for the title: Thomas Young, who died in 1829. ↩︎

Adobe’s Protection Racket

I just burned more than a day migrating my primary work computer from a machine running Windows 10 to a newer one running Windows 11. Not because I wanted to. Not because Win11 offers me anything I actually want (so far I hate every UI change from Win10). But because Microsoft has decided to end support for Win10 while preventing Win11 from running on older CPUs. And like everyone else whose work requires a secure operating system I’m being shoved along whether I like it or not.

This isn’t a trivial inconvenience. Over the last decade I’ve accumulated a small arsenal of development tools, libraries, and utilities — each with its own quirks, dependencies, and fragile installation paths. Migrating them is not a matter of clicking “Next” on a wizard. It’s a slog of registry tweaks, PATH surgery, license re‑entries, and the occasional ritual sacrifice to the gods of backward compatibility.

And just when I thought I had wrestled Windows 11 into grudging submission, Adobe decided to remind me that they can be even worse.


Adobe’s Perpetual License That Isn’t

I own a perpetual license for Lightroom 6. “Perpetual” is supposed to mean I can use it forever. The software runs fine on Windows 11 … except that Adobe has disabled it.

Adobe included one of the tedious “activation” processes in the Lightroom installation process that depends on their servers telling the software that my license is legitimate. And they have quietly shut down their activation servers, so now when I launch Lightroom 6 in Win11 I have discovered an endless loop of signing in, accepting the license agreement, and then having the software crash. To add insult to injury: Adobe makes no note on their website’s activation page that this process has been disabled for Lightroom 6. I only learned that it would not work after trying repeatedly and then asking Copilot what was happening.

This isn’t a bug. It’s a business model. Adobe has effectively disabled software that would otherwise continue to work. They’ve taken something I paid for outright and retroactively converted it into a hostage situation: either I cough up for their recurring subscription, or I lose access to the tools I already bought and the work I invested in using them to catalog and post-process more than 60,000 photos.

That’s not “end of support.” That’s a protection racket.


Why This Matters

This isn’t just about photography software. It’s about the erosion of implied contracts. We’re told we’re buying licenses, but too late discovering that those licenses can be revoked, crippled, or held hostage at the whim of the vendor. The “perpetual” in perpetual license turns out to mean “until we decide otherwise.”

For engineers, photographers, musicians — anyone who performs their work in specific software — this can be catastrophic.


Imagine you buy a plot of land from a real estate developer and build a house on it. Then one day you come home to find a gaping hole where your house used to sit. Eventually you find the developer and get the following explanation:

Sorry for the confusion: You bought the land, not the location. We moved your house and the land (i.e., the dirt) under its foundation to a new location.

Oh, and that new location is only available for rent. The monthly price? Well, if you have to ask, you’re not going to like it….

Light Interaction App

Check out this nifty little touch-screen-compatible, WebGL-powered application.

To test out the latest AI, I added GitHub Copilot to VSCode and asked it to build a simple web application that lets the user move three radiant lights (red, green, and blue) around a screen to see how adding colors works. (For example, if the three colors are right on top of each other it looks like a single white light.) Here’s a screenshot of that first app:

By default Copilot uses GPT-4o, but on a few examples I have found that Claude 3.7 Sonnet (another Copilot option) is capable of more sophisticated computer engineering, so with that selected as my Copilot “Agent” I began enhancing this app. The most significant change – and something I’ve wanted to try for a while – was to use WebGL to take advantage of the graphics processing features built into most modern electronics. Thanks to that hardware acceleration, this enhanced app supports lots of light sources, dithering to avoid color banding, and real-time dragging lights around the screen without noticeable lag. Then I added touch-screen support so that the app can be used on mobile devices.

It took some coaching from me to get this working: At several points I observed bugs and Copilot would essentially get stuck in a loop saying, “Oh, I see what’s wrong; this should fix it,” without successfully fixing it. I had to guide the Agent through more intentional debugging methods to resolve several confusing problems. But by the end I hadn’t written or even touched much of the code. I was the designer and tester, and Copilot saved me the trouble of:

  • Scouring API documentation and sites like StackOverflow for code samples needed to make it work.
  • Learning or remembering the exact syntax of the languages involved (WebGL, JavaScript, CSS, HTML).
  • Recreating common GUI tricks, like adding code to make sure that everything is visible on a screen regardless of its size or orientation.
  • Finding and fixing minor bugs.
  • Writing debug code to understand and resolve major problems.

Here’s a screenshot from the final app (shown here with all light inverted – one of the fun features accessible by right-clicking/long-tapping):

Alcohol is more efficient than sugar

Centuries ago, when bulk transportation was expensive and raw foods were at risk of loss to spoilage or pests, distant farmers might prefer to convert sugar-bearing grain and fruit into ethanol, distill it to concentrate it, and transport that concentrate to distant markets. (This fact was a primary cause of the Whiskey Rebellion in response to the imposition of a tax on distilled alcohol.)

I was surprised at how much more efficient ethanol is as a source of energy, even though energy is lost through fermentation! The chemical equation for fermentation of sugar (glucose chosen here) is:

C₆H₁₂O₆ (glucose) → 2C₂H₅OH (ethanol) + 2CO₂ (carbon dioxide) + Energy

Considered by weight: For every kilo of glucose fermented, we get half a kilo of ethanol and 300 calories of heat (which is typically wasted). But ethanol is more energy dense than sugar: Humans extract 3700 calories per kilo of glucose but 7000 calories per kilo of ethanol! I.e., the metabolic energy available in a kilo of sugar is mostly preserved when converted to alcohol, but its weight is cut in half.

Today’s LLM Challenge: Create a diagram

Ask an LLM to create a technical diagram and they’ll use a diffusion generator, which produces funny but mostly nonsensical results. But they tend to be pretty good at writing code, so I thought I’d see how they can do with the following prompt:

Describe a diagram suitable for a textbook that illustrates the definition of “cross-sectional area”. Then create SVG code to render that.

The description is easy, so I’m only going to show the SVG images they produced.


ChatGPT 4o

Claude 3.5 Sonnet


Perplexity

Bing

Bing also volunteered four diffusion images, first of trees like this:

Then I asked it to try again using a simple cylinder and got these cool but not usable results:


The smaller models available through HuggingChat were worse, and Cohere didn’t even get the SVG namespace into the XML. Here’s what I got from the top three:

Privacy.com to thwart subscriptions

Businesses love the recurring revenue from subscriptions. And they love the fact that people tend to be so lazy and forgetful that customers don’t reliably cancel subscriptions that they wouldn’t otherwise keep. As a conscientious consumer, I dislike subscription services because businesses have a disincentive to make cancelling subscriptions easy. I disapprove of the roach motel business model. My favorite countermeasure: Privacy.com.

Privacy.com lets you create credit cards with all sorts of constraints, as shown in this screenshot:

Screenshot of Privacy.com credit card limits.

I particularly like the single-use card, which I create for subscription services. If I forget about the service they can’t keep sucking money from my bank. If I want to cancel the service, I don’t have to find and navigate their arbitrary cancellation processes … and keep records of cancellation attempts to dispute recurring charges from those businesses that are either incompetent or downright fraudulent when it comes to cancelling subscriptions. If I do want to continue subscribing, I have found that every business goes above and beyond to make that as easy as possible!

Inkscape vs Adobe Illustrator for Vector Graphics

I’m about to break 100 figures (that’s photos, diagrams, and combinations thereof) for my book, which in current draft is 150 pages. Whenever possible I create vector images because they scale perfectly. The alternative is a raster (pixel-based) image, which is defined for some number of pixels and becomes pixelated when enlarged. Here’s an extreme example: I drew a simple circle, rasterized a copy of it, and then zoomed in on the top section:

The vectorized version stays smooth at any scale, because the rendering engine is essentially told, “The drawing is a black circle, this size with this line thickness.” (The SVG code is actually <circle cx="50" cy="50" r="40" stroke="black" stroke-width="1" />.) The rasterized version is a fixed rendering of that circle using a specified number of pixels, and that’s as detailed as the image can get.

Raster images are natural for things like photos that begin life as pixel arrays. But computer drawings that are built up using primitive shapes and strokes almost always benefit by preserving that construction. A bonus is that vector files tend to be much smaller than legible raster versions – even in the human-readable Scalable Vector Graphics (SVG) format.

I have created vector images directly in Word (more on this later), Excel, and Python (using matplotlib and seaborn). For just $5 I got a Ukrainian freelancer on Fiverr to vectorize a diagram from an old military publication.

Until recently I used Adobe Illustrator to do more serious vector graphics. But my latest copy of that software is from 2007 and doesn’t always work well in Windows 10. (Yes, I got it back when you could buy and keep using an application instead of having to subscribe.) If I did this stuff all the time I would probably subscribe to Adobe Creative Cloud. But I don’t, so I took a look around and found a popular open-source alternative to Illustrator: Inkscape. It took a day to get comfortable with it, and now it’s great for my purposes. For example, I liked this drawing from a 2004 dissertation by Jorma Jussila, and he was kind enough to send me the vector image, which gave me a tremendous head start creating this diagram:

How Microsoft Let Me Down This Week

In college (1998) I wrote a graphics-heavy term paper in Word. I had invested days of work, saving frequently (as always). At some point Word corrupted the document, but I didn’t realize that because I had kept it open on my computer and the in-process version didn’t give a hint that anything was wrong. Until I tried sending it to a printer, at which point it crashed and I discovered that none of the files contained much that could be salvaged. It took a frantic day to reproduce the paper.

Well, that summer I had a great internship at Microsoft, and – not one to hold a grudge – I have been drafting my current book in Word. (As of today the DOCX is 100MB!) But I have been frequently creating and checking PDFs along the way. In the course of these reviews I discovered that Word is unpredictable in creating and handling vector drawing objects. For example, adding a bevel to a circle causes Word to rasterize it, but you probably won’t notice unless you make a point of inspecting things at high zoom multiples.

Create a circle in Word and it’s a vector drawing (left). Add a bevel and Word silently rasterizes it (right).

I also discovered that Microsoft’s PDF printers and export engines rasterize everything, which is not only a loss of detail, but also a travesty given that vectors are a foundation of the Portable Document Format. Fortunately there are free PDF printers (I verified both doPDF or CutePDF) that preserve vectors.

Creating Music with FL Studio

One of my boys wanted to play with a Digital Audio Workstation (DAW) called FL Studio, so I bought a license. I figured if he was interested I might as well play with it too, and I was immediately overwhelmed. It’s a software system used by professional music producers.

I didn’t expect a steep learning curve. After all, I began playing with a keyboard synthesizer and MIDI in the 1980s. (Some of the stuff I did is still floating around my old website.) More recently I spent time transcribing and arranging music with MuseScore. But the sound production community has a separate lineage going back to analog media and hardware – think of those huge mixing boards in sound studios – with its own practices and terminology that have carried forward to the latest DAWs.

I persisted. I watched a lot of tutorials on YouTube channel In the Mix. I’ve spent on the order of a hundred hours playing with this thing. And my bewilderment at the depth and breadth of the state of the art of sound production has only grown.

The best way to learn anything is to have a test project. I had just finished watching the second season of White Lotus, which has an amusing opening theme song. So I set out to reproduce and elaborate on that. My first cut did not stray far from its inspiration:

Over the next few months I put in more hours playing with different ideas and ended up with this:

Here’s a screenshot of the project in FL Studio:

Screenshot of David Bookstaber's first project in FL Studio.

I think the results are OK, but only until I listen to something that has been done by a professional. Leaving aside the fact that I only used FL Studio’s basic synthesizer for the instruments, what I have now is relatively muddy, lacking the crispness and separation of sound that I wanted. I have just scratched the surface of a few of the dozens of standard tools and techniques used by professionals, which I didn’t even know existed until I started this project.