Finance - David Bookstaber

Software Engineering

Posted on August 11, 2026August 11, 2026 by David Bookstaber

First published in my newsletter, September 29, 2023.

I’m writing a book on ballistics, based in part on my work on ballistipedia.com, and I got to the point that I wanted a raw ballistic calculator to create examples. I found a public implementation of a well known calculator in (programming language) Python and decided to start with that.

Nothing gets you up to speed on software engineering tools like having (a) a project you want to complete and (b) existing code to work from. In this case the existing code used a lot of Cython, which is a hybrid of Python and C that I have been meaning to pick up for a long time. I have been using Python more than any other language for the last six years. But I began coding in C more than thirty years ago. So why did I end up using Python if C is still relevant?

Some features of Python:

It’s a high-level language that encourages concise and readable code.
It’s the most popular programming language. Whatever you want to do, you can probably find existing (and free) Python packages that get you most of the way there. For example: I recently needed to scrape a bunch of data from a website. Bing’s chatbot recommended a package called selenium, and in less than an hour and 30 lines of Python I was done.
It is interpreted, not compiled. This makes it fast to write and easy to prototype in, but it also means the code runs relatively slowly.

In the past decade computers have gotten so fast that the last point is usually not an issue. Lots of production systems are running in Python. But when you realize how much slower Python runs this is surprising: As an example, I just made a Cython version of a simple statistical simulation I had written in Python. The Cython version can run 10 million simulations in 15 seconds. In the same time, the Python version completes only 75 thousand simulations. The compiled Cython version is over 100 times faster!

So how does a programming language that runs two orders of magnitude more slowly than the alternatives become so common? This is interesting, because Python has been around for 30 years, but its use has only really exploded in the last decade.¹ The other thing that has exploded over this timeframe is the amount of excess compute – a term that refers not only to processing speed but also to storage capacity and information transmission bandwidth. A generation ago we marveled that a student calculator had more compute than the guidance systems that landed man on the moon. Today, there is so much compute in a typical smartphone that developers rarely have to worry about speed or memory. And so we use tools to build and publish apps that are hundreds of times larger and slower than necessary because (a) they’re easier and (b) it’s no longer worth the effort to make them small or fast.

Have you ever noticed the size of installation files for smartphone apps? Very few weigh in at less than 10 megabytes. Heck, the basic calculator app on my phone takes up 8MB of memory! Why? 30 years ago the same app under Windows 3.1 used 41kB of memory (that’s 1/200th the size).

Smartphone calculator app listing showing 8 MB of storage used — 8 Megabytes of functionality here?

It’s not a coincidence that I didn’t use Python much earlier. Most of my work is in the finance industry. In the 1990s we could never get enough storage for market data or speed to run analyses, so we spent a lot of time optimizing our C++ code to squeeze as much as we could out of our compute resources.

In the early 2000s I still knew every detail of every available piece of computer hardware because I was working them to their limits. I still remember my excitement to get a server with two 64-bit Opteron CPUs and to break out of 32-bit memory space! The early 2000s were also the point at which we could afford to use a SQL database instead of handling every detail of reading and writing our data to files and keeping track of exactly what was being held in RAM at each moment.

The last decade is when we could get really sloppy. With gigabytes of RAM on every computer it’s usually possible to keep everything of interest in memory; no need for a database. Where previously I used C++ or C# to build production systems for trading and portfolio management, computers are now so fast that I actually built and run my most recent system in Microsoft Excel! I’m not kidding: Here’s a screenshot of Excel (left) consuming realtime data and generating portfolio analysis next to an Interactive Brokers Trader Workstation (right):

Microsoft Excel consuming realtime market data beside Interactive Brokers Trader Workstation — These days, real quants can make everything work in *Excel*

1. TIOBE publishes data on programming language popularity. Stack Overflow is another good indicator, which is where I got this chart:

Stack Overflow question-share trends for C++, Python, Java and C#

Where Do Trading Profits Come From?

Posted on August 5, 2026August 5, 2026 by David Bookstaber

This is a question I first encountered in college, where it was tinged with idealistic indignation at this glaringly capitalist phenomenon. Since I have continued to work in the industry I regularly encounter it in various forms, and it’s worth explaining in detail.

Capital markets are institutions that match capital suppliers (monied investors) with capital demanders (typically, businesses that need money to make money). Demanders attract suppliers by offering to pay them for the use of their capital. For example, a business might sell standardized instruments like stocks (which confer ownership and may pay dividends) or bonds (which pay interest). Secondary (trading) markets for these instruments tend to be pretty efficient, which means that on average there’s little money to be made by trading one stock or bond for another. In fact, trading in secondary markets is roughly a zero-sum game: every dollar earned by one trader comes out of the pocket of another. Which is why investors are encouraged to buy and hold, and why active trading tends to be a money-losing activity.

But there are enterprises that make large and consistent profits by actively trading stocks and bonds. This leaves many laypeople understandably confused: They are warned that if they actively trade they should expect to lose money, while professional trading operations consistently win money. So is trading profitable, or not?

In healthy capital markets, there are three legitimate ways to make money by trading: Providing liquidity, information, or insurance. (There are also some illegal ways, like manipulation.) Nearly everything the trading industry sells, and nearly every fee it collects, traces back to one of those three services. And if you’re not providing one of those services when you trade, then you’re probably paying for them.

The first service, liquidity, is the ability to conveniently value and exchange assets. The owner of a liquid asset knows at all times what the asset is worth, and can quickly sell the asset for close to its full value. Liquidity is the foundation of capital markets: investors supply a lot more capital when they are confident that they can get it back when they need it. Without liquid markets the gears of capitalism can grind to a halt. But liquidity doesn’t happen in a vacuum, and the suppliers of this metaphorical lubricant are paid for it. In the old days of centralized stock exchanges you could point to the front-line liquidity suppliers: specialists staffing the exchange stood ready to buy and sell stocks. They posted the prices at which they would buy (bid) or sell (offer) shares, and the cost of their service was baked into the difference between the bid and offer prices. Today’s stock exchanges are more complex and less centralized, but the fact remains that they depend on liquidity providers.

Another layer of liquidity waits outside the exchange to step in when the demand for liquidity exceeds what market makers can provide. Imagine that someone wants to sell a large amount of stock. As they begin to sell, the routine supply of buyers at the market price is exhausted – nobody is left willing to buy at that price. But lower the price and the stock begins to look like a bargain, which brings in a new supply of buyers. A trader who stands ready to buy when others are desperate to sell is supplying exactly what is scarce in such moments, and is paid for the service. Liquidity provision is a source of profit that can persist even in efficient markets – compensation for storing money and accepting risk when other participants are scrambling for cash.

The second service is information. Traders with special information indicating that a stock is undervalued can buy it, and if the information is correct then they will profit as the stock’s price moves towards its fair value. For example, an analyst might conclude that a company’s new product will be more successful than expected, and that its stock price has not increased to reflect that. If the analyst buys the stock, that purchase pushes the price up marginally – communicating the information to the market, but not in a way that produces any immediate profit for the analyst. The analyst’s profit is most likely realized when the information is proven correct: If the company’s earnings beat the consensus, its price will jump and the analyst can then sell for a profit. The difficulty is that this kind of edge tends to consume itself: public information is already incorporated into market prices. To make money trading on information you have to find a proprietary source – famous ones have included counting shipping containers moving through freight hubs. And information no longer pays once the source becomes widely known or available: the market price simply moves to incorporate it without any friction that a trader can exploit.

The third service is insurance, or the assumption of risk for a premium. This is subtly built into the prices of all investments, but risk can also be explicitly traded via derivative contracts like futures and options. Someone who wants to limit potential losses on a stock can buy an option. In that transaction, the option seller pockets the option’s premium in exchange for carrying its risk.

So who gets paid for providing liquidity, information, and insurance? In theory, anyone can, but it’s a competitive market that has grown increasingly dominated by professionals with tools and skills not available to lay investors. When you see a stock price dip can you tell whether the move was caused by someone demanding liquidity and not somebody trading with special information? Can you precisely calculate the “insurance” (risk) component of an asset’s price? Professional traders can. Or rather, any trader who can’t won’t stay in business for long.

It is worth stepping back to see what all of this accomplishes. In the normal course of events capital markets are like a nuclear reactor, pooling capital and exchanging risk to create heat that powers the economy – and yes, like a reactor, they occasionally melt down. But without the concentration and free exchange of capital and risk no heat is generated and economic development is stagnant. The profits earned in trading are what the economy pays to keep that reactor running, and, stripped to their essence, they are payment for just three things: liquidity, information, and insurance. If you’re trading without providing any of those services, then you’re probably paying for them.

Streaming Real-Time Market Data into Excel: Why RTD Beats Polling an API for Analysts

Posted on July 13, 2026August 11, 2026 by David Bookstaber

Financial analysts live in Excel, and I’m no exception. Python (enhanced with pandas, numpy, etc.) is more powerful, but sometimes the convenience of having a complete data set exposed in Excel is worth the tradeoff. And modern Excel is strong enough to run a realtime trading dashboard. I’ve done that for years now. And a few years ago I discovered that it’s really strong if you use the right tools. One of those is Microsoft’s Real-Time Data (RTD) protocol, and even though that was introduced to Excel almost 25 years ago I only recently discovered it and began to unlock its power.

Before I moved my dashboard to RTD, I pulled positions and account values via DDE (Dynamic Data Exchange), which is a protocol introduced 40 years ago! Ancient doesn’t always mean obsolete, but in this domain DDE hit its limits long ago and has been (appropriately) deprecated by Microsoft. Before I abandoned it, getting DDE to reliably feed data into Excel required jumping through hoops. Just establishing a connection between Excel and the broker’s API required starting a separate Java bridge process. The spreadsheet receiving the data had to be running in its own Excel process because every DDE update would lock the GUI. And if you happened to touch that workbook at the wrong moment in its update cycle it would lose the connection or crash completely. So on top of the Excel dashboard I wanted to work in, I had to start a separate “feeder” Excel workbook and a Java bridge. And the result wasn’t literally realtime – data arrived in my dashboard when the feeder was able to push it, which could be as frequently as every 10 seconds – but that was adequate for my needs.

And kludgy. But that’s the way all Excel data feeds were: You had to shoehorn cell updates into a system not designed for continuous calculation. Anyone who has added a “volatile” function to a heavy workbook knows this peril – or gives up – because every update means a global recalculation. The countermeasures are familiar hacks: polling loops, event handlers, lots of Visual Basic, etc.

I first saw the convenience of RTD when I used it to stream market data from ThinkOrSwim. Every other data source used an Excel add-in. ThinkOrSwim just magically appeared when needed. =RTD("tos.rtd",, "LAST", "SPY") would update the last trade price of SPY multiple times per second (once Excel’s default two-second RTD throttle is dialed down). But how? Unlike add-ins, you couldn’t even tell it was available until you asked Excel for it. It never consumed a measurable amount of CPU, never blocked the GUI, never lagged, never crashed. And that function: start to type “=RTD” and Excel reveals it’s a native function. This was a realtime data feed mechanism built in to Excel!

Once I began to dig into RTD I couldn’t stop. The interface could hardly be more simple: an RTD service only has to implement six methods and register itself as a Windows component. The mechanism was elegant: Excel tells the RTD server what topics the user has requested (e.g., “the last trade price of SPY”), the RTD server tells Excel when there is new data on that topic, then Excel asks for the data as soon as it can receive it. This was the trick to avoid blocking or crashing Excel, while feeding it data updates as fast as possible.

Implementing an RTD server properly turns out to be trickier than one might expect: You have to be very careful about thread management and subscription accounting. And you have to get the RTD server talking to the data source through whatever API is on the other end. But I think it’s worth the trouble: My Excel trading dashboard now gets realtime data through virtually invisible servers that have withstood all my attempts to break them. For example, I streamed 3,000 stock quotes through the market close on a busy day, and Excel just kept working. Once I’ve seen how it can be done right, I can’t go back.

Privacy.com to thwart subscriptions

Posted on June 30, 2024June 30, 2024 by David Bookstaber

Businesses love the recurring revenue from subscriptions. And they love the fact that people tend to be so lazy and forgetful that customers don’t reliably cancel subscriptions that they wouldn’t otherwise keep. As a conscientious consumer, I dislike subscription services because businesses have a disincentive to make cancelling subscriptions easy. I disapprove of the roach motel business model. My favorite countermeasure: Privacy.com.

Privacy.com lets you create credit cards with all sorts of constraints, as shown in this screenshot:

Screenshot of Privacy.com credit card limits.

I particularly like the single-use card, which I create for subscription services. If I forget about the service they can’t keep sucking money from my bank. If I want to cancel the service, I don’t have to find and navigate their arbitrary cancellation processes … and keep records of cancellation attempts to dispute recurring charges from those businesses that are either incompetent or downright fraudulent when it comes to cancelling subscriptions. If I do want to continue subscribing, I have found that every business goes above and beyond to make that as easy as possible!

Bitcoin Bubble

Posted on May 19, 2021 by David Bookstaber

I explained this in more detail last month on my finance blog, but for the record here:

Bitcoin (BTC) is a pathological cryptocurrency. BTC does not represent the opportunities that are emerging in the decentralized-finance (“DeFi”) paradigm. BTC is one of many evolving cryptocurrencies. As an early mover it attracted speculators who think “crypto” is the future but who don’t have an intelligent way to invest in that future. The speculators drove a bubble in BTC price which then created a mob of “Greater Fools” who heard stories of people making windfall profits in BTC and didn’t want to miss out. This is a classic manic bubble.

Tax Optimization: Charitable Giving

Posted on April 15, 2021April 15, 2021 by David Bookstaber

If you don’t itemize deductions when filing your personal federal taxes then you get no tax benefit for charitable giving. But if you coordinate charitable donations with other taxpayers you can keep more money out of the hands of the government.

Important tax terminology: A “deduction” is an expenditure or loss that reduces the amount of income that is subject to tax. Individual taxpayers have the option of either listing (“itemizing”) and subtracting all allowed deductions when calculating and filing their taxes, or instead just subtracting a Standard Deduction. Itemized Deductions can’t reduce your tax bill unless they exceed your Standard Deduction. The Standard Deduction for tax year 2020 is $12,400.

The idea behind the Standard Deduction is that on average every taxpayer has some deductions. Correctly itemizing deductions – i.e., listing them out and ensuring they are valid and compliant with the tax code – can take a lot of work. In order to simplify tax compliance for the average taxpayer the tax code offers the Standard Deduction for free. Taxpayers who think they have more deductions than what is assumed by the Standard Deduction can choose to do the accounting work to claim those deductions.

When you claim Itemized Deductions you “lose” your Standard Deduction. This is only meaningful if you can shift deductions around. Shifting deductions is not something the tax code was designed for, but it can be done legally and it’s a big part of tax optimization.

Shifting as an individual (a.k.a. “bunching”)

If you claim the Standard Deduction then your Itemized Deductions have no tax value. For many taxpayers the biggest deductible expense – and often the only one that can be easily “shifted” – is Charitable Donations. In order to minimize taxes you should “bunch” donations into a single tax year in which you Itemize Deductions, and then claim the Standard Deduction the other years.

For example: Suppose you tithe $10,000 a year to your church, and you don’t have many other deductions. If you pay your tithing once per tax year you would get no tax benefit from your tithes because it doesn’t exceed the Standard Deduction of $12,400. But suppose you paid your 2019 tithing all on 1 January 2020, and then you paid your 2020 tithes on 31 December 2020 (instead of in January 2021). Now for tax year 2020 you have $20,000 in deductible donations (which is worth itemizing), and in 2021 you can claim the Standard Deduction.

In practice not so many individuals will find themselves able to profitably shift Itemized Deductions.

Shifting within groups

When coordinated within a group, shifting charitable donations can readily save a lot of taxes. Remember that as an individual claiming the Standard Deduction your donations have no tax value … to you. But if you know somebody who is Itemizing Deductions and you donate through them then your donation will reduce their tax bill!

How do you donate through someone else? Whatever you want to donate to a charity you “gift” to them, and then they donate it instead. (Shifting like this need only be a paper transaction: So long as they get the receipt they can claim the donation.) The tax code contains an annual gift exclusion – currently $15,000 per person. I.e., you can gift anyone up to $15,000 per year without triggering a gift tax. (The gift exclusion is per person, so you can give a couple up to $30,000 per year. This is relevant because spouses have to choose between the Itemized and Standard Deduction whether filing separately or jointly.)

For example, if everyone in your family gives $10,000 a year to your church, it could be that nobody is able to deduct the tithes. If everyone instead gives their tithes to one donor and that donor Itemizes Deductions, then the donor gets the full tax value of the donations and everyone else gets the full value of the Standard Deduction.

The tax benefits compound further if the donor is the group member with the highest marginal tax rate. For example, suppose you and your siblings live in states with no income tax and you only earn enough to put you in the 24% tax bracket. If you itemize and deduct everyone’s tithing then each $10,000 donation will save you 24% of $10,000, or $2,400 in taxes. But suppose your father lives in New York City: his marginal tax rate could be over 45%. If he donates and deducts $10,000 it saves him $4,500 in taxes!

What benefit do you get by shifting through someone else? There’s the principled argument, which is that big government is wasteful and does not spend money wisely, and it is therefore best to keep as much money out of the government’s hands as possible. (If you disagree, then you are welcome to pay more in taxes than required by law: Donations to the U.S. Treasury are tax deductible! But for all the big-government sanctimony out there the reality is that virtually nobody gives money to the government voluntarily.)

Then there’s the practical consideration: You could share the tax savings created by shifting. For example, if you gift your father $10,000; he donates it to your church; he saves $4,500 in taxes; he could gift that full $4,500 in savings back to you and be no worse off.

Bubbles

Posted on January 30, 2021January 30, 2021 by David Bookstaber

This post is not a reference to current stock markets or the throngs of new retail traders chasing extraordinary profits. Not at all. This is just some of my favorite pictures of one of my boys and his fascination and delight with bubbles.