I’ve always been fascinated by electronic trading and my strong desire to progress the evolution of trade automation technology is one of the things that has kept me pushing forward in the fintech industry. I’ve seen many hurdles as the role of financial institutions is to keep assets safe, not to have the flashiest technology. This ends up being detrimental to technological innovation, flexibility, growth, and speed of evolution.
My favorite projects have always revolved around designing algorithmic rules engines for institutional FX traders to alleviate the burden of their many multi-click, manual, menial tasks. Automation enables them to focus more on the strategic trading that ultimately improves their profits. I’ve learned that with strong automation of the institutional/transactional FX lifecycle, relatively little human intervention needs to take place (transactional trading differs from speculative or proprietary trading).
Algos and HFT
There is still a lot of confusion around algos and HFT, what they are, and if they’re bad or good for the markets. Simply put, an algorithm is a programmed set of steps designed to accomplish a task or solve a problem. Automated trading is electronic trading using algorithms at some stage of the trade lifecycle. The broader categories of algorithmic trading are execution and decision based algorithms. Execution based algos are those in which a human has made the decision to trade but algorithms are incorporated in the execution process to automate the process and achieve best execution. Decision based algos both make the decision to trade based on a programmed model and carries out the execution with no human interference.
High frequency trading (HFT) is a subset of algorithmic trading that falls more into the decision-making category. The trade characteristics that constitute HFT are likely not what would come to mind for those unfamiliar with algo trading. Generally occurring in large quantities, the individual trades are very low risk in that they’re small in size and profit, occur in highly liquid markets and assets, and take place in times of low volatility. They also hold risk for an extremely short amount of time, keeping a position on for only a matter of milliseconds.
My first hand experiences with institutional algorithmic trading
Whenever I mention the words “trading algorithm” to someone unfamiliar with technology, I feel like I get a mix of uncomfortable, confused, or scared looks. This is where I backtrack and elaborate by explaining how algos can be something very basic and used for every day convenience, like how hitting thumbs down on three consecutive disco songs tells Pandora I probably hate disco. They can also be more complex and predatory, like HFT algo stop hunt strategies that slowly drive up a securities price, then offer a very low quote to trigger other market participant’s stop orders and dip the price so they can buy cheap before driving it up again.
Having been on both the institutional and retail sides of trading, I know there is animosity, confusion, and anxiety amongst individual investors around what institutions do with algorithmic and high frequency trading (HFT) behind the scenes. Trading algorithms can be complex, and unless one has a solid grasp on both quantitative trading and how the technology behind them works, the intricacies can seem mysterious. Since it’s human nature to fear the unknown, it’s easy to see how not understanding the ways institutions develop and leverage trading algos can seem scary and murky.
One of the benefits of building institutional trading platforms was I developed a deep understanding of the infrastructure that underlies the institutional investment industry. Though not all of it was flashy and sexy (APIs, FIX messaging, post-trade settlement), knowing the asset management and investment banking electronic ecosystem inside and out makes understanding trading now so much easier. The investment “interface” society experiences and interacts with as a whole via news, online stock charts, entering orders through their brokerage accounts, or simply choosing a security to invest in is just the very tip of the iceberg. There is such a massive world of technology operating under the surface to keep quotes updating, orders matching, and transactions moving. Not to mention the diverse group of market players impacting a security’s price movements.
Building multi-bank, dealer agnostic trading platforms allowed me to see a holistic picture of the trade lifecycle door-to-door, buy-side to sell-side. When I was helping train new hires or interns, I learned there was a lack of online resources that explained institutional trading. An abundance of information on retail and day trading exists, but most only guess or assume what the “big guys” are doing. I determined it was because there aren’t many who work in institutional FX trading technology as only a few larger platforms dominate the space and also because FX is still a relatively new asset class to be moved to electronic (a lot of phone, or “voice,” trading still happens, believe it or not).
I had the great fortune of having the opportunity to help design and build an institutional FX trading platform from the the ground up at one of the fintech startups I was an early employee of. Below I will explain some of the basics of how I experienced institutional trading. Though I specialize in FX, much of the concepts are the same across securities and most platforms are multi-asset.
The majority of FX volume traded is institutional, estimated throughout the industry to be over 90%. Most buy-side institutions are managing assets on behalf of their clients, which trace back to either individual investors or companies entrusting them to manage, for example, all of their employee’s 401ks. When these (very large) asset managers use trading algos, whether proprietary or those of their counterparty bank, they’re generally the safe and non-predatory kind. This is to ensure the trades are executed in a controlled manner and without the risk of losses from a complex, speculative quantitative algo strategy.
To best automate asset manager’s workflows, I spent a lot of time learning everything I possibly could about trading algorithms. At the fintech firm, I put my knowledge to work and started by using Progress Corticon, a heavier front-end tool that enables business analysts to build rules based algos.
I’d make a very simple staging algo along the lines “if this incoming order can be netted with other like orders, flag it as nettable” then, another might be “sweep the blotter for orders <1mm USD equivalent with like given/settle currencies, value dates, broker restrictions, and custodians and net them all together”. The next would be “if the netted order is under X amount, send it to the account’s custodian bank for execution to avoid trade away fees incurred from putting it out in competition and executing with a bank other than their custodian” (on small orders, the trade away fee cancels out any profits from better pricing).
Things get a bit more complicated with execution management systems (EMS), where actual acceptance of a price and execution of a trade takes place. This is where designing FX rules engines can help in achieving best execution in a multitude of ways. Headless trading isn’t actually as scary as it sounds and, with proper controls in place, can sometimes be safer than manual execution. In the example above, another algo could trigger that would accept whatever price the custodian gave (PIP thresholds added to reject anything out of a certain price range). Things like indicative rates (aggregation of rates being offered by various liquidity providers) and transaction cost analysis (TCA) reporting can be incorporated into the algo’s execution processes.
With competitive institutional trading, a buy-side selects counterparties and does a request for stream (RFS, quotes generated from a pricing engine) or RFQ (competitive quotes submitted by either an engine or human trader). In FX, most banks have pricing engines that receive an order, run a calculation, and stream their prices back through the API to the buy-side trader’s firm. On the trader’s screen, they see each bank’s prices updating at whatever speed interval the trader’s firm has configured. HFT firms can receive price updates extremely rapidly with less risk of performance issues and some opt to forgo any stream lifetime parameters (i.e. stream prices with no time limits). Generally, on a market order, non-HFT traders will just take the best price available by clicking a best price button within the allotted time window and resubmit upon time expiry. Or, they can set up order rules, like a limit order to execute at a certain price and keep the price stream running.
Many third party trading venues support the algorithms offered by investment banks and liquidity providers. FX bank algorithms were mostly VWAP (because currency markets trade OTC dealers do not officially report transaction volume, so banks have various versions of this), TWAP, etc. The purpose of these algos is to enable asset managers to achieve best execution on their order, which in the end, benefits the “little guy” (i.e. the teacher who’s pension is being managed by the firm). This is some of the more complex automation people imagine when they think of trading algos. I see algos as a very positive technological advancement that contributes to market health, efficiency, balance, and progression.
HFT in FX
Since I am most familiar with FX electronic trading, a research paper by the Bank for International Settlements Market’s Group on HFT in the FX market made the workings of this type of algo trading very clear to me. It was stated that since most of the early FX HFT strategies were originally developed for equities, there is overlap, so anyone familiar with equities should be able to follow along no problem.
FX was one of the last markets to move to electronic trading, mostly because it’s OTC and currencies are non-standardized instruments that don’t trade on an exchange. RFQ was (and still is, a bit) done primarily over the phone and it wasn’t until the early 2000s when FX dealing banks began to offer single dealer platforms.
The advent of these platforms was what lead to retail traders being able to access the FX market. Previously, only dealing banks and asset managers participated in foreign exchange trading. In 2012 when I was at a Boston FX hedge fund that created trade replication technology, the FX retail market was described as a very lonely place where traders operated solo on computers in their basement. It was also presented as dangerous for investors who were not professional FX traders, which is true, hence the firm’s trade replication technology.
As FX electronic trading evolved, it wasn’t long before HFT permeated the space. HFT strategies generally operate in highly liquid assets, so the $5.3 trillion a day FX market would be appealing, especially in the G10s.
HFT is conducted primarily by independent, specialized firms, which trade mostly for their own account. Banks do some HFT in proprietary trading accounts, but they’re not major players and don’t see HFT as an integral part of their business, but rather, as a way to keep up with the technology.
HFT is transacted over various platforms, some of which are geared more towards interdealer trading (EBS, Reuters), and others, mostly electronic communication networks (ECN)s, that are leveraged more by HFT firms. Both the interdealer and ECN platforms have similar connectivity APIs and messaging standards (FIX protocol). Non-HFT institutions leverage both platforms, while newer, smaller HFT firms leverage ECNs to access some of their prepackaged algos. Larger and more sophisticated HFT firms tend to use the wholesale interdealer platforms, which have higher liquidity, larger trade size requirements, and larger spreads.
HFT firms permeated the FX market after equities since it is more fragmented with no “exchanges,” so price sources are all over, produced by dealers directly or through any number of trading venues. One way is by trading on the same platforms the banks and buy-sides use. They will either secure access to multiple platforms to take advantage of price discrepancies and gain a broader view of the market, or gain access to dealers via prime brokers.
Banks don’t really like HFT firms as the quick “fill in the gaps” trading causes spreads to tighten, and bank’s money is made on the spread. They feel the ultra quick position holding time of HFT strategies doesn’t embody the true spirit of the risk of trading. The bank isn’t taking on nearly as much risk by acting as their counterparty since they’re in and out of the deal in seconds or less, unlike traditional buy-sides who keep positions on for much longer. I still see it as dog eat dog in trading, quite literally every man for themselves and if you don’t use all available resources to make the profit, someone else will.
Current State of HFT
When trading as a whole was moving to electronic, ample opportunities existed for HFT strategies to use their speed for exploiting price discrepancies. This was due to market fragmentation across all asset classes as well as any lag in institutions updating their technology, which put them at a huge speed disadvantage.
Now that the first big wave of electronic trading evolution has calmed and most all market participants are on a similar modern technology playing field, HFT firm’s revenues have been dropping. According to Tabb Group, HFT firm revenue was set to fall below $1 billion, $6.2 billion less than what it was in 2009. The US equity daily volume that HFT firms are responsible for has also leveled out over the past few years, now falling somewhere around 55%.
Ever since reading Michael Lewis’ Flash Boys book on HFT in 2014, I always saw the technology aspect of the movement as a temporary phenomena (one of the founders of a startup I was at had us all read it when we were working there as the “for the people” transparency of IEX was similar to what we were building). A sky’s the limit race to the top is one thing, but HFT is physically a race to the bottom. And the thing about a race to the bottom is there’s a solid end, and until someone learns how to bend the space-time continuum, time can only be reduced to a certain point.
The initial flourishing of HFT was just a temporary discrepancy resulting from the evolutionary fragmentation of electronic trading systems and their technology that’s being exploited until the resources (speed, time) are used up. Now it’s just part of the landscape and what were once unbelievably fast millisecond entries and exits will be the norm.
Pros and Cons of HFT
HFT is not malicious; it is the leveraging of HFT technology and incorporation of it into predatory algorithms that causes problems for investors. It’s hard to quantify if any piece of HFT itself benefits the market, but as a whole, it’s impact is pushing the industry to innovate and get more powerful with their technology to stay competitive, which is a good thing. Markets are more accurate with higher processing speeds, which contributes to better data integrity and pricing. Below are a few pros and cons of HFT:
Liquidity: Because HFT firms produce such a high volume of trades so rapidly and across a broad market, they increase and distribute liquidity (like bees). They also contribute to market making by providing prices, rather than just taking prices via execution.
Volatility: Since HFT likes to operate in low volatility and high liquidity markets and assets, intraday trading is a good time for them. Much of their trading causes market movements (likely from triggering technical analyst’s orders), which contributes to waking up sleepy midday markets and generating volatility.
Spread tightening: HFT balances the discrepancies resulting from market fragmentation. By moving in and out of positions so quickly, it literally fills in the gaps resulting from lags.
Inspires innovation and competition: As I mentioned before, non-HFT traders will need to innovate to compete.
Technical analysis: Increased algorithmic and HFT market activity means individual investors are very likely to be transacting with a robot. Those who formerly relied solely on fundamentals will now benefit from incorporating similar technical trading systems into their approach. Those who are already technical traders will benefit from more market participants joining in on the same signals that will move prices further in their direction.
Retail traders: Since HFT strategies aim to make a small profit off many transactions, they target larger orders put out by institutional traders. For the most part, predatory HFT strategies won’t be targeting the smaller orders of retail traders.
Quality of liquidity: Since HFT strategies have such short position holding times, there is the phenomena of liquidity evaporation. The liquidity will be there one (milli)second, and gone the next, making it difficult for traders to execute and be filled on the quote they see. Also, since their lots are often small, problems arise with depth of book. A price might be available but only for a small fill, so depth of book is important. Since HFT strategies don’t thrive in volatility, when markets get choppy, HFT market makers could leave quickly, pulling with them much needed liquidity.
Not always stable in volatility: If volatility arises quickly and HFT firms remain in the market, they can basically turn into a bunch of chickens with their heads cut off. Should a fast negative market move occur, HFT strategies can be triggered, exacerbating the direction of the move, thus amplifying it (2010 Flash Crash).
Difficult for retail traders to compete: HFT makes it hard for even the most technology rich institutions moving multi-billion dollar orders to compete, so retail traders are at even more of a disadvantage. Not having access to HFT resources means retail traders must educate themselves on how they operate and implement strategies to work with them.
Understand who your fellow market participants are
It’s clear to see market participants have evolved since the pre-computer days of pit trading at the Merc. Understanding who is participating in the markets and knowing who you’re trading against is imperative to making an informed decision on how you believe the market will react.
Both JPMorgan and Tabb Group have reported that US equity volume is roughly both 90% institutional and 90% algorithmic. The Congressional Research Services estimates that as of 2016, HFT is 55% of trading volume in U.S. equity markets and about 40% in European equity markets. Likewise, HFT has grown in futures markets to roughly 80% of foreign exchange futures volume and two-thirds of both interest rate futures and Treasury 10-year futures volumes.
Psychology plays a huge role in market movements so knowing what’s going on inside the heads (or CPUs) of those you’re competing with is very important.
How to thrive in the current market: Adapt, evolve, and embrace the machines
Technology is constantly evolving and should be embraced, not feared or fought. It has been evolving the markets since the beginning. Take ticker tape, for example; what started in the 1800s as stock quotes being hand delivered, evolved into typewriters printing prices, then 1930s machines with a 15-20 minute price update delay, to electronic computerization of tickers in the ‘60s, to today’s streaming multi-price per second updates displayed on individual user interfaces. Each technical evolution increased market efficiency, accuracy, and fairness.
Well functioning markets require maximum market participation. The more participants from a diverse group, the better. If people are scared of algos and HFT because they don’t understand it, they’re less likely to participate. Putting in the time to educate yourself on the technical side of things will help you to avoid being prayed on by them.
Investors must understand that we’ve entered into an era where a large portion of market participants, given the fact they’re robots, are leveraging technical analysis. They’re trading against machines now and they don’t have the same emotions human’s do. This means they won’t always react the same way to news or an event. It also means they’re a lot better than a human at not letting their emotions cause them to deviate from their strategy. I believe fundamental analysis definitely has a place and is useful, but at this point when a lot of volume is moving off rules based algo trading systems, the same fundamentals don’t always apply. So, at the very least, incorporate some technical analysis in.
Many think it’s so random when the market moves in a way they didn’t expect, but it’s really not that random or surprising. If the majority of trading is algorithmic, then the majority of market movement is coming from algos following their systematic rules and responding to various price actions.
Take the 2010 flash crash for example; HFT algorithmic trading strategies are not built to operate in volatility. At first, the algos started absorbing some of the fundamental investors (non-HFT) equity selling pressure, causing them a net long position. Since HFTs have a low risk appetite, they don’t hold their positions long so they began unwinding, competing with the fundamental traders for liquidity and bringing the entire market down with them. In the absence of fundamental buyers, the HFT algos ended up playing a game of hot potato with one another.
No matter how complex these algorithmic trading strategies are, it is still always a human that conceptualized, designed, and built them. I realized this after spending countless hours talking to institutional traders at both asset management firms and investment banks to understand their workflows and get inside their heads to replicate and automate their trading strategies. No matter what, it will always be a human behind the robot. So think about what the human would program a robot to (unemotionally) do to make a profit while trading.
If you can’t beat them, join them really couldn’t be more fitting here. If you can move with algo strategies or trigger them to move with you, you will drive prices up or down in your favor.
New technologies, such as AI and machine learning, are continuously being leveraged by the investment industry. Fintech companies like Quantopian, Algoriz, and QuantConnect are emerging and offering education and front-end tools for non-programmer individual traders to build quantitative models.
Experiencing both fintech startups and corporate trading environments, I know how eager small tech firms are and how fast they move to disrupt that space. The finance and investment industries are going to keep on evolving and those who don’t get on board will definitely be washed away in the wake. Don’t fear the machines; adapt, embrace, and move with them to keep progressing.