Commitments of Traders – Is it a Useful Tool to Time Markets?

Trading with insider information is an illegal practice in the financial markets, however the Commodity Futures Trading Commission issues a report each week that might come as close to legal insider information that one can get, at least for the U.S. futures markets. But is it useful as a tool or method to help the trader to be on the correct side of the markets?Before attempting to answer that question it is best to explain what this report is and what it contains in the way of useful information. The Commitments of Traders data (which I’ll simply call COT data from this point forward) is released each Friday (or Monday if Friday is a holiday) based on data collected the previous Tuesday. This delay in the release of data may be a factor for very short-term traders, but should not be a concern to the longer-term position trader. This data breaks down the number of contracts held long and short for commercial traders, large traders, and small traders. Every U.S. futures market is represented that has at least 20 large traders holding a position. The current data is available in text format and can easily be importing into an Excel spreadsheet. Also, all the history data is available in both text and Excel format. Once the data is in Excel, it can be plotted along with price and various technical indicators using charting software such as TradeStation. There are also software programs that can help manage the data, as well as vendors that sell the data, however the data is free for anyone to download directly from the government CFTC website.When looking at the raw data, you will notice many columns of numbers. Most of these columns are of little value in analysis. On the CFTC site there is a page that tells you what each column represents. The only columns useful for analysis are the long and short positions of the large, reporting traders, the commercial traders, and the non-reporting traders, and of course the date. The reporting level will vary by market. Also, there are many oddball markets that most traders will never trade. I just delete or ignore these. Some markets with low open interest will appear one week and then be gone the next if the level of large trader participation drops below 20, so it is best to stay with the liquid markets.One more point on the available data is that there are two reports: one showing the futures contracts only, while the other shows the combined futures and corresponding options. In the past the futures data was released on Friday, while the combined futures and option data was not available until the following Monday. For this reason most traders would collect the futures only data. Now both reports are released on Friday. Also, it seemed in the past that the true price driver was the futures contract, with the option being of less importance as it was often used by small traders, or for various spread strategies. Now with the growth in volume and interest in the futures markets in general, and especially the increase in hedge fund volume, options may have more significance in the analysis. There is debate on this issue. Use your own judgement. You can easily try both and see which method seems better for the markets you trade. On some contracts the futures only and combined reports produce similar results, while in other markets there is a noticeable difference.Once the data is imported and plotted, along with the weekly prices of the underlying market, the most common way to view the data is to plot three indicator lines representing the net position of each of the three trader groups. You simply subtract the number of short contracts from the long contracts for the large traders to get the large trader net position, and plot that line in one color. Then do the same thing for the commercials and plot on the same sub-graph with a different color. And then do the same thing for the smaller, non-reporting traders in another color. If you want a quick view of what this looks like without having to manage the data, there are many web sites that one can visit with charts already posted, although your ability to do further study on this data will be limited, and in some cases the data may not be as timely or accurate as you might like.NOTE: for TradeStation users please go to end of this article for more information on importing data.

So are these three lines of any help in trying to determine the future course of prices? There is a saying on Wall Street (and LaSalle Street) to follow the smart money. But what is the smart money? It would seem that the insiders and therefore the smartest money would be the commercials. They have the inside knowledge of what is happening with their own market. If they are farmers, they would certainly know the condition of their crop, what the weather is doing, what the demand should be for their crop. If they are a flour mill they should know the condition of the wheat crop and what the likely supply should be and what their needs will be for buying in the future. Since the insiders know everything that can be known about their supply and demand situation, it seems that it should be an easy matter to simply follow the smart money and just trade in the direction of that smart money. But a quick look at the price action shows that the commercials are almost always on the wrong side of the market. How can this be?

soybeans.png
The above chart has weekly prices in the upper graph. The lower graph displays the three lines described above, with the red line the commercial net position, the blue line the large trader net position, and the yellow line the non-reporting or small trader net position.

To explain this discrepancy it is important to understand the different motivation and resources of the commercial trader from that of the speculative trader. It is said that commercials have deep pockets and staying power, so they can ride out large adverse price moves. The basic function of the commercial is to lock in, or hedge the price of an asset or the need of an asset at some point in the future. If a farmer has crops in the ground and he has expenses to meet, he often can’t take a chance that the crops will decline in value, so it is wise to fix the price now by hedging in the futures market. The fact that the farmer actually owns the crop and can deliver if needed creates the sense of deep pockets and staying power. Even if the price goes into an uptrend the producer of the commodity will often stay with the hedge so he doesn’t have to worry about price fluctuations. So the motivation of the commercial hedger or user is far different from that of the trader betting only on price direction. Often commercials will bias their hedges based on future assumptions about price direction and lift or increase their hedges based on their view of the market conditions. However they, as a group, still have a different reason for being in the markets, and price speculation is of secondary concern, at least in theory. The large trader is only interested in where the price is going and has no interest in owning or delivering the actual commodity.

When looking at the net positions of the large, reporting traders, it seems that their net position is almost always a mirror image to the commercials. So if the commercials are the smart money, but generally are on the wrong side of the trends, then it would seems that the large traders are really the smart money, as their net position usually grows in the direction of the trend. This seems odd since most of the large traders, which are mostly hedge funds and pools that often rely on technical trend following systems, can’t possibly posses the inside knowledge and fundamental information of the commercials. Yet the charts speak for themselves.

Could the answer be in using the small, non-reporting data. It is said that it is a good idea to fade, or trade against, the dumb money, and small traders are often regarded as dumb money. This assumption could be misleading. The non-reporting category is made up of traders that fail to meet the reporting requirements of that particular market. Also, there is no distinction between commercial traders and speculators when they fall under the reporting requirement. This group isn’t made up necessarily of one and two lot traders. They could still be hedge funds and professional advisors. Just because their position is under the reporting requirement doesn’t make them dumb, or even that small. There are many good traders that trade large size, but just not large enough at times to require reporting. In some cases the reporting threshold is very high for an individual or a small hedge fund, producer, or user. Also, since the small trader category is made up of both hedgers and speculators, the line on the COT chart is often bouncing around the zero line, or net neutral position, therefore offering little in the way of patterns or clues.

So it seems that of the three groups of traders, the obvious group worth watching to stay of the correct side of a trade is that of the large trader. In a trending mode this would seem to be the correct assumption. It seems wise to follow the money going into a market with the group that is speculating on the direction that price seems to be moving, rather that betting with the group that has other reasons for being in the market. But there does come a point where the composition of traders becomes too one sided. That perhaps is the key to interpreting and successfully using the COT data. There is a point where the large traders, in the case of an extended uptrend, become too long, and that is usually the same point where the commercials have gotten too short. It is difficult to put a number on the net position of the large trader where that point occurs. It is different with each market and with each trend. It is pointless to curve fit what was successful in the past, because each occurrence will be unique. Extremes can often be gauged from experience by watching the interaction of price and the COT data over longs spans of data and over many markets. An aid to help determine the relative overbought or oversold level of the net position of each group is the use of a stochastic type indicator applied to the net position value. If the input parameter of the stochastic is set sufficiently large, such as back 52 weeks or longer, a relative value between the current net position level and a range of past levels can be evaluated that might not be apparent by looking at the net value by itself. In fact some markets, such as silver, rarely have a net positive position for the commercials, and using the stochastic on the net position makes determining overbought and oversold levels relative to the past range readily apparent.

In many instances the turning point of a market will occur by an extreme reading in the large trader net position, and then a sudden reversal, indicating that the funds that have been fully invested are now heading for the exits. Most often there is a corresponding opposite extreme reading in the commercial net position with a quick uptick indicating that the commercials are beginning to cover their shorts. This is always very clear in hindsight, but in real time this can be deceptive. What can look like a top in the market can be nothing more than some profit taking with the large traders eventually moving back into the market in even larger in size, with prices pushing higher that what seems logical. One can often trade the resumption of a trend by keeping close watch on the large trader net figure as viewed by the stochastic. If the uptrend is still intact, often an oversold reading of the stochastic will produce a good entry point. But it is important to keep a watch on the pattern of new highs if accompanied by lower highs, or divergences, in the large trader net position, viewed either directly from the COT large trader net position line, or by the stochastic indicator of that line. If prices continue to edge higher, but accompanied by less and less large trader participation, the uptrend may be tired and coming to a conclusion. The same analysis can be applied to the commercial net position data, but one has to look upside down, which makes the divergences more difficult to spot.

corn.png
The above chart of Corn explains some the these ideas. The red line of the large trader net position shows that group has been net long for some time, and prices had been in an uptrend. The trend stalled as the large trader was liquidating some of their position, as can be seen by the still positive but declining red line near the middle of the chart. The lowest sub-graph is a stochastic based on the large trader net position with a lookback period of one year. You can see that when the stochastic reached the lower reference line (blue ellipse drawn) that prices started the next leg of the long uptrend. On the right side of the chart you can see when the trend turned into a bubble that the large trader net position started to show signs of divergence. It was somewhat subtle looking at the actual COT data, however the stochastic displayed a very clear divergence pattern. The large traders as a group were not increasing their long position at that last high.

It seems at first glance that using the COT data is a lot of trouble for a method that seems so imprecise. There is indeed an art to interpreting this data so the information can become useful rather than confusing. It seems unlikely one could devise a mechanical system using this data since what can be gained from the information is quite subjective. But it is useful to know how the larger traders that influence price and create trends are positioned. No one indicator is perfect. One cannot build a house with just a hammer. But it would be very difficult to build a house without a hammer. The COT data is just one more element to consider, or one piece of the puzzle in determining price direction and possible turning points.

INSTRUCTIONS FOR TRADESTATION USERS:

To get started using the CFTC data in TradeStation, follow these steps:

  1. Go to cftc.gov website and find the Commitments of Traders section. In the upper right corner there is a section to download historical data. Scroll down to compressed reports and download either the complete historical file, or just select the years you want to download. Back to 2001 or so should be enough data. Choose if you want futures only, or the combined futures and options data. I use the combined reports. Also, select the Excel format, assuming you have an Excel program. Unzip the file and open it as an Excel file. Then save the file with a different name, such as COT2001, etc.
  2. On the first line of data there will be abbreviations explaining what the columns represent. I would suggest renaming the following columns as follows: A – CONTRACT, C –DATE, I – LARGE LONG, J – LARGE SHORT, L – COMM LONG, M – COMM SHORT, P – SMALL LONG, Q – SMALL SHORT. You can abbreviate these names if you want so they fit the cell size better. They are just used as a temporary reference. After renaming, then deleted all the rest of the columns. When you delete column B, then the old column C will now become column B, so that’s why it is important to rename the columns first or confusion will result. Make sure you deleted all the numerous columns to the right. I also delete most of the contracts I’m not interested in, although you can just ignore them if you prefer.
  3. Do step #2 for each year of historical data. You will then need to create a file for each contract you want to follow. Open a blank worksheet with the first historical worksheet also open. On the first line, type in each cell each of the following: Date, Open, High. Low Close, Volume, OpenInt. In other words, in cell A1 type in Date, and in B1 type in Open, etc.
  4. In the worksheet with the historical data you need to find the first contract you want to use. You will need to highlight the data to copy. Start with the date. You don’t need the contract name. Each file you save will have the name of the contract. You just need the date and the six fields following the date. Place cursor on the first date and then drag down to the last date (dates will be in reverse order on the cftc files) and then drag over to the right to the sixth data field. When you let go of the mouse the entire field should be inverted to black. Then right click on the inverted field and click on copy. Then go to the other worksheet and place the cursor under the Date cell and click on paste. You should now have all the dates in reverse order followed by six columns of data. Before going on, save this file with the contract name, or a shortened version of the name, and save as a CSV format file. Then paste in each additional year until the file is up to date.
  5. You will need to re-sort the data so it appears as the oldest date first and the most recent date at the end. Highlight all the cells after the first row. You don’t need to highlight the first row that has Date, Open, etc. Start with cell A2 and highlight all the way down to the last row, and then over to the last column to the right, which should be column G. So the entire field, with the exception of the first row with the words, should now be inverted to black. Make certain all the columns and all the rows are highlighted or the sort won’t work properly. The go up to Data, and select Sort, make sure the radio button on “header row” is checked, in the sort by box make sure “Date” is selected, and “ascending” is selected, make sure both the “then by” fields are left blank, and finally click OK. The data should now be in the proper order.
  6. Double check data to make certain there are no duplicate dates. If TradeStation finds a duplicate date it will assume the data to be intra-day and will not plot. I know the 2002 files from the CFTC have a couple of duplicate dates near the end of the year, so you will want to delete those rows. Then save the data again. You will only need to keep the CSV format files.
  7. To update these files on a weekly basis, you can either type in the date and six cells of data manually or copy and paste the entire row. If you do it manually and only have a few contracts to follow, you can click on each exchange in the current report section of the website. I follow many markets so I find it easier to copy and paste. I go down to the “other available formats” section and open the text file. I then copy the entire file and paste it into a note pad file and save it as text, which can then be imported into Excel. I then rename and delete the unwanted columns as in step #2. I then open the individual contract CSV files and copy and paste the current weeks data (date and six fields) onto each worksheet, and of course re-save. It will ask you if you want to save in Excel format as well, which I don’t.
  8. To get data to display in TradeStation can be a bit tricky. I first display the weekly price data on the contract, and then I insert a second data field. You need to plot 3rd party data. The first file you plot will create an attribute file. It will ask a few questions. Answer the best you can. The most important is to change the display to “As Is” from the default of 1/100. It will also ask if you want to use the same setting with all files in the same directory. Answer yes to that if you have all the COT files in the same directory, which is the best way. If you use other 3rd party data, just have those in a different directory. If the data fails to load, check again for duplicate dates, and to make certain all the header columns are spelled correctly, such as Date, Open, OpenInt, etc.
  9. When you create indicators keep in mind that the Open field is actually the Large Trader Long position, and the High becomes the Large Trader Short, and Low is the Commercial Long, etc. To get the net position of each group simply write an indicator that plots Open – High, Low – Close, and Volume – OpenInt, and make certain when you plot this you do so on Data 2.

 

7 thoughts on “Commitments of Traders – Is it a Useful Tool to Time Markets?

  1. Good Morning Doug,
    Oh my gosh! For me way too much information on market timing. By the time I got near the end where you were talking about corn I had forgotten what the topic was about. When my eyes can refresh I am going to read it again. When my brain circut breakers have been reset I will read it again for a third time and try to figure how to incorporate this into Tradestation. Until then I must rest, TMI. Phew!
    Best of trading to you,
    Bert

  2. Bert,
    It takes work and focus to trade successfully. Most traders are lazy and want someone else to tell them what to do, either via a broker, chat room, trading system, indicator, etc. Most traders lose money. You need every edge you can get. It takes work.
    Doug

  3. Now Boys, don’t fight.
    But, I would mention that in 2005 the esteemed Larry Williams published a book, “Trade Stocks and commodities with the Insiders, Secrets of the COT report” related to and very interestingly about this very subject. And, while I normally avoid anything with the words “secrets of” in its title, this was a very focused and interesting book. Also, as an aside I note that Mr. Williams addresss at the time of writing was St. Croix, US Virgin Islands. Not bad for an author wouldn’t you say.
    Yours was a refreshing and informative article and thank you for bringing the subject to our attention.

  4. Thanks for posting this Doug!

    I also use the COT reports but I use the ready-made graphs available at –
    http://www.timingcharts.com/

    Your mention of the use of stochastics to detect trend changes in the COT reports is very useful.

    I personally did not find this material to be too deep or complex. In fact, I would prefer to see a deeper analysis, but I won’t ask for it. Riding the coattails of the big market players is extremely important for understanding the overall trends. This is the type of technical analysis that most trading groups should be looking at, up close and in detail.

    In the trading world, it’s fairly common knowledge that the COT is a treasure trove of information. Thanks for sharing your thoughts!

  5. Thanks for the plugin to TimingCharts.com. The reason Commercials are opposite to the price chart, is that the producers are marketing product (short positions), as the price increases. Whereas the Commercial Consumer will increase long positions as the price declines. There are a variety of methods to interpret the information.
    1) COT Index
    2) Hedger’s A-B-C-D Market Phases
    4) COT Index (4 week) of Commercials & Small Speculators
    3) Trend Weaver, etc.

    For a complete description, please visit the Forum at TimingCharts.com

  6. I have noticed that the large and commercial are on opposite sides from themselves on the S&P vs ES. That is if large are net long on the S&P, then large are net short on the ES. Ditto with the commercials.
    Does anyone have insight on that ?

  7. Hi Doug,
    Read your article on the CTFC data. A lot of informtion. However, Dean says we can’t fight. Doug, I will never fight with you, LOL, because you gave me the TuckerCCI and the TuckerCycle. OMG! What a pair of fantastic indicators. With those two and the TwiggMFAlt (not your creation of course) for confirmation, I have become a successful trader. One can’t fight with success! Thank you Doug for having the understanding to know there was a need, the knowledge to be able to create them, and the wisdom to test and fine-tune them into great indicators. And above all, with the love you have for us “poor” traders, you are so willingly to share, you allowed me to try them. Wow, what a gift!
    Gratefully yours,
    Bert

Comments are closed.