Download Exploring the relationship between Google Trends data and ... and more Slides Finance in PDF only on Docsity!
Exploring the relationship between Google Trends data and
stock price data.
Author: Dartanyon Shivers, Advisor: Chris Deotte PhD, Self Published: June 2017 , UCSD Abstract: In this work, we provide a brief description of the stock market and search engines, more specifically Google’s search engine. We then suggest some efficient methods to gathering historical stock price data and google search data. Additionally, we propose using a test that we created to explore the relationship, if any, of stock prices and the popularity of google searches. Finally, we share our results from the test and discuss the possibility of using the popularity of google searches to predict future stock price movement.
1. Introduction 1.1 Stock Market The stock market is a global network in which individuals can purchase ownership, commonly known as shares, of a public company. All companies start out as privately owned companies; they do not become public until they are listed under a stock exchange. Exchanges are organized markets in which financial instruments are bought and sold. These financial instruments can be broken up into two categories, equity-based and debt- based. Equity-based financial instruments represent ownership of an asset, and debt-based financial instruments represent a loan made by an investor to the owner of that asset. Both of these categories contain different types of instruments, however for the purpose of this paper we will focus on stocks, which are shares in ownership of a company and they are sold on stock exchanges. The collection of stock exchanges is what makes up the stock market. The United States main exchanges are the New York Stock Exchange and National Association of Securities Dealers Automated Quotations or commonly known as NYSE and NASDAQ respectively. Not just any company can be listed under these exchanges, they must meet certain requirements such as possessing a specified number of shareholders and an evaluation above some minimum worth. All businesses are considered private until they decide to “go public” and be listed under an exchange. Business owners choose to “go public” for one reason, to raise money. This could be to raise money for themselves, however in most cases they do this to use that money to grow their business, via purchasing new equipment, developing better products, expanding operations, etc. Whatever the reason, once they make this decision, shares of their business become available for anyone in the world to purchase. Before becoming listed, banks and other financial firms come together to evaluate the company’s worth. Once these financial institutions and the company agree on the number of shares to be listed and their worth, an initial public offering, IPO, is held. During an IPO, the shares of the company are all sold for the exact price that was agreed upon. The money made from the sale goes to the company, and individuals who purchased stock in the company can now call themselves a “shareholder” of that company. Although shareholders have some sense of ownership of the company, they do not own any property of the company nor do they have much control over it. Being a shareholder provides voting privileges and rights to their percentage of the company’s worth if it were to be liquidated. Shareholders of the company are unable to make decisions on the company’s behalf and they are not entitled to the equipment, products, or anything else that makes up the business. The tradeoff for this is that shareholders relinquish all liability to the company itself. Under the law, a corporation is treated as a legal person, in the sense that it can borrow money, own property, be sued and file taxes. This is beneficial to the investor because if a company is to go bankrupt or be sued, that investor’s personal assets outside of the company are not at risk. So if individuals have little to no authority within the company nor are they able to take what they believe is their fair share from the company, why would anyone even consider investing? Well, stocks have proven to be the best investments over time. Investing beats placing your money into a savings account because due to inflation you are essentially losing money in the long run. Inflation is the general increase in prices and decrease of purchasing power of money. So, if you were to place $1,000 into a savings account today and not touch it in 10 years, your $1,000 in the future will not be able to buy you the same things you could have bought 10 years
prior. It is true that most banks pay some form of interest if individuals leave their money in their savings account, however that interest is generally around .01% per year. This is problematic because inflation rises at an average rate of 3% per year. This means that you’re essentially losing 2.99% of your money’s purchasing power each year. Hence investing has proven to be superior to saving, and since buying stocks have proven to be the best form of investing over time, it’s no mystery to why so many individuals have embraced the stock market. There are two ways in which investors make money from the stock market. The first is via dividends payed to shareholders. When businesses are flourishing, they will often take some of the profit and award that money to its shareholders. This form of payment is referred to as a dividend and they are generally disbursed on a quarter- annual basis. Dividends are typically not substantial, however if you own enough shares of a company your dividend payment could be appreciable. The other way investors make money from the stock market is by selling a stock for more than they had originally paid for it. After a company’s IPO, their stock is then sold for whatever price an individual is willing to sell or buy the stock. The U.S. stock market only operates from 9:30am-4pm EST, and is closed on the holidays. Once the market opens, individuals could place their orders on any stock in the market, provided they have a brokerage account. A brokerage account is an arrangement between an investor and a licensed brokerage firm that allows the investor to deposit funds with the firm and place investment orders through the brokerage. Through the brokerage, two primary orders can be made, market and limit orders. Limit orders make up a queueing system in which people can propose the price that they would like to buy or sell a stock. This queueing system is known as the bid-ask spread. A very basic example of a bid-ask spread is provided below. A market order is a buy or sell order to be executed immediately at the current market prices. So if we placed a market order to buy 5 shares we would need at least 5 * $90.22 = $451.10 in our brokerage account to make the purchase. Now if we wanted to sell 10 shares immediately, we would place a market order to sell 10 shares and we would receive 10 * $90.21 = $902.10 in our brokerage account. Now, there are other factors that go into the buy and sell process such as taxes from the government and fees from whatever financial firm you chose to open a brokerage account through, however these details are encouraged for the reader to explore. Notice that after a company’s IPO, the price of a stock is essentially in the hands of the public. Generally, the price at which the stocks are being bought and sold are relatively reasonable, meaning they are selling at about a company’s actual worth. However now and again, a company’s stock price will greatly precede or proceed its true value. This typically happens when there is a high demand for the stock, and this demand could be to buy or sell. History has proven that investors have not always behaved in the most logical manner. Since the stock market is complex, individuals are quick to listen to anyone who calls themselves an expert in the field of finance. These experts have their opinions of what they believe will happen with companies, however no one ever really knows for sure what is to come. Businesses are constantly changing, and there are too many factors that go into a business’s success or failure, most of which are unpredictable. The exponential growth of technology and constant existence of competition ensure that no business is safe, and all it takes is one brilliant
miss out on “the next big thing.” Those companies made no money and soon had to file for bankruptcy, leaving investors to lose everything. There are other examples of situations like this that go to show that competency in business and finance does not imply you can always predict which direction a stock’s price will move. This erratic behavior in the stock market has intrigued the world, and many individuals have come up with some fascinating experiments to try and predict the movement of stock prices. One interesting idea was to use news articles to predict stock price movements. An algorithm was created to search for specific words within articles about a company and provide a score on the positive or negative tone the articles had towards that company. That score was then analyzed with the company’s stock prices to determine if the articles could have possibly had any effect on its movement. After hearing about this experiment we thought about how individuals were likely finding these articles. The probable answer to this question is via a search engine. Soon after we found out that the most popular search engine, Google Search, tracks and stores its users searches through their program called Google Trends. This then motivated us to explore the relationship, if any, between Google Trends data and stock price data. If we could find such a relationship, could we then use Google Trends to inform us when to buy and sell stocks for a profit? Once having gathered the data, we needed to use some test to measure if stock prices and popularity of searches correlated in any way. Our main goal was to find stocks and searches that when their data was plotted, the two graphs would be similar. Initially we tried to use a built in MatLab function called corrcoef(). This function takes in two vectors of equal dimension and spits out the Pearson correlation coefficient, say c , which is a value between - 1 and 1. The closer c is to 1 or - 1 the more linearly dependent the two vectors are said to be. We hoped that if we could find “good” c values, the graphs of the data would be similar. However, this is not how it turned out. Below are the graphs of the popularity of the search “alarm” and the stock prices of Amazon.com Inc, with ticker symbol “AMZN”, over a five year time period. The two had a correlation coefficient value of .8813. As you can see, these two graphs don’t rise and fall together month to month. They only have a common overall trend. We preferred a test that would detect month to month changes, but this correlation test removes time and does not recognize localized ups and downs. Even if we removed the overall linear trends, corrcoef() would still be heavily influenced by long scale trends. For example, if both data sets started low, rose in the middle and ended low, corrcoef() would report a strong correlation regardless of month-to-month correlation. In addition
the test only considered the graphs to be similar if the overall magnitude of the peaks and valleys were proportional. The following image is an example of this. This pair of vectors inputted into corrcoef() returned a value of .5570, which is an average result. However, we considered graphs like these to be perfect because we wanted graphs that had local extrema at the same points in time so we could use that information as buying and selling signals. Since our stock price data is in a way unbounded above, the prices could, in theory, go to infinity, however the popularity of the searches was bounded above by 100. This means that it is likely that the peaks and valleys would not be proportional as a whole, which is what the Pearson correlation coefficient is essentially calculating. These issue urged us to seek out another method in which we could test the data sets for a correlation. In our search we failed to find something that would be promising, so we decided to create our own test that would aid us in tracking down stocks and searches that would have similar graphs. The rest of the paper is structured as follows: in Section 2, we reveal the sources in which we obtained our data and how we went about gathering it; Section 3, we discuss our correlation test; Section 4, we demonstrate the test’s effectiveness.; Section 5, we present our test results comparing Google Trends with stock prices; and finally in Section 6, we disclose our conclusion on our hypothesis and close with a few final remarks.
2. Data 2.1 Stock Price Data Yahoo! Finance is a media property that is part of Yahoo!’s network. This site provides financial news, data and commentary including stock quotes, press releases, and financial reports. One service Yahoo! Finance offers is the ability to download any company’s stock price history over any length of time beyond 1962. Below is an image of the Yahoo! Finance page after searching for Coca-Cola Co stock, using its ticker symbol KO. A ticker symbol is an abbreviation used to uniquely identify publicly traded shares of a particular stock on a particular stock market.
The yellow column represents the closing price of the stock for that day; this was the only data we needed from this table. When we first began testing our data and examining the graphs we had noticed that some of the graphs for stocks had some drastic pitfalls. An example of this is provided below.
After a brief investigation we had realized that we weren’t taking stock splits into consideration. A stock split is a decision made by a company to increase its total number of shares by issuing more shares to its current shareholders. One reason for doing this is that the company’s stock price has increased to levels that are either too high or beyond the price levels of similar companies in their sector. When a company decides to split its stock the shareholders are not negatively affected, they retain the same portion of ownership and the amount of money invested in the company also remains the same. We readjusted our data manually by looking up the company’s stock split history and multiplied the data from the stock split day forward by the split ratio. The images below demonstrates this process. Image (1) displays the stock price history of AAPL around the time of its 7:1 stock split in 2014. We originally used the data tables, image (2), from stocksplithistory.com to identify stock splits of each company during our five year time period. Since the stock split was 7 for 1 we multiplied all data beyond the stock split data, 6/9/2014, by 7. Later we realized that Yahoo! Finance also provides stock split history, image (3), which we could utilize in the future to automate this process. An alternative to this whole process is to take the “Adj Close” column from the csv file that is highlighted in blue from our previous csv file image. We learned after putting in all this work that the “Adj Close” column takes into account stock splits, however the prices become relative to the latest stock split in the data. This means that if I took the stock price data of the last two weeks, and there was a 2:1 stock split last week, the prices of the week prior would be divided by two as well. Our research required us to gather historical price data on many stocks, and going through this process proved to be very timely. So we sought some way to automate this process. We noticed that the “download data” link allowed us to copy its link address (Blue arrow from Yahoo! Finance image). Here is a picture of two link
Using this function we created a program that would prompt the user to type a company’s ticker symbol and would then retrieve the stock price history of that company over a five year time period, 1/1/12 to 12/31/16, on a weekly basis. The program would continue to ask the user for ticker symbols until a blank string was entered, which would then end the program. As the ticker symbols were entered, our program would merge all stocks with their data into a single excel document. In our research we gathered data from over 200 stocks; 48 stocks from Vanguard’s ETF selection, 75 from Standard and Poor’s S&P400 Midcap index, and 103 Standard and Poor’s S&P500 index. Below is a list of all the stocks ticker symbols from each category. 2.2 Google Trends Data As previously mentioned, Google Trends is a database that shows how often a particular search-term is entered relative to the total search-volume across various regions of the world. Google Trends data is a random sample of Google search data. Therefore, it is not a direct reflection of what is being searched, however it is a pretty good approximation. Data that is excluded are searches made by very few people and repeated searches from the same person over a short period of time. Google Trends also filters out apostrophes and other special characters in a search. The samplings are taken from a specified geographical region over a designated period of time determined by the user. The following is an image of the Google Trends site after searching “baseball” over the five year time period 1/1/12-12/31/16.
The red arrow points to the search term. The yellow and blue arrows point to parameters in which you could adjust to observe that search terms popularity over a specified region and time. The graph just under those arrows is a plot of the popularity of the search over the given time period. Notice that the max for the data is
- This is because each data point is divided by the total searches of the geography and time range it represents and those resulting numbers are scaled on a range of 0 to 100 based on the proportionality of its popularity over that time period. The data could then be collected by following the green arrows:
- Click the gray arrow in the right direction and the box with the following three options will appear.
- Click the “download CSV” option.
- A csv file with the data will be downloaded to the default download folder for your web browser. Unlike the data from Yahoo! Finance, Google Trends makes it difficult to automate the download of data with scripts. Therefore, we downloaded the search data’s CSV files into a targeted folder, and then ran a program that read all the files, extracted the data, and merged that data into a single CSV file. Below is a portion of our program that combined the data into one file.
however, it is difficult to identify how they have coded the dates. After doing some research, we discovered that this was not the first time that Yahoo! Finance has updated its URL links. It is likely that Yahoo! Finance does not want computers gathering massive amounts of this data at once because they need actual users to visit their site for the advertisements on the page. In spite of the update impeding our program from running with the original URL, we were able to make a small tweak to the old URL that would then work. Notice the red arrows in the above image; the new URLs have an “s” at the end of “http” while the old URL does not. By adding that “s” to our old URL, our program worked just as it did before. We are unsure of how long the URL will work, however we can say that for now we have a sort of “back door” for gathering this data off of Yahoo! Finance. Although this “back door” exists for now, we don’t expect it to be there much longer. Therefore, data automation via this site may still be possible, however it will likely require more cleverness in constructing a URL that will actually work. 2.3.2 Google Search Data As mentioned before, when downloading the Google Trends data, it would automatically be downloaded to whatever the default download location was for your browser. Instead of trying to access this default location through a series of directory commands in MatLab, we changed our web browser’s download location to a folder located in the same folder as our gettrends() program. The web browser we used was Google Chrome and below is an image with directions on how to achieve this task.
- Click the button of the firs red arrow
- Select the “Settings” option (Second Red Arrow)
- Scroll down until you see a blue link titled “Advanced settings” and click on it
- Scroll down until you see “Downloads” (Green Box)
- Click the “Change” button (Green Arrow)
- Select your new download location 3. Correlation Test Since both sets of data are a series of points indexed in time order, we will refer to, from this point forward, each subset of data, i.e. a single stock’s price history or a search’s “popularity” over time, as a time series. 3.1 Description When we originally began testing our data, we used the MatLab function corrcoef(). This test was not very effective in providing us with time series pairs that would likely have matching graphs because it essentially did
not recognize ups and downs. Therefore we created our own test that simply counted matching ups and downs of the two time series. For example, when one time series increases in value, does the other increase? When one decreases does the other decrease? Our hope was that time series with high percentages of matching ups and downs would imply matching graphs. 3.1.1 Mathematical Description of our test: Let {𝑥! (!) } be our original time series for 1 ≤ 𝑘 ≤ 𝑛 and 𝑗 = 1 , 2. The number k in 1 ≤ 𝑘 ≤ 𝑛, denotes 5 years of weekly data with 𝑛 = 261 and the number 𝑗 = 1 , 2 denotes the two sequences we are comparing. Our new time series is 𝑠! ! 𝑥!^ =
(!)
𝑥! (!) 0 𝑖𝑓 𝑥!!! (!) = 𝑥! (!) − 1 𝑖𝑓 𝑥!!! ! < 𝑥! (!) for 1 ≤ 𝑘 ≤ 𝑛 − 1. Our test denoted by T returns a number between 0 and 1 inclusive, 𝑇 𝑥!^ , 𝑥!^ , 𝑑 ∈ 0 , 1. The number d is a time shift. The test converts the sequences 𝑥(!)^ 𝑎𝑛𝑑 𝑥(!)^ into sequences 𝑠(!)^ 𝑎𝑛𝑑 𝑠(!)^ as describe above and then compares them. When comparing the two sequences, we compare element 𝑠! (!) with element 𝑠!!! (!) for 1 ≤ 𝑘 ≤ 𝑛 − 1 − 𝑑. The test T reports what percentage of elements from sequence 𝑠(!)^ are the same as 𝑠(!)^ with time shift d. If 𝑑 < 0 , we compare element 𝑠!!! (!) with element 𝑠!!! (!) for 1 ≤ 𝑘 ≤ 𝑛 − 1 + 𝑑. Below is a plot of two time series:
- x = [3 4 2 5 5 8 6 2 2 5 7]
- y = [5 5 3 8 8 9 4 5 5 7 8] The green and red arrows represent matching and mismatching directions of the graph respectively. Since there are 11 data points there are 10 comparisons, and 8 of those 10 comparisons matched which implies an 8 0% similarity according to our test. The two graphs somewhat look similar, however the jaggedness of the graphs makes it more difficult to see this. With our data, this issue was even more so the case as you can see below.
As you can see the graphs started looking much more similar after five iterations of smoothing. Applying these two conditioning functions to the data allowed us to capture the clearer patterns of the data and aided our efforts in identifying sets that are behaving similarly. The following two images are portions of our smoothing and flattening functions respectively. In both cases we rescaled the y-axis to values between 10 and - 10 to make it easier to compare the graphs of the two data sets.
3.2 Statistical Significance Statistical significance is the likelihood that a relationship between two or more variables is caused by something other than random chance. When developing this test we had to consider the possibility of a random search and stock having a high correlation value according to our test. The similarity of our two sequences is a binomial distribution. A binomial distribution is the probability of a specific number of successes or failures in an experiment that is surveyed multiple times. Since we were simply answering yes or no to whether the two data sets were similar in sign for the difference of their points on an interval our probability value is .5. Our test is essentially equivalent to flipping two coins n times and measuring the number of times that they land on the same side. Using the binomial distribution model we could calculate the probability of our results randomly occurring. The less probable a result is, the more statistically significant it is. 3.2.1 Mathematical explanation of statistical significance of our test (without smoothing): If the sequences 𝑥(!)^ 𝑎𝑛𝑑 𝑥(!)^ contain random data with 𝑥!!! (!) ≠ 𝑥! (!) then the similarity of sequences 𝑠(!)^ 𝑎𝑛𝑑 𝑠(!)^ is a binomial distribution with 𝑝 =. 5. Therefore, we expect 𝑇 𝑥!^ , 𝑥!^ , 𝑑 =. 5 with standard deviation 𝜎 = ! !!!!|!|
. 25 (𝑛 − 1 − |𝑑|). For the case of 𝑛 = 261 and 𝑑 = 0 , the standard deviation is 𝜎 =. 031. Therefore given two random sequences 𝑥(!)^ 𝑎𝑛𝑑 𝑥(!)^ the probability of 𝑇 𝑥!^ , 𝑥!^ , 0 ≥. 5 + z𝜎 is p = 0.05, 0.025, 0.01, 0.005, 0.001, and 0.0005 for z = 1.645, 1.960, 2.327, 2.576, 3.091, 3.291 respectively. For example, there is only a 1% chance that 𝑇 𝑥!^ , 𝑥!^ , 0 ≥. 5 + 2. 327 𝜎 =
We confirmed all of these values by generating 1,000 random fake stock price data series and empirically calculating these values. Additionally we confirmed that actual stock prices appear to be random data because
To demonstrate the effectiveness of our test we have provided examples of a pair of searches and a pair of stocks that our program calculated to have a strong correlation and which we would expect this to be the case. 4.1 Google Trends versus Google Trends Using smoothing iterations = 5, shift = 0, and flip = 1, our test determined that the searches “hotels” and “cheap flights” correlated with a value of T = .842. This is statistically significant with the chances of this randomly occurring being less than one in one billion (using z-score = 6 where actual z-score = 6.61). We would expect this pair to correlate well. Although the magnitude of the hills are not the same, it is incredible that the peaks and troughs of the graphs are all at the same points in time. This means that “hotels” and “cheap flights” are at their lowest and highest points of being searched at the same time. 4.2 Stock prices versus stock prices Using smoothing iterations = 5, shift = 0, and flip = 1, our test determined that the stock prices of the companies Exxon Mobil Corporation and Chevron Corporation correlated with a value of T = .815. This is statistically significant with the chances of this randomly occurring being less than one in one billion (using z-score = 6 where actual z-score = 6.09). Both of these companies are in the oil industry, which makes sense of why these two would be correlated. The following image illustrates this high correlation. Exxon and Chevron have ticker symbols XOM and CVX respectively.
The two companies’ graphs of stock prices are nearly identical. Much of their peaks and valleys share the same magnitude which means they are almost perfectly correlated. This is impressive since this data reflects a five year time span, and begs the question “Can we use one stocks price movement to predict another?” Here are tables of some other stocks that correlated well with one another according to our test.
