Other Data Providers
In the previous chapters, we introduced many ways to get financial data that researchers regularly use. We showed how to load data from Yahoo Finance and commonly used file types, such as comma-separated or Excel files. We relied on the tidyfinance package for downloading and cleaning data from various sources and on fmpapi for financial statements. Then, we introduced remotely connecting to WRDS and downloading data from there. However, this is only a subset of the vast amounts of data available these days.
In this short chapter, we aim to provide an overview of common alternative data providers for which direct access via dedicated packages exists. Such a list requires constant adjustments because both data providers and access methods change. However, we want to emphasize two main insights: First, the number of packages that provide access to (financial) data is large. Too large actually to survey here exhaustively. Instead, we can only cover the tip of the iceberg. Second, both R and Python provide the functionalities to access basically any form of files or data available online. Thus, even if a desired data source does not come with a well-established package, chances are high that data can be retrieved by establishing your own API connection or by scraping the content.
In our non-exhaustive list below, we restrict ourselves to listing data sources accessed through easy-to-use packages.
For further inspiration on potential data sources, we recommend reading the R task view empirical finance. Further inspiration (on more general social sciences) can be found here.
For further inspiration on potential data sources, we recommend reading the Awesome Quant curated list of insanely awesome libraries, packages, and resources for Quants (Quantitative Finance). In fact, the pandas-datareader package provides comprehensive access to a lot of databases, including some of those listed below.
Also, the requests library in Python provides a versatile and direct way to interact with APIs (Application Programming Interfaces) offered by various financial data providers. The package simplifies the process of making HTTP requests, handling authentication, and parsing the received data.
Apart from the list below, we want to advertise some amazing data compiled by others. First, there is Open Source Asset Pricing related to Chen and Zimmermann (2022). They provide return data for over 200 trading strategies with different time periods and specifications. The authors also provide signals and explanations of the factor construction. Moreover, in the same spirit, Global factor data provides the data related to Jensen et al. (n.d.). They provide return data for characteristic-managed portfolios from around the world. The database includes factors for 153 characteristics in 13 themes, using data from 93 countries.
If you feel that we miss a fantastic financial data source, please get in touch via [email protected] - thank you very much for your support!
The data sources differ across languages because the available packages differ. We hence list them separately for R and Python.
| Source | Description | R packages |
|---|---|---|
| Macroeconomic Variables | ||
| FED | The Federal Reserve Bank of St. Louis provides more than 818,000 US and international time series from 109 sources via the API FRED. The data is freely available and can be browsed online on the FRED homepage. | fredr (Boysel and Vaughan 2021) and alfred (Kleen 2021) |
| ECB | The European Central Bank’s Data Portal (which replaced the former Statistical Data Warehouse) provides data on Euro area monetary policy, financial stability, and other topics relevant to the activities of the ECB and the European System of Central Banks (ESCB). | ecb (Persson 2021) |
| Financial data | ||
| Bloomberg | Bloomberg’s Fundamental coverage includes current and normalized historical data for the balance sheet, income statement, cash flows statement, and financial ratios. Additionally, it provides industry-specific data for communications, consumer, energy, health care, and many more. In order to retrieve Bloomberg data, a paid subscription is needed. | Rblpapi (Armstrong et al. 2022) |
| LSEG Eikon | Eikon (now part of LSEG, formerly Refinitiv) provides access to real-time market data, news, fundamental data, analytics, trading, and messaging tools. Eikon is a paid service. Apart from the CRAN version, there is also https://github.com/philaris/eikonapir. |
DatastreamDSWS2R (Cara 2021) and eikonapir |
| Nasdaq Data Link (Quandl) | Quandl is a publisher of alternative data. Quandl publishes free data, scraped from many different sources from the web. However, some of the data requires specific subscriptions on the Quandl platform. | Quandl (McTaggart et al. 2021) |
| Simfin | Simfin makes fundamental financial data freely available to private investors, researchers, and students. The data provider applies automating data collection processes to collect a large set of publicly available information from firms’ financial statements. | simfinapi (Gomolka 2021) |
| High-frequency data | ||
| IEX | The IEX Group operates the Investors Exchange (IEX), a stock exchange for US equities. IEX offers US reference and market data including end-of-day and intraday pricing data. IEX offers an API which is freely available. | Riex (Ibrahim 2021) |
| TAQ | TAQ data provides subscribed users access to all trades and quotes for all issues traded on NYSE, NASDAQ, and the regional exchanges. TAQ data can be accessed from WRDS via Postgres. The highfrequency package delivers useful workflows to clean TAQ data. |
highfrequency (Boudt et al. 2022) |
| Other (free) data | ||
| CoinMarketCap | The data provider CoinMarketCap provides cryptocurrency information and historical prices, as well as information on the exchanges they are listed on. | crypto2 (Stoeckl 2022) |
| CoinGecko | CoinGecko is an alternative crypto data provider of current and historical data on a myriad of coins and exchanges. | geckor (Mastitsky 2021) |
| X (Twitter) | X (formerly Twitter) provides API access to extract and analyze posts. Note that access has become considerably more restricted and largely paid since the platform’s rebranding. | rtweet (Kearney 2019) |
| SEC company fillings | The EDGAR database provides free public access to corporate information, allowing you to research a public company’s financial information and operations by reviewing the filings the company makes with the SEC. You can also research information provided by mutual funds (including money market funds), exchange-traded funds (ETFs), and variable annuities. | edgarWebR (Waldstein 2021) |
| Google trends | Google offers public access to global search volumes through its search engine through the Google Trends portal. | globaltrends (Puhr and Müllner 2021) and gtrends (Massicotte and Eddelbuettel 2022) |
| Source | Description | Packages |
|---|---|---|
| FRED | The Federal Reserve Bank of St. Louis provides more than 818,000 US and international time series from 109 sources via the FRED API. Data can be browsed online on the FRED homepage. | fredapi, pandas-datareader |
| ECB | The European Central Bank’s Statistical Data Warehouse offers data on Euro area monetary policy, financial stability, and more. | sdmx |
| OECD | The OECD monitors trends, collects data, and forecasts economic development in various public policy areas. It’s a major source of comparable statistical, economic, and social data. | requests, pandas-datareader |
| World Bank | Provides access to global statistics including World Development Indicators, International Debt Statistics, and more, available for over 200 countries. | requests, pandas-datareader |
| Eurostat | Eurostat, the EU’s statistical office, provides high-quality data on Europe. | requests, pandas-datareader |
| Econdb | Offers access to economic data from over 90 official statistical agencies through a comprehensive database. | requests, pandas-datareader |
| Bloomberg | Provides data on balance sheets, income statements, cash flows, and more, with industry-specific data in various sectors. Paid subscription required. | blpapi |
| LSEG Eikon | Eikon offers real-time market data, news, analytics, and more. It’s a paid service of LSEG. | refinitiv-data |
| Nasdaq Data Link | Quandl publishes alternative data, with some requiring specific subscriptions. | nasdaqdatalink, pandas-datareader |
| Simfin | Automates data collection to offer financial data freely to investors, researchers, and students. | simfin |
| PyAnomaly | A Python library for asset pricing research, offering various portfolio analytics tools. | pyanomaly |
| IEX | Operates the IEX, offering US reference and market data, including intraday pricing data. | pyEx, pandas-datareader |
| CoinMarketCap | Offers cryptocurrency information, historical prices, and exchange listings. | coinmarketcap |
| CoinGecko | An alternative crypto data provider for current and historical coin and exchange data. | pycoingecko |
| X (Twitter) | Offers limited access for academic research on Tweets. | tweepy |
| SEC company filings | The EDGAR database provides public access to corporate and mutual fund information filed with the SEC. | sec-api |
| Google Trends | Provides public access to global search volumes via Google Trends. | pytrends |
Key Takeaways
- Access a wide range of financial and macroeconomic data through R and Python packages tailored to specific providers.
- Even if a specific package doesn’t exist, R and Python can often retrieve data through APIs or web scraping methods.
Exercises
- Select one of the data sources in the table above and retrieve some data. Browse the homepage of the data provider or the package documentation to find inspiration on which type of data is available to you and how to download the data into your session.
- Generate summary statistics of the data you retrieved and provide some useful visualization. The possibilities are endless: Maybe there is some interesting economic event you want to analyze, such as stock market responses to Twitter activity.