Reliable security master data (stock code, company name, listing date and industry) is the starting point of almost every empirical study of the Taiwanese equity market. This report collects that information for Taiwan Stock Exchange (TWSE) listed common stocks from two independent providers: the exchange’s official ISIN registry and the FinMind data platform. The two datasets are cleaned, validated and then compared using the stock code as the key.
The assignment brief contains two tasks:
strMode value is 2
for listed securities.Interpretation notes:
strMode=2 page is the complete listed
(TWSE) board. It contains many security types (stocks, preferred shares,
ETFs, depositary receipts, beneficiary securities, warrants and so on),
so a filtering step is required even though the URL already refers to
listed securities.| Source | Role in this report | Access method |
|---|---|---|
| TWSE ISIN registry (https://isin.twse.com.tw/isin/C_public.jsp?strMode=2) | Official, authoritative list of listed securities with ISIN, listing date, market, industry and CFI classification code. | HTML table read with rvest (the page is Big5-encoded).
The cleaned result is stored in
data_cache/twse_clean_saved.csv. |
| FinMind (https://finmindtrade.com/) | Third-party financial data platform with a REST API. Used as an
independent source of stock information (TaiwanStockInfo
dataset). |
JSON API via httr2. The raw response is stored in
data_cache/finmind_raw.csv. |
TWSE is the reference record. FinMind is an aggregator, so differences in naming, industry labels or coverage are expected and are analysed rather than treated as errors.
To keep the report reproducible and fast, local files are
used whenever they are valid; the network is only used when a
required file is missing or when refresh: true is set in
the YAML header. The raw and the cleaned versions of the data are kept
strictly separate (Sections 5 and 6).
Install the packages once (only the first time) by running this line
in the R console. httr2, rvest,
xml2 and jsonlite are only needed when a
download or an HTML parse is actually performed.
install.packages(c("dplyr", "stringr", "stringi", "readr", "tibble", "knitr", "rmarkdown",
"httr2", "rvest", "xml2", "jsonlite"))core_pkgs <- c("dplyr", "stringr", "stringi", "readr", "tibble", "knitr", "rmarkdown")
missing_core <- core_pkgs[!vapply(core_pkgs, requireNamespace, logical(1), quietly = TRUE)]
if (length(missing_core) > 0) {
stop("Please install the missing package(s) first: ",
paste(missing_core, collapse = ", "), call. = FALSE)
}
invisible(lapply(core_pkgs, library, character.only = TRUE))
# Packages for the network / HTML steps are loaded only when those steps run.
use_pkgs <- function(pkgs) {
miss <- pkgs[!vapply(pkgs, requireNamespace, logical(1), quietly = TRUE)]
if (length(miss) > 0) {
stop("This step needs the package(s): ", paste(miss, collapse = ", "),
". Install them with install.packages().", call. = FALSE)
}
invisible(lapply(pkgs, library, character.only = TRUE))
}
# ---- Paths (all relative to the detected project root) ------------------------
proj_path <- function(...) file.path(project_root, ...)
cache_dir <- proj_path("data_cache")
dir.create(cache_dir, showWarnings = FALSE, recursive = TRUE)
path_twse_clean_saved <- file.path(cache_dir, "twse_clean_saved.csv") # cleaned TWSE common stocks
path_twse_raw <- file.path(cache_dir, "twse_raw.csv") # optional: all categories
path_fm_raw <- file.path(cache_dir, "finmind_raw.csv") # raw FinMind response
path_fm_clean_saved <- file.path(cache_dir, "finmind_clean_saved.csv") # earlier result (cross-check only)
path_manual_html <- c(proj_path("twse_isin_strMode2.html"),
file.path(cache_dir, "twse_isin_strMode2.html")) # optional manual fallback
out_twse <- proj_path("TWSE_Common_Stocks.csv")
out_finmind <- proj_path("FinMind_Common_Stocks.csv")
out_compare <- proj_path("TWSE_FinMind_Comparison.csv")
out_unmatched <- proj_path("Unmatched_Stocks.csv")
# ---- Settings ------------------------------------------------------------------
twse_urls <- c("https://isin.twse.com.tw/isin/C_public.jsp?strMode=2",
"http://isin.twse.com.tw/isin/C_public.jsp?strMode=2") # second = fallback
finmind_url <- "https://api.finmindtrade.com/api/v4/data"
# TRUE = ignore local files and download again; FALSE = use valid local files
refresh <- if (exists("params")) isTRUE(params$refresh) else FALSE
# Chinese labels used by the TWSE ISIN page (kept in one place)
CAT_STOCK <- "股票" # category title "stocks" (common and preferred shares)
MARKET_LISTED <- "上市" # market value "listed"
INNOV_PATTERN <- "創新[板版]" # Taiwan Innovation Board label (two spellings)
# ---- Small helpers ---------------------------------------------------------------
read_chr_csv <- function(path) {
readr::read_csv(path, col_types = readr::cols(.default = readr::col_character()),
na = c("", "NA"), trim_ws = TRUE, show_col_types = FALSE, progress = FALSE)
}
# First candidate name found in a vector of column names (case-insensitive)
pick_col <- function(nms, candidates) {
i <- match(tolower(candidates), tolower(nms))
i <- i[!is.na(i)]
if (length(i) > 0) nms[i[1]] else NA_character_
}
# Dates are accepted as 2003/06/30 or 2003-06-30
parse_date_multi <- function(x) {
x <- str_squish(x)
coalesce(as.Date(x, format = "%Y/%m/%d"), as.Date(x, format = "%Y-%m-%d"))
}
# Automatic table numbering
tab_cap <- local({ i <- 0L; function(caption) { i <<- i + 1L; paste0("Table ", i, ". ", caption) } })
# Table, or a short note when there is nothing to show
show_tbl <- function(df, caption, empty_note = "No rows to display.") {
if (is.null(df) || nrow(df) == 0) {
knitr::asis_output(paste0("*", empty_note, "*"))
} else {
kable(df, caption = tab_cap(caption), format.args = list(big.mark = ","))
}
}
# One row of the quality-assurance tables
# rule "zero" -> PASS when the value is 0, otherwise REVIEW
# rule "positive" -> PASS when the value is > 0, otherwise REVIEW
# rule "ge90" -> PASS when the value is >= 90, otherwise REVIEW
# rule "info" -> INFO (descriptive, no pass/fail judgement)
qa <- function(check, value, rule = "info") {
num <- suppressWarnings(as.numeric(value))
ok <- length(num) == 1 && !is.na(num)
status <- if (!ok) "INFO" else switch(rule,
zero = if (num == 0) "PASS" else "REVIEW",
positive = if (num > 0) "PASS" else "REVIEW",
ge90 = if (num >= 90) "PASS" else "REVIEW",
"INFO")
tibble(Check = check,
Result = if (ok) format(num, big.mark = ",", scientific = FALSE) else as.character(value),
Status = status)
}
cat("Project root used:", project_root, "\n")## Project root used: C:/Users/acer/Downloads/HW4## Current working directory: C:/Users/acer/Downloads/HW4## Refresh mode: FALSEThe ISIN page is a single HTML table. Category rows (a row that contains only a security-type title) are interleaved with data rows. When the page has to be processed, the code (i) detects the header row by its text, (ii) checks that the columns appear in the expected order, (iii) records the current category for every data row, and (iv) splits the first column into stock ID and company name. Three levels of data are distinguished:
data_cache/twse_raw.csv (optional, produced only if the
page is processed).data_cache/twse_clean_saved.csv (the file already created
in an earlier run).TWSE_Common_Stocks.csv.If the cleaned file is valid, it is used directly: the page is neither downloaded nor parsed again.
twse_raw_cols <- c("Category", "Stock_ID", "Company_Name", "Listing_Date_Raw", "Industry", "CFI_Code")
# ---- Level 2: the cleaned file written in an earlier run ----------------------------
# Expected columns: stock_id, company_name, listing_date, industry, isin, market, cfi_code
load_twse_clean_saved <- function(path) {
if (!file.exists(path)) return(NULL)
d <- tryCatch(read_chr_csv(path), error = function(e) NULL)
if (is.null(d) || nrow(d) == 0) return(NULL)
nm <- names(d)
m <- list(Stock_ID = pick_col(nm, c("stock_id", "Stock_ID", "stock_code", "code")),
Company_Name = pick_col(nm, c("company_name", "Company_Name", "stock_name", "name")),
Listing_Date_Raw = pick_col(nm, c("listing_date", "Listing_Date", "list_date")),
Industry = pick_col(nm, c("industry", "Industry", "industry_category")),
ISIN = pick_col(nm, c("isin", "ISIN")),
Market = pick_col(nm, c("market", "Market")),
CFI_Code = pick_col(nm, c("cfi_code", "CFI_Code", "cfi")))
if (any(is.na(unlist(m[c("Stock_ID", "Company_Name", "Listing_Date_Raw", "Industry")])))) return(NULL)
g <- function(k) if (is.na(m[[k]])) rep(NA_character_, nrow(d)) else d[[m[[k]]]]
out <- tibble(Category = rep(NA_character_, nrow(d)),
Stock_ID = str_squish(g("Stock_ID")),
Company_Name = str_squish(g("Company_Name")),
ISIN = g("ISIN"),
Listing_Date_Raw = g("Listing_Date_Raw"),
Market = g("Market"),
Industry = g("Industry"),
CFI_Code = g("CFI_Code"))
if (mean(str_detect(coalesce(out$Stock_ID, ""), "^[0-9A-Za-z]{4,6}$")) < 0.5) return(NULL)
out
}
# ---- Level 1: the raw ISIN page (only used when no valid cleaned file exists) -------
parse_twse_table <- function(doc) {
rows <- html_elements(doc, "tr")
cells <- lapply(rows, function(r) str_squish(html_text(html_elements(r, "td, th"))))
is_header <- vapply(cells, function(x) length(x) >= 6 && any(str_detect(x, "ISIN")), logical(1))
hdr_idx <- which(is_header)[1]
if (is.na(hdr_idx)) stop("Header row not found in the TWSE table.", call. = FALSE)
header <- cells[[hdr_idx]]
n_col <- length(header)
if (!(str_detect(header[2], "ISIN") && str_detect(header[6], "CFI"))) {
stop("Unexpected TWSE column order. Header seen: ", paste(header, collapse = " | "), call. = FALSE)
}
category <- NA_character_
rows_out <- vector("list", length(cells))
k <- 0L
for (i in seq_along(cells)) {
if (i <= hdr_idx) next
x <- cells[[i]]
if (length(x) == 1 && nzchar(x)) {
category <- x # category title row
} else if (length(x) == n_col) {
k <- k + 1L
rows_out[[k]] <- c(category, x) # data row
} # any other row is ignored
}
if (k == 0L) stop("No data rows were parsed from the TWSE table.", call. = FALSE)
mat <- do.call(rbind, rows_out[seq_len(k)])
colnames(mat) <- c("Category", paste0("V", seq_len(n_col)))
df <- as_tibble(mat)
# Column 1 holds "stock code <full-width space> company name"
sp <- str_match(df$V1, "^([^\\s\\x{3000}\\x{00A0}]+)[\\s\\x{3000}\\x{00A0}]+(.*)$")
tibble(Category = df$Category,
Stock_ID = sp[, 2],
Company_Name = str_squish(sp[, 3]),
ISIN = df$V2,
Listing_Date_Raw = df$V3,
Market = df$V4,
Industry = df$V5,
CFI_Code = df$V6) |>
mutate(across(where(is.character), ~ na_if(str_squish(.x), "")))
}
# Decode raw bytes (the site serves Big5). An encoding is accepted only when the
# "stocks" category becomes readable, which proves that the decoding worked.
decode_and_parse_twse <- function(bytes) {
for (enc in c("BIG5", "BIG5-HKSCS", "UTF-8")) {
doc <- tryCatch(read_html(bytes, encoding = enc), error = function(e) NULL)
if (is.null(doc)) next
tab <- tryCatch(parse_twse_table(doc), error = function(e) NULL)
if (!is.null(tab) && any(tab$Category == CAT_STOCK, na.rm = TRUE)) {
return(list(data = tab, encoding = enc))
}
}
stop("The TWSE page could not be decoded or the 'stocks' category was not found. ",
"The page structure or encoding may have changed.", call. = FALSE)
}
download_twse_bytes <- function(urls) {
use_pkgs("httr2")
last_err <- "no response"
for (u in urls) {
bytes <- tryCatch({
resp <- request(u) |>
req_user_agent("Mozilla/5.0 (university assignment; R httr2)") |>
req_timeout(60) |>
req_retry(max_tries = 3) |>
req_perform()
resp_body_raw(resp)
}, error = function(e) { last_err <<- conditionMessage(e); NULL })
if (!is.null(bytes) && length(bytes) > 1000) return(bytes)
}
stop("Could not download the TWSE ISIN page. Last error: ", last_err, call. = FALSE)
}
read_twse_raw_cache <- function() {
if (!file.exists(path_twse_raw)) return(NULL)
d <- tryCatch(read_chr_csv(path_twse_raw), error = function(e) NULL)
if (!is.null(d) && nrow(d) > 0 && all(twse_raw_cols %in% names(d))) d else NULL
}
# Returns the raw table (all categories), a note on its origin and whether it is new.
get_twse_raw <- function() {
if (!refresh) {
d <- read_twse_raw_cache()
if (!is.null(d)) return(list(data = d, fresh = FALSE,
note = "raw table loaded from data_cache/twse_raw.csv"))
}
use_pkgs(c("rvest", "xml2"))
res <- tryCatch({
parsed <- decode_and_parse_twse(download_twse_bytes(twse_urls))
list(data = parsed$data, fresh = TRUE,
note = paste0("downloaded live from TWSE (decoded as ", parsed$encoding, ")"))
}, error = function(e) e)
if (inherits(res, "error")) {
manual <- path_manual_html[file.exists(path_manual_html)]
if (length(manual) > 0) { # manually saved HTML page
bytes <- readBin(manual[1], "raw", n = file.size(manual[1]))
parsed <- decode_and_parse_twse(bytes)
res <- list(data = parsed$data, fresh = TRUE,
note = paste0("parsed from the manually saved page ", basename(manual[1])))
} else {
d <- read_twse_raw_cache()
if (is.null(d)) {
stop(conditionMessage(res), "\nSave the page manually as 'twse_isin_strMode2.html' ",
"in the project folder and knit again.", call. = FALSE)
}
res <- list(data = d, fresh = FALSE, note = "download failed; older raw cache used")
}
}
res
}twse_clean_file <- if (!refresh) load_twse_clean_saved(path_twse_clean_saved) else NULL
twse_all <- NULL # raw table with all security types, when available
twse_fresh <- FALSE # TRUE only if the raw page was processed in this run
if (!is.null(twse_clean_file)) {
twse_level <- "clean"
twse_input <- twse_clean_file
twse_source <- paste0("local file data_cache/twse_clean_saved.csv (last modified ",
format(file.mtime(path_twse_clean_saved), "%Y-%m-%d %H:%M"),
"); no download and no HTML parsing performed")
twse_all <- read_twse_raw_cache() # optional, only used for the exclusion statistics
} else {
twse_res <- get_twse_raw()
twse_level <- "raw"
twse_all <- twse_res$data
twse_input <- twse_all
twse_source <- twse_res$note
twse_fresh <- twse_res$fresh
}
tibble(Item = c("Project root", "TWSE input used", "Processing level",
"Raw all-category table available", "Rows in the input table"),
Value = c(project_root, twse_source,
if (twse_level == "clean") "cleaned common-stock file (re-validated below)"
else "raw page (cleaned below)",
if (is.null(twse_all)) "No" else paste0("Yes (", format(nrow(twse_all), big.mark = ","), " rows)"),
format(nrow(twse_input), big.mark = ","))) |>
kable(caption = tab_cap("TWSE data source used in this run"))| Item | Value |
|---|---|
| Project root | C:/Users/acer/Downloads/HW4 |
| TWSE input used | local file data_cache/twse_clean_saved.csv (last modified 2026-10-11 11:30); no download and no HTML parsing performed |
| Processing level | cleaned common-stock file (re-validated below) |
| Raw all-category table available | No |
| Rows in the input table | 1,053 |
The filter rests on official classification information rather than on the length of the stock code. Two official markers are used together:
ES (common shares), preferred shares start with
EP, and funds/ETFs, depositary receipts and warrants use
other prefixes.The same rule is applied to both levels of input. When the already-cleaned file is used, the rule acts as a re-validation and is expected to remove nothing; the number of removed rows is reported below. Markers that are absent from the input (for example the category title in the cleaned file) are skipped and this is stated in the validation table.
twse_input <- twse_input |>
mutate(Listing_Date = parse_date_multi(Listing_Date_Raw))
cat_available <- any(twse_input$Category == CAT_STOCK, na.rm = TRUE)
cfi_available <- any(!is.na(twse_input$CFI_Code))
twse_clean <- twse_input
if (cat_available) twse_clean <- twse_clean |> filter(Category == CAT_STOCK)
if (cfi_available) twse_clean <- twse_clean |> filter(str_detect(coalesce(CFI_Code, ""), "^ES"))
n_removed_by_filter <- nrow(twse_input) - nrow(twse_clean)
n_dup_removed <- sum(duplicated(twse_clean$Stock_ID))
twse_clean <- twse_clean |>
distinct(Stock_ID, .keep_all = TRUE) |>
arrange(Stock_ID)
if (nrow(twse_clean) == 0) stop("The TWSE common-stock filter returned no rows.", call. = FALSE)
n_twse <- nrow(twse_clean)
cat("Listed common stocks after filtering:", format(n_twse, big.mark = ","), "\n")## Listed common stocks after filtering: 1,053# Keep the files in data_cache consistent with what was actually processed.
if (twse_level == "raw") {
if (twse_fresh) write_excel_csv(twse_all, path_twse_raw)
write_excel_csv(
twse_clean |>
transmute(stock_id = Stock_ID, company_name = Company_Name,
listing_date = format(Listing_Date, "%Y/%m/%d"),
industry = Industry, isin = ISIN, market = Market, cfi_code = CFI_Code),
path_twse_clean_saved)
}han_share <- 100 * mean(str_detect(coalesce(twse_clean$Company_Name, ""), "\\p{Han}"))
qa_twse <- bind_rows(
qa("Securities in the raw all-category table",
if (is.null(twse_all)) "not available in this run" else nrow(twse_all)),
qa("Rows in the input table used", nrow(twse_input)),
qa("Listed common stocks after filtering", nrow(twse_clean), "positive"),
qa("Rows removed by the common-stock filter (0 expected for an already-cleaned file)",
n_removed_by_filter, if (twse_level == "clean") "zero" else "info"),
qa("Category marker available in the input (1 = yes, 0 = no)", as.integer(cat_available)),
qa("CFI marker available in the input (1 = yes, 0 = no)", as.integer(cfi_available)),
qa("Duplicate Stock IDs removed", n_dup_removed, "zero"),
qa("Duplicate Stock IDs in the final table", sum(duplicated(twse_clean$Stock_ID)), "zero"),
qa("Missing Stock ID", sum(is.na(twse_clean$Stock_ID)), "zero"),
qa("Missing company name", sum(is.na(twse_clean$Company_Name)), "zero"),
qa("Missing listing date", sum(is.na(twse_clean$Listing_Date)), "zero"),
qa("Listing dates that could not be parsed (text present, date invalid)",
sum(!is.na(twse_clean$Listing_Date_Raw) & is.na(twse_clean$Listing_Date)), "zero"),
qa("Listing dates in the future", sum(twse_clean$Listing_Date > Sys.Date(), na.rm = TRUE), "zero"),
qa("Listing dates before 1962-01-01 (before TWSE trading began)",
sum(twse_clean$Listing_Date < as.Date("1962-01-01"), na.rm = TRUE), "zero"),
qa("Earliest listing date", format(min(twse_clean$Listing_Date, na.rm = TRUE))),
qa("Latest listing date", format(max(twse_clean$Listing_Date, na.rm = TRUE))),
qa("Missing industry", sum(is.na(twse_clean$Industry))),
qa("Retained rows whose CFI code is not 'ES...'",
if (cfi_available) sum(!str_detect(coalesce(twse_clean$CFI_Code, ""), "^ES")) else "CFI not available", "zero"),
qa("Retained rows whose CFI code is preferred 'EP...'",
if (cfi_available) sum(str_detect(coalesce(twse_clean$CFI_Code, ""), "^EP")) else "CFI not available", "zero"),
qa("Retained Stock IDs that are not exactly 4 digits",
sum(!str_detect(coalesce(twse_clean$Stock_ID, ""), "^[0-9]{4}$")), "zero"),
qa("Retained rows with a market value other than the listed market",
sum(!is.na(twse_clean$Market) & twse_clean$Market != MARKET_LISTED)),
qa("Retained rows that refer to the Taiwan Innovation Board (market or industry label)",
sum(str_detect(coalesce(twse_clean$Market, ""), INNOV_PATTERN) |
str_detect(coalesce(twse_clean$Industry, ""), INNOV_PATTERN))),
qa("Company names containing Chinese characters (%; checks the text encoding)", round(han_share, 1), "ge90")
)
kable(qa_twse, caption = tab_cap("TWSE quality checks (PASS / REVIEW / INFO)"))| Check | Result | Status |
|---|---|---|
| Securities in the raw all-category table | not available in this run | INFO |
| Rows in the input table used | 1,053 | INFO |
| Listed common stocks after filtering | 1,053 | PASS |
| Rows removed by the common-stock filter (0 expected for an already-cleaned file) | 0 | PASS |
| Category marker available in the input (1 = yes, 0 = no) | 0 | INFO |
| CFI marker available in the input (1 = yes, 0 = no) | 1 | INFO |
| Duplicate Stock IDs removed | 0 | PASS |
| Duplicate Stock IDs in the final table | 0 | PASS |
| Missing Stock ID | 0 | PASS |
| Missing company name | 0 | PASS |
| Missing listing date | 0 | PASS |
| Listing dates that could not be parsed (text present, date invalid) | 0 | PASS |
| Listing dates in the future | 0 | PASS |
| Listing dates before 1962-01-01 (before TWSE trading began) | 0 | PASS |
| Earliest listing date | 1962-02-09 | INFO |
| Latest listing date | 2026-09-01 | INFO |
| Missing industry | 0 | INFO |
| Retained rows whose CFI code is not ‘ES…’ | 0 | PASS |
| Retained rows whose CFI code is preferred ‘EP…’ | 0 | PASS |
| Retained Stock IDs that are not exactly 4 digits | 0 | PASS |
| Retained rows with a market value other than the listed market | 0 | INFO |
| Retained rows that refer to the Taiwan Innovation Board (market or industry label) | 0 | INFO |
| Company names containing Chinese characters (%; checks the text encoding) | 99.2 | PASS |
PASS means the check met its expectation,
REVIEW means a value that should be zero (or positive) was
not, and INFO is descriptive. A REVIEW is not
necessarily an error; it marks something to inspect.
twse_clean |>
count(Market, name = "Stocks", sort = TRUE) |>
show_tbl("Market values of the retained TWSE common stocks")| Market | Stocks |
|---|---|
| 上市 | 1,053 |
if (!is.null(twse_all)) {
twse_all |>
filter(!(Category == CAT_STOCK & str_detect(coalesce(CFI_Code, ""), "^ES"))) |>
mutate(CFI_Prefix = str_sub(CFI_Code, 1, 2)) |>
count(Category, CFI_Prefix, name = "Excluded_Rows") |>
arrange(desc(Excluded_Rows)) |>
head(15) |>
show_tbl("Securities excluded from the common-stock list (largest 15 groups; preferred shares have CFI prefix EP)")
} else {
knitr::asis_output(paste0("*The raw all-category table was not processed in this run (the cleaned file ",
"was used), so a breakdown of the excluded security types is not shown.*"))
}The raw all-category table was not processed in this run (the cleaned file was used), so a breakdown of the excluded security types is not shown.
twse_out <- twse_clean |>
select(Stock_ID, Company_Name, Listing_Date, Industry, ISIN, Market, CFI_Code)
# The four fields requested in the brief
write_excel_csv(twse_out |> select(Stock_ID, Company_Name, Listing_Date, Industry), out_twse)
n_twse <- nrow(twse_out)The cleaned dataset contains 1,053 listed common
stocks and was saved as TWSE_Common_Stocks.csv (stock ID,
company name, listing date, industry). Stock IDs are stored as text.
twse_out |>
select(Stock_ID, Company_Name, Listing_Date, Industry) |>
head(10) |>
show_tbl("First 10 observations of the cleaned TWSE dataset")| Stock_ID | Company_Name | Listing_Date | Industry |
|---|---|---|---|
| 1101 | 台泥 | 1962-02-09 | 水泥工業 |
| 1102 | 亞泥 | 1962-06-08 | 水泥工業 |
| 1103 | 嘉泥 | 1969-11-14 | 水泥工業 |
| 1104 | 環泥 | 1971-02-01 | 水泥工業 |
| 1108 | 幸福 | 1990-06-06 | 水泥工業 |
| 1109 | 信大 | 1991-12-05 | 水泥工業 |
| 1110 | 東泥 | 1994-10-22 | 水泥工業 |
| 1201 | 味全 | 1962-02-09 | 食品工業 |
| 1203 | 味王 | 1964-08-24 | 食品工業 |
| 1210 | 大成 | 1978-05-20 | 食品工業 |
twse_out |>
count(Industry, name = "Stocks", sort = TRUE) |>
head(15) |>
show_tbl("Largest 15 TWSE industry categories (common stocks)")| Industry | Stocks |
|---|---|
| 電子零組件業 | 103 |
| 半導體業 | 91 |
| 光電業 | 67 |
| 電腦及週邊設備業 | 63 |
| 建材營造業 | 55 |
| 生技醫療業 | 54 |
| 其他業 | 52 |
| 電機機械 | 49 |
| 通信網路業 | 46 |
| 其他電子業 | 45 |
| 紡織纖維 | 42 |
| 汽車工業 | 40 |
| 金融保險業 | 31 |
| 鋼鐵工業 | 31 |
| 化學工業 | 28 |
The complete TWSE dataset can be browsed below.
FinMind exposes a REST API
(https://api.finmindtrade.com/api/v4/data). The
TaiwanStockInfo dataset returns one row per security with a
stock ID, name, industry category, market type and a date
field. The raw response is kept in
data_cache/finmind_raw.csv and is used directly whenever it
is valid; the API is only called when that file is missing or
refresh: true is set. The optional API token is
never written in this file; it is read from the
environment variable FINMIND_TOKEN and sent only in an HTTP
header.
To set it safely in RStudio (do not type it into the
.Rmd), run once in the Console:
Sys.setenv(FINMIND_TOKEN = "paste-your-token-here") # Console only; do NOT save this line in the Rmdor add FINMIND_TOKEN=... to your user
.Renviron file (usethis::edit_r_environ()),
then restart R.
# Standardise a raw FinMind table (from the local file or from the API).
# Required columns: stock_id, stock_name, industry_category. Returns NULL if they are missing.
standardise_fm <- function(d) {
nm <- names(d)
m <- c(stock_id = pick_col(nm, c("stock_id", "Stock_ID")),
stock_name = pick_col(nm, c("stock_name", "Stock_Name", "company_name")),
industry_category = pick_col(nm, c("industry_category", "Industry_Category", "industry")),
type = pick_col(nm, c("type", "Type", "market_type")),
date = pick_col(nm, c("date", "Date")),
listing_date_src = pick_col(nm, c("listing_date", "list_date", "ipo_date", "listed_date")))
if (any(is.na(m[c("stock_id", "stock_name", "industry_category")]))) return(NULL)
g <- function(k) if (is.na(m[[k]])) rep(NA_character_, nrow(d)) else as.character(d[[m[[k]]]])
out <- tibble(stock_id = str_squish(g("stock_id")),
stock_name = g("stock_name"),
industry_category = g("industry_category"),
type = g("type"),
date = g("date"),
listing_date_src = g("listing_date_src")) |>
mutate(across(everything(), ~ na_if(.x, "")))
attr(out, "orig_names") <- nm
out
}
load_fm_raw <- function(path) {
if (!file.exists(path)) return(NULL)
d <- tryCatch(read_chr_csv(path), error = function(e) NULL)
if (is.null(d) || nrow(d) == 0) return(NULL)
standardise_fm(d)
}
fetch_finmind_info <- function(dataset = "TaiwanStockInfo") {
use_pkgs(c("httr2", "jsonlite"))
token <- Sys.getenv("FINMIND_TOKEN")
req <- request(finmind_url) |>
req_url_query(dataset = dataset) |>
req_user_agent("university assignment; R httr2") |>
req_timeout(60) |>
req_error(is_error = function(resp) FALSE) |> # status codes are inspected below
req_retry(max_tries = 3,
is_transient = function(resp) resp_status(resp) %in% c(429, 500, 502, 503, 504))
if (nzchar(token)) req <- req_auth_bearer_token(req, token) # token only in the header
resp <- tryCatch(req_perform(req),
error = function(e) stop("Could not connect to FinMind: ",
conditionMessage(e), call. = FALSE))
http <- resp_status(resp)
js <- tryCatch(jsonlite::fromJSON(resp_body_string(resp, encoding = "UTF-8"), flatten = TRUE),
error = function(e) NULL)
if (is.null(js)) stop("FinMind returned a response that is not valid JSON (HTTP ", http, ").",
call. = FALSE)
api_status <- suppressWarnings(as.integer(js$status))
if (http != 200 || length(api_status) != 1 || is.na(api_status) || api_status != 200L) {
stop("FinMind request failed. HTTP ", http, ", API status ",
paste(js$status, collapse = ""), ", message: ", paste(js$msg, collapse = ""),
". If this is a rate-limit or login message, set FINMIND_TOKEN and retry later.",
call. = FALSE)
}
df <- as_tibble(js$data)
if (nrow(df) == 0) stop("FinMind returned an empty dataset.", call. = FALSE)
df |> mutate(across(everything(), as.character))
}fm_raw <- if (!refresh) load_fm_raw(path_fm_raw) else NULL
if (!is.null(fm_raw)) {
fm_source <- paste0("local file data_cache/finmind_raw.csv (last modified ",
format(file.mtime(path_fm_raw), "%Y-%m-%d %H:%M"), "); no download performed")
} else {
fm_dl <- tryCatch(fetch_finmind_info("TaiwanStockInfo"), error = function(e) e)
if (inherits(fm_dl, "error")) {
stop("data_cache/finmind_raw.csv is missing or lacks the required columns, and the API ",
"download failed: ", conditionMessage(fm_dl), call. = FALSE)
}
fm_raw <- standardise_fm(fm_dl)
if (is.null(fm_raw)) {
stop("The FinMind response lacks the required columns (stock_id, stock_name, ",
"industry_category). Columns found: ", paste(names(fm_dl), collapse = ", "), call. = FALSE)
}
write_excel_csv(fm_dl, path_fm_raw)
fm_source <- "downloaded live from the FinMind API and saved to data_cache/finmind_raw.csv"
}
tibble(Item = c("FinMind input used", "Dataset", "Rows in the raw table",
"Columns in the raw file", "Authenticated request"),
Value = c(fm_source, "TaiwanStockInfo", format(nrow(fm_raw), big.mark = ","),
paste(attr(fm_raw, "orig_names"), collapse = ", "),
if (grepl("^downloaded", fm_source)) as.character(nzchar(Sys.getenv("FINMIND_TOKEN"))) else "not applicable")) |>
kable(caption = tab_cap("FinMind data source used in this run"))| Item | Value |
|---|---|
| FinMind input used | local file data_cache/finmind_raw.csv (last modified 2026-10-11 11:42); no download performed |
| Dataset | TaiwanStockInfo |
| Rows in the raw table | 4,333 |
| Columns in the raw file | industry_category, stock_id, stock_name, type, date |
| Authenticated request | not applicable |
FinMind does not carry the CFI code, so the common-stock universe is identified as reliably as the available fields allow. A row is kept only when
type column is
present),The four-digit rule is a proxy: it cannot tell a common stock from another security that happens to have a four-digit code (for example a depositary receipt), which is why the label rule is applied as well. The result is cross-checked against the official TWSE list in Section 10, which acts as an independent test of these rules without being used to build them. Securities labelled as Taiwan Innovation Board stocks are not excluded by these rules; they are counted separately in the validation table.
has_type <- any(!is.na(fm_raw$type))
# Industry labels that indicate non-common-stock instruments
excl_pattern <- "ETF|ETN|Index|index|大盤|存託憑證|受益|基金|權證|Warrant|所有證券"
fm_flagged <- fm_raw |>
mutate(Exclusion_Reason = case_when(
has_type & str_to_lower(coalesce(type, "")) != "twse" ~ "Market type is not TWSE",
str_detect(coalesce(industry_category, ""), excl_pattern) ~ "ETF / ETN / index / DR / fund / warrant-type industry label",
!str_detect(coalesce(stock_id, ""), "^[1-9][0-9]{3}$") ~ "ID is not a plain 4-digit code (preferred, ETF, warrant, etc.)",
TRUE ~ "Retained"))
fm_flagged |>
count(Exclusion_Reason, name = "Rows", sort = TRUE) |>
show_tbl("FinMind raw rows by common-stock filter outcome (rows, before de-duplication)")| Exclusion_Reason | Rows |
|---|---|
| Retained | 1,988 |
| Market type is not TWSE | 1,931 |
| ETF / ETN / index / DR / fund / warrant-type industry label | 374 |
| ID is not a plain 4-digit code (preferred, ETF, warrant, etc.) | 40 |
# A listing-date column exists only if the response really contains one
has_fm_listing <- any(!is.na(fm_raw$listing_date_src))
fm_date_sentence <- if (has_fm_listing) {
"A column holding a listing date was found in the response and was used."
} else {
paste0("The TaiwanStockInfo response contains **no field documented as a listing date**. ",
"Its `date` column is not verified to be a listing date and is therefore not used as one; ",
"the FinMind Listing_Date column is empty by design and is not filled from TWSE.")
}
# FinMind can list the same stock under several industry labels -> one row per stock
fm_clean <- fm_flagged |>
filter(Exclusion_Reason == "Retained") |>
group_by(Stock_ID = stock_id) |>
summarise(
Company_Name = first(stock_name),
Industry = na_if(paste(sort(unique(industry_category[!is.na(industry_category) &
industry_category != ""])),
collapse = "; "), ""),
Market_Type = if (has_type) first(type) else NA_character_,
Listing_Date = if (has_fm_listing) parse_date_multi(first(listing_date_src)) else as.Date(NA),
Date_Field = last(sort(unique(na.omit(date)))),
Source_Rows = n(),
.groups = "drop"
) |>
arrange(Stock_ID)
if (nrow(fm_clean) == 0) stop("The FinMind common-stock filter returned no rows.", call. = FALSE)
# Verification against the official TWSE list (used only for labelling, never for selection)
fm_clean <- fm_clean |> mutate(TWSE_Verified = Stock_ID %in% twse_out$Stock_ID)
n_fm <- nrow(fm_clean)
write_excel_csv(fm_clean |> select(Stock_ID, Company_Name, Listing_Date, Industry, TWSE_Verified),
out_finmind)FinMind provides 1,215 comparable TWSE common
stocks, of which 1,053 are confirmed by the official
TWSE list (saved as FinMind_Common_Stocks.csv, with the
verification flag in the last column). The TaiwanStockInfo response
contains no field documented as a listing date. Its
date column is not verified to be a listing date and is
therefore not used as one; the FinMind Listing_Date column is empty by
design and is not filled from TWSE.
fm_coverage <- tibble(
Field = c("Stock ID", "Company name", "Industry", "Market type", "Listing date"),
Available_In_FinMind = c("Yes", "Yes", "Yes",
if (has_type) "Yes" else "No",
if (has_fm_listing) "Yes" else "No"),
Missing_Values = c(sum(is.na(fm_clean$Stock_ID)), sum(is.na(fm_clean$Company_Name)),
sum(is.na(fm_clean$Industry)),
if (has_type) sum(is.na(fm_clean$Market_Type)) else NA_integer_,
if (has_fm_listing) sum(is.na(fm_clean$Listing_Date)) else NA_integer_)
)
kable(fm_coverage, caption = tab_cap("Field coverage of the FinMind dataset (NA = field not provided)"))| Field | Available_In_FinMind | Missing_Values |
|---|---|---|
| Stock ID | Yes | 0 |
| Company name | Yes | 0 |
| Industry | Yes | 0 |
| Market type | Yes | 0 |
| Listing date | No | NA |
fm_clean |>
select(Stock_ID, Company_Name, Listing_Date, Industry, TWSE_Verified) |>
head(10) |>
show_tbl("First 10 observations of the FinMind dataset")| Stock_ID | Company_Name | Listing_Date | Industry | TWSE_Verified |
|---|---|---|---|---|
| 1101 | 台泥 | NA | 水泥工業 | TRUE |
| 1102 | 亞泥 | NA | 水泥工業 | TRUE |
| 1103 | 嘉泥 | NA | 水泥工業 | TRUE |
| 1104 | 環泥 | NA | 水泥工業 | TRUE |
| 1107 | 建台 | NA | 其他 | FALSE |
| 1108 | 幸福 | NA | 水泥工業 | TRUE |
| 1109 | 信大 | NA | 水泥工業 | TRUE |
| 1110 | 東泥 | NA | 水泥工業 | TRUE |
| 1201 | 味全 | NA | 食品工業 | TRUE |
| 1203 | 味王 | NA | 食品工業 | TRUE |
Rows with more than one industry label in FinMind: 712 (their labels are joined with a semicolon).
# Cross-check with the result saved in an earlier run (information only)
fm_saved <- tryCatch({
if (file.exists(path_fm_clean_saved)) {
d <- read_chr_csv(path_fm_clean_saved)
ic <- pick_col(names(d), c("stock_id", "Stock_ID"))
if (!is.na(ic)) {
ids <- unique(d[[ic]])
list(n = length(ids), only_saved = length(setdiff(ids, fm_clean$Stock_ID)),
only_new = length(setdiff(fm_clean$Stock_ID, ids)))
} else NULL
} else NULL
}, error = function(e) NULL)
qa_fm <- bind_rows(
qa("Rows in the raw FinMind table", nrow(fm_raw)),
qa("Rows with market type TWSE", if (has_type) sum(str_to_lower(coalesce(fm_raw$type, "")) == "twse") else "type not available"),
qa("Comparable common stocks retained (one row per Stock ID)", n_fm, "positive"),
qa("Duplicate Stock IDs in the retained table", sum(duplicated(fm_clean$Stock_ID)), "zero"),
qa("Missing company name", sum(is.na(fm_clean$Company_Name)), "zero"),
qa("Missing industry", sum(is.na(fm_clean$Industry))),
qa("Retained Stock IDs that are not exactly 4 digits", sum(!str_detect(fm_clean$Stock_ID, "^[0-9]{4}$")), "zero"),
qa("Retained rows with an ETF / ETN / index / DR / fund / warrant label",
sum(str_detect(coalesce(fm_clean$Industry, ""), excl_pattern)), "zero"),
qa("Retained rows labelled as Taiwan Innovation Board stocks",
sum(str_detect(coalesce(fm_clean$Industry, ""), INNOV_PATTERN))),
qa("Retained stocks confirmed by the official TWSE list", sum(fm_clean$TWSE_Verified)),
qa("Retained stocks NOT confirmed by the official TWSE list (see Section 10)", sum(!fm_clean$TWSE_Verified)),
qa("FinMind listing date available (1 = yes, 0 = no)", as.integer(has_fm_listing))
)
if (!is.null(fm_saved)) {
qa_fm <- bind_rows(qa_fm,
qa("IDs in data_cache/finmind_clean_saved.csv (earlier run)", fm_saved$n),
qa("IDs in the earlier file but not in this run's result", fm_saved$only_saved),
qa("IDs in this run's result but not in the earlier file", fm_saved$only_new))
}
kable(qa_fm, caption = tab_cap("FinMind quality checks (PASS / REVIEW / INFO)"))| Check | Result | Status |
|---|---|---|
| Rows in the raw FinMind table | 4,333 | INFO |
| Rows with market type TWSE | 2,402 | INFO |
| Comparable common stocks retained (one row per Stock ID) | 1,215 | PASS |
| Duplicate Stock IDs in the retained table | 0 | PASS |
| Missing company name | 0 | PASS |
| Missing industry | 0 | INFO |
| Retained Stock IDs that are not exactly 4 digits | 0 | PASS |
| Retained rows with an ETF / ETN / index / DR / fund / warrant label | 0 | PASS |
| Retained rows labelled as Taiwan Innovation Board stocks | 37 | INFO |
| Retained stocks confirmed by the official TWSE list | 1,053 | INFO |
| Retained stocks NOT confirmed by the official TWSE list (see Section 10) | 162 | INFO |
| FinMind listing date available (1 = yes, 0 = no) | 0 | INFO |
| IDs in data_cache/finmind_clean_saved.csv (earlier run) | 1,215 | INFO |
| IDs in the earlier file but not in this run’s result | 0 | INFO |
| IDs in this run’s result but not in the earlier file | 0 | INFO |
The complete FinMind dataset can be browsed below.
rmarkdown::paged_table(fm_clean |> select(Stock_ID, Company_Name, Listing_Date, Industry, TWSE_Verified))The two datasets are joined on Stock_ID. Company names
are compared after Unicode (NFKC) normalisation and whitespace trimming.
Industry labels are reported side by side but not required to be
identical, because the two providers use different classification
systems. A missing value is labelled Missing; a value
present in both sources but different is labelled
Mismatch.
norm <- function(x) str_to_lower(str_squish(stringi::stri_trans_nfkc(x)))
t_side <- twse_out |>
transmute(Stock_ID, In_TWSE = TRUE,
TWSE_Company_Name = Company_Name, TWSE_Industry = Industry,
TWSE_Listing_Date = Listing_Date)
f_side <- fm_clean |>
transmute(Stock_ID, In_FinMind = TRUE,
FinMind_Company_Name = Company_Name, FinMind_Industry = Industry,
FinMind_Listing_Date = Listing_Date,
FinMind_Date_Field_Unverified = Date_Field)
comparison <- full_join(t_side, f_side, by = "Stock_ID") |>
mutate(In_TWSE = coalesce(In_TWSE, FALSE),
In_FinMind = coalesce(In_FinMind, FALSE),
In_Both = In_TWSE & In_FinMind) |>
mutate(
Name_Check = case_when(
!In_Both ~ NA_character_,
is.na(TWSE_Company_Name) | is.na(FinMind_Company_Name) ~ "Missing",
norm(TWSE_Company_Name) == norm(FinMind_Company_Name) ~ "Match",
TRUE ~ "Mismatch"),
Industry_Label_Check = case_when(
!In_Both ~ NA_character_,
is.na(TWSE_Industry) | is.na(FinMind_Industry) ~ "Missing",
norm(TWSE_Industry) == norm(FinMind_Industry) ~ "Same label",
TRUE ~ "Different label"),
Listing_Date_Check = if (has_fm_listing) {
case_when(
!In_Both ~ NA_character_,
is.na(TWSE_Listing_Date) | is.na(FinMind_Listing_Date) ~ "Missing",
TWSE_Listing_Date == FinMind_Listing_Date ~ "Match",
TRUE ~ "Mismatch")
} else {
if_else(In_Both, "Not available in FinMind", NA_character_)
},
Matching_Status = case_when(
In_TWSE & !In_FinMind ~ "Found only in TWSE",
!In_TWSE & In_FinMind ~ "Found only in FinMind (comparable universe)",
Name_Check == "Mismatch" | coalesce(Listing_Date_Check == "Mismatch", FALSE) ~
"Found in both - information differs",
Name_Check == "Missing" | coalesce(Listing_Date_Check == "Missing", FALSE) ~
"Found in both - missing field(s)",
TRUE ~ "Found in both - consistent")
) |>
select(Stock_ID, In_TWSE, In_FinMind,
TWSE_Company_Name, FinMind_Company_Name,
TWSE_Industry, FinMind_Industry,
TWSE_Listing_Date, FinMind_Listing_Date, FinMind_Date_Field_Unverified,
Name_Check, Industry_Label_Check, Listing_Date_Check, Matching_Status) |>
arrange(Stock_ID)
write_excel_csv(comparison, out_compare)
# How FinMind's own filter treated each stock (used to explain the unmatched IDs)
fm_reason_lookup <- fm_flagged |>
group_by(stock_id) |>
summarise(FinMind_Filter_Outcome = paste(sort(unique(Exclusion_Reason)), collapse = "; "),
.groups = "drop")
unmatched_out <- comparison |>
filter(!(In_TWSE & In_FinMind)) |>
left_join(fm_reason_lookup, by = c("Stock_ID" = "stock_id")) |>
mutate(Source = if_else(In_TWSE, "Only in TWSE", "Only in FinMind"),
Company_Name = coalesce(TWSE_Company_Name, FinMind_Company_Name),
FinMind_Filter_Outcome = if_else(
In_TWSE,
coalesce(FinMind_Filter_Outcome, "Not present in the FinMind TaiwanStockInfo table"),
"Retained by the FinMind filter")) |>
select(Source, Stock_ID, Company_Name, TWSE_Industry, FinMind_Industry,
FinMind_Date_Field_Unverified, FinMind_Filter_Outcome) |>
arrange(Source, Stock_ID)
write_excel_csv(unmatched_out, out_unmatched)
# Headline numbers used in the text below
n_both <- sum(str_detect(comparison$Matching_Status, "^Found in both"))
n_only_twse <- sum(comparison$Matching_Status == "Found only in TWSE")
n_only_fm <- sum(comparison$Matching_Status == "Found only in FinMind (comparable universe)")
n_consistent <- sum(comparison$Matching_Status == "Found in both - consistent")
n_differs <- sum(comparison$Matching_Status == "Found in both - information differs")
n_incomplete <- sum(comparison$Matching_Status == "Found in both - missing field(s)")
n_name_diff <- sum(comparison$Name_Check == "Mismatch", na.rm = TRUE)
n_ind_diff <- sum(comparison$Industry_Label_Check == "Different label", na.rm = TRUE)
n_unmatched <- n_only_twse + n_only_fmtibble(
Metric = c("Total TWSE listed common stocks",
"Total comparable FinMind stocks",
"Stock IDs found in both sources",
"Unmatched Stock IDs (only in one source)",
" - only in TWSE",
" - only in FinMind",
"Both sources, all compared fields consistent",
"Both sources, information differs",
"Both sources, a field is missing",
"Company names that differ (both present)",
"Industry labels that differ (both present)",
"TWSE missing industry / FinMind missing industry",
"FinMind listing date available"),
Value = as.character(c(n_twse, n_fm, n_both, n_unmatched, n_only_twse, n_only_fm,
n_consistent, n_differs, n_incomplete, n_name_diff, n_ind_diff,
paste(sum(is.na(twse_out$Industry)), "/", sum(is.na(fm_clean$Industry))),
if (has_fm_listing) "Yes" else "No"))
) |>
kable(caption = tab_cap("Comparison summary (calculated from the data)"))| Metric | Value |
|---|---|
| Total TWSE listed common stocks | 1053 |
| Total comparable FinMind stocks | 1215 |
| Stock IDs found in both sources | 1053 |
| Unmatched Stock IDs (only in one source) | 162 |
| - only in TWSE | 0 |
| - only in FinMind | 162 |
| Both sources, all compared fields consistent | 1046 |
| Both sources, information differs | 7 |
| Both sources, a field is missing | 0 |
| Company names that differ (both present) | 7 |
| Industry labels that differ (both present) | 747 |
| TWSE missing industry / FinMind missing industry | 0 / 0 |
| FinMind listing date available | No |
comparison |>
count(Matching_Status, name = "Stocks") |>
show_tbl("Matching status of all Stock IDs")| Matching_Status | Stocks |
|---|---|
| Found in both - consistent | 1,046 |
| Found in both - information differs | 7 |
| Found only in FinMind (comparable universe) | 162 |
comparison |>
filter(Name_Check == "Mismatch") |>
select(Stock_ID, TWSE_Company_Name, FinMind_Company_Name) |>
show_tbl("All stocks whose company names differ between TWSE and FinMind",
"No company-name differences were found.")| Stock_ID | TWSE_Company_Name | FinMind_Company_Name |
|---|---|---|
| 1443 | 立益物流 | 立益 |
| 6757 | 台灣虎航 | 台灣虎航-創 |
| 6794 | 向榮生技 | 向榮生技-創 |
| 6869 | 雲豹能源 | 雲豹能源-創 |
| 6873 | 泓德能源 | 泓德能源-創 |
| 6902 | GOGOLOOK | GOGOLOOK-創 |
| 8422 | 可寧衛* | 可寧衛 |
comparison |>
filter(Industry_Label_Check %in% c("Same label", "Different label")) |>
count(TWSE_Industry, FinMind_Industry, name = "Stocks", sort = TRUE) |>
head(15) |>
show_tbl("Industry label crosswalk: most frequent TWSE / FinMind label pairs")| TWSE_Industry | FinMind_Industry | Stocks |
|---|---|---|
| 電子零組件業 | 電子工業; 電子零組件業 | 103 |
| 半導體業 | 半導體業; 電子工業 | 89 |
| 光電業 | 光電業; 電子工業 | 66 |
| 電腦及週邊設備業 | 電子工業; 電腦及週邊設備業 | 62 |
| 生技醫療業 | 化學生技醫療; 生技醫療業 | 52 |
| 其他業 | 其他 | 50 |
| 建材營造業 | 建材營造 | 50 |
| 電機機械 | 電機機械 | 49 |
| 通信網路業 | 通信網路業; 電子工業 | 44 |
| 其他電子業 | 其他電子業; 電子工業 | 43 |
| 紡織纖維 | 紡織纖維 | 42 |
| 汽車工業 | 汽車工業 | 39 |
| 金融保險業 | 金融保險 | 31 |
| 鋼鐵工業 | 鋼鐵工業 | 31 |
| 化學工業 | 化學工業; 化學生技醫療 | 28 |
# FinMind-only IDs: what kind of securities are they?
unm_fm <- comparison |> filter(In_FinMind & !In_TWSE)
unm_fm |>
count(FinMind_Industry, name = "Stocks", sort = TRUE) |>
head(15) |>
show_tbl("FinMind-only Stock IDs by FinMind industry label (largest 15)",
"Every FinMind stock is also in the TWSE list.")| FinMind_Industry | Stocks |
|---|---|
| 光電業; 電子工業 | 22 |
| 半導體業; 電子工業 | 19 |
| 金融保險 | 15 |
| 其他 | 10 |
| 電子工業; 電腦及週邊設備業 | 9 |
| 化學生技醫療; 生技醫療業 | 6 |
| 電子工業 | 6 |
| 電子工業; 電子零組件業 | 6 |
| 通信網路業; 電子工業 | 5 |
| 電子工業; 電子通路業 | 5 |
| 其他電子業; 電子工業 | 4 |
| 創新板股票; 創新版股票; 半導體業; 電子工業 | 4 |
| 創新板股票; 化學生技醫療; 生技醫療業 | 4 |
| 化學工業; 化學生技醫療 | 4 |
| 電器電纜 | 4 |
unm_fm |>
count(First_Digit = str_sub(Stock_ID, 1, 1), name = "Stocks") |>
show_tbl("FinMind-only Stock IDs by first digit of the code",
"Every FinMind stock is also in the TWSE list.")| First_Digit | Stocks |
|---|---|
| 1 | 19 |
| 2 | 48 |
| 3 | 33 |
| 4 | 8 |
| 5 | 5 |
| 6 | 29 |
| 7 | 9 |
| 8 | 8 |
| 9 | 3 |
# The FinMind `date` field is NOT a listing date; it is shown only as a descriptive diagnostic.
date_diag <- comparison |>
filter(In_FinMind) |>
mutate(Group = if_else(In_TWSE, "In both sources", "Only in FinMind"),
D = suppressWarnings(as.Date(FinMind_Date_Field_Unverified)))
if (any(!is.na(date_diag$D))) {
date_diag |>
group_by(Group) |>
summarise(Stocks = n(), With_Date = sum(!is.na(D)),
Earliest = min(D, na.rm = TRUE), Median = median(D, na.rm = TRUE),
Latest = max(D, na.rm = TRUE), .groups = "drop") |>
show_tbl("Descriptive summary of the FinMind `date` field (not a listing date and not used as one)")
} else {
knitr::asis_output("*The FinMind `date` field is not available in this dataset.*")
}| Group | Stocks | With_Date | Earliest | Median | Latest |
|---|---|---|---|---|---|
| In both sources | 1,053 | 1,053 | 2026-07-04 | 2026-10-11 | 2026-10-11 |
| Only in FinMind | 162 | 162 | 2024-12-03 | 2026-06-02 | 2026-10-11 |
# TWSE-only IDs: how did FinMind's own data and filter treat them?
comparison |>
filter(In_TWSE & !In_FinMind) |>
left_join(fm_reason_lookup, by = c("Stock_ID" = "stock_id")) |>
mutate(FinMind_Filter_Outcome = coalesce(FinMind_Filter_Outcome,
"Not present in the FinMind TaiwanStockInfo table")) |>
count(FinMind_Filter_Outcome, name = "Stocks", sort = TRUE) |>
show_tbl("TWSE-only Stock IDs: treatment in the FinMind data",
"Every TWSE common stock is also in the FinMind list.")Every TWSE common stock is also in the FinMind list.
All unmatched stocks are saved in Unmatched_Stocks.csv
with the source in which they were found and, where relevant, how the
FinMind filter treated them.
qa_cmp <- bind_rows(
qa("Duplicate Stock IDs in the comparison table", sum(duplicated(comparison$Stock_ID)), "zero"),
qa("Stock IDs in both sources", n_both, "positive"),
qa("Stock IDs found only in TWSE", n_only_twse),
qa("Stock IDs found only in FinMind", n_only_fm),
qa("Company names that differ (both present)", n_name_diff),
qa("Company names missing in one source (both IDs present)", sum(comparison$Name_Check == "Missing", na.rm = TRUE)),
qa("Industry labels that differ (different taxonomies)", n_ind_diff),
qa("Industry labels missing in one source (both IDs present)", sum(comparison$Industry_Label_Check == "Missing", na.rm = TRUE)),
qa("Listing dates comparable between the sources (1 = yes, 0 = no)", as.integer(has_fm_listing))
)
kable(qa_cmp, caption = tab_cap("Comparison quality checks (PASS / REVIEW / INFO)"))| Check | Result | Status |
|---|---|---|
| Duplicate Stock IDs in the comparison table | 0 | PASS |
| Stock IDs in both sources | 1,053 | PASS |
| Stock IDs found only in TWSE | 0 | INFO |
| Stock IDs found only in FinMind | 162 | INFO |
| Company names that differ (both present) | 7 | INFO |
| Company names missing in one source (both IDs present) | 0 | INFO |
| Industry labels that differ (different taxonomies) | 747 | INFO |
| Industry labels missing in one source (both IDs present) | 0 | INFO |
| Listing dates comparable between the sources (1 = yes, 0 = no) | 0 | INFO |
Coverage. The official TWSE list yields 1,053 listed common stocks and FinMind yields 1,215 comparable stocks. 1,053 Stock IDs appear in both sources; 0 appear only in TWSE and 162 only in FinMind. Of the shared IDs, 1,046 are fully consistent on the compared fields, 7 differ and 0 have a missing field. The unmatched IDs are listed in Section 10.2 and in Unmatched_Stocks.csv. The FinMind-only IDs could not be confirmed as common stocks listed on the TWSE by this comparison, so they are flagged rather than silently accepted or removed. Possible explanations are securities that FinMind classifies differently from TWSE, records that are no longer listed, and listings or delistings that one provider has not yet reflected; these explanations were not verified individually.
Availability of fields. TWSE supplies all four requested fields. FinMind supplies stock ID, name and industry; no field documented as a listing date was returned, so the listing-date comparison is not possible and no value was substituted.
Classification. TWSE assigns industry through its own official classification, whereas FinMind uses its own category labels (and may attach more than one label to a stock). 747 shared stocks have different industry labels (Section 10.1), which reflects differing taxonomies rather than data errors. Company names differ for 7 shared stocks, typically because of naming conventions or the timing of name changes.
Limitations.
date field was not treated as one.All automated checks of this report are collected below. A run without R errors does not by itself show that the data are valid; the statuses show where a result deserves inspection.
qa_all <- bind_rows(qa_twse |> mutate(Block = "TWSE"),
qa_fm |> mutate(Block = "FinMind"),
qa_cmp |> mutate(Block = "Comparison")) |>
select(Block, Check, Result, Status)
n_pass <- sum(qa_all$Status == "PASS")
n_review <- sum(qa_all$Status == "REVIEW")
n_info <- sum(qa_all$Status == "INFO")
kable(qa_all, caption = tab_cap("Consolidated quality-assurance results"))| Block | Check | Result | Status |
|---|---|---|---|
| TWSE | Securities in the raw all-category table | not available in this run | INFO |
| TWSE | Rows in the input table used | 1,053 | INFO |
| TWSE | Listed common stocks after filtering | 1,053 | PASS |
| TWSE | Rows removed by the common-stock filter (0 expected for an already-cleaned file) | 0 | PASS |
| TWSE | Category marker available in the input (1 = yes, 0 = no) | 0 | INFO |
| TWSE | CFI marker available in the input (1 = yes, 0 = no) | 1 | INFO |
| TWSE | Duplicate Stock IDs removed | 0 | PASS |
| TWSE | Duplicate Stock IDs in the final table | 0 | PASS |
| TWSE | Missing Stock ID | 0 | PASS |
| TWSE | Missing company name | 0 | PASS |
| TWSE | Missing listing date | 0 | PASS |
| TWSE | Listing dates that could not be parsed (text present, date invalid) | 0 | PASS |
| TWSE | Listing dates in the future | 0 | PASS |
| TWSE | Listing dates before 1962-01-01 (before TWSE trading began) | 0 | PASS |
| TWSE | Earliest listing date | 1962-02-09 | INFO |
| TWSE | Latest listing date | 2026-09-01 | INFO |
| TWSE | Missing industry | 0 | INFO |
| TWSE | Retained rows whose CFI code is not ‘ES…’ | 0 | PASS |
| TWSE | Retained rows whose CFI code is preferred ‘EP…’ | 0 | PASS |
| TWSE | Retained Stock IDs that are not exactly 4 digits | 0 | PASS |
| TWSE | Retained rows with a market value other than the listed market | 0 | INFO |
| TWSE | Retained rows that refer to the Taiwan Innovation Board (market or industry label) | 0 | INFO |
| TWSE | Company names containing Chinese characters (%; checks the text encoding) | 99.2 | PASS |
| FinMind | Rows in the raw FinMind table | 4,333 | INFO |
| FinMind | Rows with market type TWSE | 2,402 | INFO |
| FinMind | Comparable common stocks retained (one row per Stock ID) | 1,215 | PASS |
| FinMind | Duplicate Stock IDs in the retained table | 0 | PASS |
| FinMind | Missing company name | 0 | PASS |
| FinMind | Missing industry | 0 | INFO |
| FinMind | Retained Stock IDs that are not exactly 4 digits | 0 | PASS |
| FinMind | Retained rows with an ETF / ETN / index / DR / fund / warrant label | 0 | PASS |
| FinMind | Retained rows labelled as Taiwan Innovation Board stocks | 37 | INFO |
| FinMind | Retained stocks confirmed by the official TWSE list | 1,053 | INFO |
| FinMind | Retained stocks NOT confirmed by the official TWSE list (see Section 10) | 162 | INFO |
| FinMind | FinMind listing date available (1 = yes, 0 = no) | 0 | INFO |
| FinMind | IDs in data_cache/finmind_clean_saved.csv (earlier run) | 1,215 | INFO |
| FinMind | IDs in the earlier file but not in this run’s result | 0 | INFO |
| FinMind | IDs in this run’s result but not in the earlier file | 0 | INFO |
| Comparison | Duplicate Stock IDs in the comparison table | 0 | PASS |
| Comparison | Stock IDs in both sources | 1,053 | PASS |
| Comparison | Stock IDs found only in TWSE | 0 | INFO |
| Comparison | Stock IDs found only in FinMind | 162 | INFO |
| Comparison | Company names that differ (both present) | 7 | INFO |
| Comparison | Company names missing in one source (both IDs present) | 0 | INFO |
| Comparison | Industry labels that differ (different taxonomies) | 747 | INFO |
| Comparison | Industry labels missing in one source (both IDs present) | 0 | INFO |
| Comparison | Listing dates comparable between the sources (1 = yes, 0 = no) | 0 | INFO |
Overall, 21 checks passed, 0 are marked for review and 26 are descriptive. No check requires review.
Task 1 was completed from the official TWSE ISIN registry
(strMode=2): the cleaned list was re-validated against the
page category and the CFI code, retaining only common stocks. This gives
1,053 stocks with ID, name, listing date and industry
(TWSE_Common_Stocks.csv). Task 2 was completed with the
FinMind TaiwanStockInfo dataset, giving
1,215 comparable stocks
(FinMind_Common_Stocks.csv), with unavailable fields
reported rather than imputed. ETFs, funds and preferred stocks are
excluded from both datasets. The comparison
(TWSE_FinMind_Comparison.csv) shows 1,053 shared Stock IDs
and quantifies the differences in coverage, naming and classification
between the two providers; the stocks found in only one source are
listed in Unmatched_Stocks.csv.
TaiwanStockInfo): https://api.finmindtrade.com/api/v4/data## Report generated: 2026-10-11 12:23## R version: R version 4.6.1 (2026-06-24 ucrt)## Project root: C:/Users/acer/Downloads/HW4## Files written: TWSE_Common_Stocks.csv, FinMind_Common_Stocks.csv, TWSE_FinMind_Comparison.csv, Unmatched_Stocks.csv