Monthly house-price indices from web-scraped data: 13 countries for Eurostat

The problem

House prices move the macro economy, and policy makers watch them closely. But the official EU house-price indices (HPIs) are quarterly, built on transaction records, and arrive with a lag, while the asking prices on property portals update every day. The question Eurostat wanted answered: can web-scraped asking prices produce monthly indices sound enough to serve as indicator series for the official quarterly ones? I was technical lead of this study at KR&A, within the MMIT KR&A consortium contracted by Eurostat, leading the technical side for eight people across the two organisations, and the results are published as a Eurostat statistical working paper.

Thirteen countries of listings

The collection side ranked roughly 100 residential portals across the EU by geography, size, representativeness and scrapability, then approached them in that order. The final selection covers 13 countries, chosen with margin in case a portal stopped cooperating or became unscrapable: 9 portals scraped monthly and 4 delivering monthly files, from Denmark's Boliga to Cyprus's Bazaraki. Collection ran from May 2023 to April 2025 and assembled around 34.8 million offers, of which around 30.2 million were usable, on 6.7 million distinct properties.

Raw scraped listings are not statistics. Every record passes through deduplication, standardization (prices arrive in thousands, with decimal points or decimal commas, in English or the local language), cleansing, matching (so a property listed across several months is tracked as one property), and enrichment: records get latitude-longitude where the source does not provide it, and from there LAU and NUTS3 regions, degree of urbanisation and population density.

From listings to an index

The indices come from a log-price hedonic regression: regress log asking price on the property attributes plus a dummy per month, then exponentiate the time dummies to read off the index, with the first month set to 100. The idea is standard; making it run across sources that disagree about everything is the work.

I built the index-calculation system around per-source configuration files. Each config declares which variables enter the regression for that portal and what type each one is: continuous, discrete, or log-continuous. That matters for two reasons. Firstly, a continuous variable with poor coverage would silently shrink the sample, so low-coverage variables are dropped rather than allowed to throw away listings. Secondly, forcing a type is a modelling choice worth experimenting with: bedrooms as a discrete variable with an "unknown" dummy keeps sparse sources usable, but loses the ordinal meaning of the values, and the config system lets that trade-off be revisited per source without touching code. Forward variable selection then picks the composition where adding more variables stops improving the fit. Regressions run on the Eurostat-defined subsets: new versus existing dwellings, and dwellings versus plots. The whole system is generalized to the point that standing up an index for a new country takes hours rather than days.

The first two countries, chosen for their attribute coverage, were Cyprus and Denmark. The fits are decent for an object as heterogeneous as housing, and outright poor for plots, where both the sample and the usable attributes are thin. Real estate carries attributes no scrape can see, such as distance to the sea in Cyprus, and the fit statistics say so.

Do asking prices track transactions?

The point of the exercise is the comparison against the official transaction-based indices, and the early answer was: it depends on the country. The Cyprus asking-price index tracked the transaction-based index well. Denmark was less cooperative: the transaction index rose and then fell sharply, a movement the asking-price index did not reproduce, although lagging the asking-price series by three months made the fit look considerably better. Asking prices moving ahead of registered transactions is theoretically plausible, but with only four complete quarters of overlap at the time, no significance could be derived from it, and the report stated as much.

The final step tested the monthly indices as indicator series for temporal disaggregation: converting the quarterly official HPI into a monthly series using methods from the Denton family through Chow-Lin. The regression-based methods among them assume the two series are cointegrated, and that assumption could not yet be tested on four quarters of data, which is the honest boundary of what this phase could claim.

The published working paper extends the comparison to six quarters. Asking-price series can often reflect short-term movements in the transaction-based indices, although performance varies across countries and over time; the evidence does not support a universal one-quarter leading relationship at country level. Their main value lies in timely auxiliary information, most promising for European aggregates and for countries where portal coverage is strong, and the paper is explicit that these findings remain indicative rather than definitive.

What stayed with me

This project is where I learned what production data engineering means in official statistics: the regression is one page of the report, and the other forty are source selection, information density per attribute per portal, and the processing that turns 34.8 million scraped offers into 30.2 million usable ones. It is also the project that convinced me configuration belongs in the schema rather than in the code, a principle I have carried into my own research pipelines since.