How Procore is powering construction analytics with OpenSharing

How Procore is powering construction analytics with OpenSharing | Databricks Blog Skip to main content

Summary

Procore replaced an API-driven, single-daily-refresh analytics product with OpenSharing on Databricks, lifting the once-a-day refresh ceiling so customers can pull data for their downstream systems and business processes hourly.

OpenSharing now distributes hundreds of curated tables, maintaining partitioned access per customer to manage hundreds of thousands of distinct data shares at scale.

Doubled data availability for end customers and internal product teams, accelerating new product experiences and giving real estate facility owners, general contractors, and specialty contractors a single trusted view of every job.

Connecting an industry built on disconnected data Construction runs on data that rarely sits in one place. Information is scattered across platforms, regions, and teams, often locked inside the systems of whichever party generated it. When field data isn't connected to the financial system, that disconnect can hide financial risk. When a project manager, a construction executive thousands of miles away, and a superintendent on the jobsite can't see the same view of a project, decisions get made based on stale or partial information. Procore, a leading global construction management software provider serving tens of thousands of customers across the U.S., Europe, the Middle East, Asia, and Australia, was built to close those gaps — connecting everyone in construction on a single platform. That means three groups in particular: owners (real estate developers and anyone owning and operating the real estate facility or infrastructure themselves), general contractors who manage projects end to end, and specialty contractors like electricians and plumbers who execute the work in the field. But connecting data inside Procore is only half the story. Customers also need that data in their own environments, alongside everything else they run the business on (e.g. a general contractor pulling Procore data into Power BI to track workforce and personnel metrics across active jobs, or a customer joining it with their ERP and financial systems to see project execution and cost in one place). That is what Procore Analytics delivers: programmatic access to the connected Procore dataset, pulled into whatever the customer already uses — Power BI, Tableau, Snowflake, BigQuery, their own Databricks workspace, or on-prem SQL Server — for reporting and downstream analysis. Evolving beyond the first generation of Procore Analytics The first generation of Procore Analytics predated OpenSharing entirely. It was built on a SQL Server backend. Data was extracted from Procore’s public APIs and loaded into an Azure database that customers could query. The architecture worked, but it had a hard ceiling. Refreshes were limited to a single daily run, and even customers who paid for additional refreshes were capped at a maximum of three per day. For construction projects where conditions change by the hour, that lag was the single biggest limiting factor. API-driven extractions also created cost and reliability problems at scale. Pulling data for tens of thousands of customers through public APIs meant running deep recursive calls, which were expensive, brittle, and prone to timing out or hitting limits. Why OpenSharing By the time Procore began designing the next generation of Procore Analytics, Databricks was already powering other parts of the Procore platform. That made OpenSharing a natural fit for the customer-facing extract product as well, with several specific advantages over the old API path. Operational efficiency. Sharing curated files in place removes the cost, brittleness, and timeout risk of running deep recursive API calls for every customer, the same pressure that capped refresh frequency in Procore Analytics 1.0. Scale. Procore now can share hundreds of curated tables to each customer, maintaining partitioned access per customer to manage hundreds of thousands of distinct data shares at scale. Some of those companies are running hundreds or thousands of concurrent projects, and every one of them needs a current view of the work. Reach. Procore's customers run on every major cloud and every IT stack imaginable, from Excel and Power BI to fully built-out Databricks environments. OpenSharing's open protocol meets them all. BI tools like Power BI and Tableau work out of the box, and customers who already run on Databricks get a native Databricks-to-Databricks experience without moving any data. Customers on other warehouses and platforms can consume the same shares using the open OpenSharing client libraries. Reliability and performance. Customers consume curated, governed data that is already cleaned, modeled, and quality-checked, rather than chaining together API calls and stitching the results. Security and governance. Sharing runs through Unity Catalog, so every table is governed centrally and access is enforced per user and per project. A general contractor can share project progress with the owner while keeping its own financials private. The implementation Procore built Procore Analytics 2.0 on a medallion architecture inside Databricks. Raw events from the Procore application database and integration events land in the bronze layer untouched. Silver applies cleaning, validation, and data quality checks, producing the confirmed datasets that everything downstream depends on. Gold is where the data is aggregated, optimized, and modeled for analytics. Bronze, silver, and gold all run across multiple regions so processing happens close to the source data. Gold tables are then pulled from each region into a single primary analytics region, which is where OpenSharing fans the data out to customers. Two recipient paths. Customers without Databricks generate an OpenSharing token directly inside the Procore application and connect using the open protocol. Customers who do have Databricks add their workspace identifier to receive shares natively through Databricks-to-Databricks sharing. Automated entitlements. A scheduled job watches for new shares and permission changes, reads each user's permissions out of the Procore application, and reconciles the OpenSharing entitlements accordingly. The result is per-user, per-project enforcement with no manual provisioning, so a general contractor can share project progress with the owner while keeping...