Be part of the occasion trusted by enterprise leaders for almost 20 years. VB Remodel brings collectively the individuals constructing actual enterprise AI technique. Study extra
Gaining visibility — and, finally, insights — into enterprise cloud belongings is rising ever more difficult.
Cloud estates are sprawling and fragmented, and stock capabilities in present instruments will be slender and unintuitive, separating parts like price and safety information into disconnected platforms with restricted flexibility.
Cloud governance firm CloudQuery is positioning itself to deal with this drawback by centralizing cloud belongings, safety metadata and price in a single place, and making it accessible via straightforward, built-in SQL queries and studies. The corporate is taking a developer-first method to cloud governance, pulling information from 60-plus sources — together with AWS, GCP, Azure, Okta and Wiz — right into a single, queryable information warehouse.
The corporate is now asserting a $15 million funding spherical led by Partech to additional scale its method to cloud visibility.
“The largest problem with present instruments is that they’re siloed — one for safety, one for price, one for asset stock — making it onerous to get a unified view throughout domains,” CQ founder Yevgeny Pats informed VentureBeat. “Even easy questions like ‘What EBS quantity is connected to an EC2 that’s turned off? are onerous to reply with out stitching collectively a number of instruments.”
CloudQuery underneath the hood
CloudQuery makes use of two key applied sciences underneath the hood: Information warehouse and open-source database ClickHouse and the Apache Arrow framework for growing information analytics functions.
This high-performance plugin structure inbuilt Go connects on to APIs like AWS, Azure, Google Cloud Platform (GCP) and plenty of different platforms pulling in configuration, safety, and price metadata. The platform constantly syncs information from dozens of cloud suppliers and providers right into a normalized, centralized asset stock.
“We place a robust emphasis on information accuracy and freshness, syncing at excessive frequency to make sure groups are working with essentially the most dependable, up-to-date data,” mentioned Pats.
That information, he defined, is structured relationally to energy CloudQuery’s SQL engine and built-in studies, in order that groups can have full flexibility with out counting on black-box instruments.
The corporate additionally “selectively” makes use of giant language fashions (LLMs) for pure language querying, SQL era and proposals, “however all the time on high of a basis of correct, clear information,” mentioned Pats. He identified that as a result of AI understands SQL effectively, instruments like Claude and OpenAI can create personalized studies and evaluation in plain English.
Taking a developer-first method is important, mentioned Pats, as a result of builders are finally those constructing, working and securing right now’s cloud infrastructure. Nonetheless, many cloud visibility instruments have been constructed for top-down governance, not for the individuals truly within the trenches.
“Whenever you put builders first, with accessible information, versatile APIs and native language like SQL, you empower them to maneuver quicker, catch points earlier and construct extra securely,” he mentioned.
Prospects are discovering methods to make use of CloudQuery past asset stock. “Many begin with visibility, then shortly develop into use circumstances like compliance monitoring, safety posture administration, price optimization, all from the identical core platform,” mentioned Pats.
How Hexagon constructed a serverless information lake for all its cloud shops
One enterprise already seeing outcomes is Hexagon. The software program firm’s cloud heart of excellence (CCoE) crew had a objective to construct a completely serverless information lake that would acquire information from all of its cloud accounts and retailer it in a single information lake.
In addition they needed the power to question this information utilizing SQL and visualize it with instruments they have been accustomed to (reminiscent of AWS QuickSight), and discover the historical past of their cloud configuration over time.
The crew constructed a serverless information pipeline utilizing CloudQuery to gather information from all accounts and retailer it in S3. AWS Glue then ingests information into Glue DB in a format that Amazon Athena can question, which Athena then does, then visualises in QuickSight.
“Having a completely serverless answer was an vital requirement,” Hexagon cloud governance and FinOps professional Peter Figueiredo and CloudQuery director of engineering Herman Schaaf wrote in a weblog submit. “This choice introduced a lot of advantages since there is no such thing as a want for time-consuming updates and just about zero upkeep.”
They did have to beat some challenges, significantly with Amazon S3 assist plugins. The CCoE crew was one of many first to check out CloudQuery options within the S3 vacation spot and provided insights resulting in new options. These embody:
- Parquet assist: The CloudQuery file vacation spot initially solely supported CSV and JSON information codecs. Errors in JSON interpretations led CloudQuery so as to add Parquet assist.
- Information partitioning: A CloudQuery file vacation spot plugin now permits partitioning on preliminary write (which beforehand wasn’t accessible, leading to additional pointless steps).
- Useful resource view for Athena: CloudQuery initially solely provided a sources view for AWS appropriate with Postgres. However Athena didn’t assist this, so CloudQuery added a operate that may retrieve a listing of all tables to construct or replace a sources view.
Figueiredo’s crew used CloudQuery to interchange AWS’s VPC IP handle supervisor (IPAM) — which he referred to as costly and restricted in that it doesn’t cowl different cloud suppliers.
In the end, his crew runs CloudQuery in ‘information lake’ mode utilizing “extremely low cost infrastructure” together with AWS S3, ECS, Glue, Athena and Lambda,” Figueiredo informed VentureBeat. This retains prices low and permits Hexagon to merge all its IP addresses throughout completely different cloud suppliers.
“We are able to shortly question any IP throughout the board and discover who the house owners are,” mentioned Figueiredo. “We at the moment are capable of acquire all we’d like at a really low price with close to zero upkeep. That is the holy grail for our crew.”
