Large organizations have always struggled to get visibility into their data. Frequently, data comes in many sources and formats while also being siloed across the organization. DLTs add ledger data and metadata to the mix.
Similarly, consortiums need to interoperate with each other. Yet organizations often use disparate tools for logs, metrics and tracing. All these are deployed on different clouds or on-prem. Then they build their own tools to take data from the ledgers and put it into a SQL database. Consortiums may include competitors who don’t trust one another, so they don’t want to share data. But, if no one shares data, it can be challenging to determine if an organization has a problem or if it lies in the network.
Splunk, a company focused on removing barriers between data and action, has enabled customers to have complete monitoring and observability into their own data by bringing it all into one place, regardless of format.
Splunk is now offering open source solutions allowing the ingestion of ledger data and corresponding metadata while correlating with other data sources. Not only the blocks and transactions but also chaincode events. In a nutshell, Splunk has created a solution that allows any organization to answer any question, from security to observability. This is the launch of a new era in Hyperledger Fabric networks, where silos between organizations, infrastructure providers, and data sources are eliminated.
Splunk’s customers include 92 Fortune 100 companies. Before choosing a blockchain platform, it wanted to know what its customers were already using. And many of them used Hyperledger Fabric.
“The ecosystem of Hyperledger is enterprise-friendly,” says Nate McKervey, Head of Blockchain and DLT at Splunk. “It makes sense for Splunk to enable enterprises to turn data into doing when it comes to distributed ledger data, just like Splunk does for other data.
Bringing DLT data in wasn’t simple, but, because of Splunk’s origins, it wasn’t too difficult, either.
Splunk already ingests data without caring about structure, schema, or format. This feature saved the company the trouble of formatting Hyperledger Fabric data before ingestion.
That flexibility allowed developers to send data to Splunk and then figure out what to do with it. “It was really more of a question of what else can we collect besides the ledger data to help our customers ask any question they have,” says McKervey.
And to find out those questions, they asked their customers.
Blocks and transaction data allowed users to analyze and correlate that data with other data they had in Splunk. “Our customers said, ‘That’s great. But we need more than that. We want chaincode events,’” explains McKervey.
“When metrics were introduced, we needed to get metrics into Splunk. More recently, it’s been private data collection,” says McKervey.
Initially, the platform focused on uses from an IT perspective. If there are issues, can users drill down and figure out problems with less downtime? The company wanted users to feel confident going into production with Hyperledger Fabric.
Then its focus shifted to security. Customers wanted to know what else they could do to secure their infrastructure. Keeping it up and running was important, but so was making sure nothing would compromise their Hyperledger Fabric environment
One of those interested customers was S&P Global.
S&P Global delivers data, research, and credit ratings, among other things, to governments, companies, and individuals. Its Ratings division provides independent data and insight to the marketplace.
In 2019 it was entering a new region. S&P took this opportunity to explore modern technologies and new ways of doing business. It decided to build a content management solution from scratch with innovative technology and security.
“Our solution is essentially a blockchain based content management system,” explains Mark Wang, Global Head of Cloud Architecture at S&P Global Ratings. S&P securely stores and shares files with different stakeholders. These might be regulators, external entities, and internal users. But this content includes critical and sensitive information. So the solution needed to provide permanency for records, and it needed to protect the security of these crucial, powerful data assets
“As a credit rating agency, we’re heavily regulated. We want to be multi-cloud. We need to satisfy regional data localization requirements. And data security is highly critical,” Wang says. “We needed a secure solution that’s tamper proof and immutable.”
For over a decade, S&P had been using Splunk for its infrastructure monitoring. “Everything we deploy has an automation with Splunk,” says Wang. At a Splunk conference in 2019, S&P saw the Hyperledger Fabric-based applications Splunk was developing. “We saw enormous potential for some of our emerging use cases,” says Wang. “Since we already had the platform, it made perfect sense to leverage it with Hyperledger Fabric.”
It also helped that Hyperledger Fabric is a private permission blockchain and is enterprise ready.
S&P could now get user interactions and metadata—like who uploaded documents or modified documents and when. S&P developed applications to retrieve that metadata and present it to the user for document searches. This opened up possibilities for providing an audit trail.
“Say a regulator comes to us and needs a complete audit trail of how a document that impacted the market was generated,” explains Wang. “They need to know who modified it, who saw it, things like that. We can put together all those pieces and connect those dots with the data the logger is capturing from the chain network.”