Gaian Database – also known as GaianDB – originated in 2006 from a patented idea in IBM Hursley Emerging Technology Services (ETS) that defines a connection strategy for building a scale-free network. This became a reality in 2007 when GaianDB design and development started, funded jointly by MoD/DoD under the International Technology Alliance (ITA) fundamental research programme, and there have subsequently been many more ideas patented by the team to cover other aspects of this lightweight distributed data-federation technology. ITA Research goals were to create an information layer “IOT ready” small enough to be embedded anywhere and capable of dealing with tough conditions and unreliable network links in the field. In the following 4 years to 2011, GaianDB went through various phases of research, development and integrations as a result of ITA and other services engagements. From 2012, an effort to raise GaianDB code to product-quality level paved the way to its inclusion in several defence-related customer solutions and in 2 IBM products: “Application Performance Management”in the Tivoli division, and “PureApplication Software”in the Cloud division. In 2015, GaianDB was open-sourced to Github to further grow exposure and facilitate services engagements.
Here is a sample network of 30 nodes – We have experimented with networks up to 1024 nodes to verify that the diameter and query times increase logarithmically with the number of nodes.
Blue Unicorn
Following IBM’s restructuring in 2015 around CAMSS strategic imperatives (Cloud, Analytics, Mobile, Security and Social), IBM Analytics initiated a new programme, now known as “Blue Unicorn”, aimed at seeking out innovation from within IBM that would have business value and possibly lead to new product offerings. Blue Unicorn is structured in phases: Many project proposals initially enter the programme (138 in 2015) and then progressively get eliminated at each phase. The projects are initially voted on by IBMers and then in later phases they are reviewed by senior IBM executives. A key feature of the Blue Unicorn programme is to foster a culture of entrepreneurship, where small teams can focus on an exciting new project together, and with real-world rewards offered for successful innovation.
DataConfluence
Recognising the opportunity, ETS and Analytics teams collaborated on a proposal named “DataConfluence” to create a disruptive new cloud service with GaianDB at its core. With funding from both divisions the joint team was able to quickly construct an initial prototype which immediately demonstrated disruptive impact. This involved key high-value items that had been identified in the past by ETS and required a sizeable effort to implement – such as query optimisations, cloud service integration and a new UI.
A challenging aspect of the project has been having to balance long-term design and implementation against the “agile”requirement to produce regular demos to the Blue Unicorn review panel. A mixed approach has allowed us to simultaneously work towards product quality whilst retaining the backing of our stakeholders.
Angel Phase
At the time of writing, the project has entered the final “Angel” phase of Blue Unicorn. Most high-value technical items are well underway. Other progress includes benchmarking of our technology against other products in the field. Most importantly, we have engaged both internally with senior executives, fellows and distinguished engineers; and externally with a multitude of customers – e.g. at IBM’s “World of Watson”conference where we pre-arranged meetings with 2 customers and also established 9 new leads. Pitching our technology has provided great feedback and increased exposure. Above all, it has allowed us to continually refine requirements and use-cases, so we have an ever better understanding of what our technology is suited to or limited with. Our next target is to have a Beta ready for trials by the end of January, and we are encouraging as many customers as possible to participate.
Here is a screenshot of a recent demo showing Data Source Experience R Studio integration with DataConfluence – the analysis queries a network of 24 Raspberry Pis and completes in a fraction of the time (minimum factor 20) taken by alternate centralised or IoT solutions:
David Vyvyan is a Senior Software Engineer at IBM UK’s Hursley Laboratory. He has spent most of his IBM career (since 2004) as a member of the Emerging Technology Services department, working on a variety of projects, often related in some way to Distributed Data Analytics, such as Grid Computing. David’s main accomplishment, starting in 2007, is the creation of the Gaian Database (also known as GaianDB), which is a distributed data-federation technology with a self-organising network topology. David has taken GaianDB through many client engagements and also worked to include it in several IBM products.