Microsoft Dataflex/Dataverse/CDS - What is it and what does this mean?

NOTE: I updated this today (Aug 5th 2020) after meeting w/ some Microsoft representatives around what we currently know about Dataflex.

NOTE: As of August 12, 2020, it appears this will be renamed from "Dataflex".  Perhaps Microsoft staff used Bing to search for the term instead of Google and didn't realize that trademark was in use.

NOTE: As of November, 2020, they've re-rebranded the CDS to be Dataverse.

Microsoft announced yesterday (July 22, 2020) a new (old-ish) method for accessing/storing data for "Teams apps" which still brings some questions of what in the world a "Teams app" is and/or what is included in what license and what we essentially can do with it.

I don't have a lot of answers beyond what Microsoft has shared to date per Dataflex:

• it is included in base Teams licenses
• "Dataflex" is a subset of "Dataflex Pro"
• "Dataflex Pro" is a rebrand of the CDS ($$$)

And while I have been driving down a road toward divorcing PowerApps and Microsoft Forms from the ultimate data storage repositories and using semistructured data formats (i.e. JSON) for much of my recent recommendations, this seems to be an attempt to truly pin down some of the issues around deploying, supporting, and ultimately governing some aspects of the o365 landscape within Teams.

What is it?

Well, it's essentially a trimmed down version of SmartSheets in some respects.  In another, it is a basic RDB (MS-SQL) tied to individual Teams.  A method to both store data, but also to interact with it in a more collaborative way.  Excel gives us tons of methods for manipulating data, but it isn't remotely good at doing so across multiple users.  SmartSheets (and its various competitors) have started this discussion a little differently and are more about the process/flow of data.

This is a data storage method (based upon the old CDS which is based upon SQL Server) that simplifies the security model by embedding it into Teams.  Of course, you'd immediately shoot back with: SharePoint Lists already do this for free!

You aren't wrong.  In fact, the main reason we see precious little active integration for SharePoint Lists into Teams and now this "out of nowhere" play to embed a data storage medium into Teams is about Microsoft trying to move us in a direction where they want us to go...

One trick here is that what SharePoint doesn't do (well) is embed business rules within it.  And that, my friends, is where Microsoft is aiming.

Why ditch SharePoint?

Well, if you're asking that question, then you don't use SharePoint enough or you use it too much.  SharePoint is part of the same kind of problem that many premise-based manufacturers had when they migrated toward cloud and SharePoint Online shows how Microsoft is wrestling w/ the features limitations required for a full browser-based experience.  SharePoint is a behemoth and the migration to cloud and ultimately to a more streamline data storage experience has been troublesome.

W/ the move from premise->cloud for all manufacturers, optimizations that used to be the "end-user's problem" (i.e.throw more RAM/CPU at it) are now the cloud host's.  This has forced lots of organizations who never cared about bloated code or little-used features (I'm looking at YOU Oracle) to re-assess entire products from top->bottom.

Microsoft took the data from their applications to recognize what features were more essential and drove toward a destination where less features were exposed/supported within the web environment. Then, with the opportunity to solve a different problem using Microsoft Teams (blending security models w/ business models) took it further and restricted the use-cases even more.

What this has shown is that SharePoint, with all of its benefits, is also a fat pig of a framework.  And while Teams heavily uses SharePoint behind the scenes for everything it does, Microsoft is acknowledging that it isn't especially good at delivering/managing/optimizing data.  And when you have tens (if not hundreds) of millions of daily users querying thing across tens of thounsands of enterprises, this matters.

Microsoft has purpose-built their CDS into a streamlined data capture workflow to learn/optimize business process flow.  Starting w/ Customer Relationship Management, they extended this workflow use-case into the broader enterprise and build a data structure specifically designed to capture/categorize/analyze that data.  Or at least, that's what they "purpose building" toward.  And they want us all to help them build it out (and pay for it).

Why not use Dataflex Pro / CDS?

This all comes down to my prior statements per the CDS and how it is a cash grab.  There's a definite value proposition within it that orgs can recognize and sign up for, but it's a tough check to write until you see the clear benefits.  Right now, people are just wanting to stash their data "somewhere".  They don't need a pre-built business data framework that they'll never fully adopt because it duplicates too many other disparate systems.  It is a massive cliff of costs that any Enterprise CIO would shudder at beholding, let alone jumping off of.

Plus, I don't think any of us trust that Microsoft won't re-shuffle the license deck around tomorrow on the CDS.  In fact, this new "lite" version of the CDS is essentially that.

Part of this move is the recognition that people aren't adopting their optimized storage medium.  They're still using SharePoint because it's free.  So now, here's another free option that we'd much rather you use so here's a bunch of other free stuff to try and get you to do it.

I don't think -Pro/CDS are valueless, I merely point out that all of us are kind of wondering where the true value resides and if our organizations can afford to invest in the hopes that it will pay off.

NOTE: As of fall 2022, I have no intention of ever using Dataverse because the distance to value from the cost of investment are large enough that it is hard to point to any organizational payout that will happen before I retire (in 15+ years).  There's value here, just not for my organization to the degree that I'm willing to waste my time chasing it.

So what are the features/restrictions/capabilities?

Who knows.  Probably not even the Microsoft Product Management team at present.

NOTE: After meeting w/ Microsoft, this is 100% true.

This marketing/branding/licensing effort is a focus on adoption of the technology and the storage medium (Dataflex Pro / CDS).  I would expect we will see some restrictions on sizing of tables and automation.  I do not expect we will see restrictions on integrations WITHIN TEAMS.  Meaning, that if we all stay inside of Teams for all aspects of our application, then we won't hit (m)any walls.

NOTE: Again, after meeting w/ Microsoft, this is 100% true.  I'm starting to get a little cocky.

You can see this from some of the graphs that appear to show that PowerApps is an extension beyond what we'd consider a "Teams App" (definition still TBD).  And yet, I also believe that they'll bundle the term "Teams App" to include even custom-coded appliations (i.e. .NET) and Virtual Agents (a still fuzzy licensing model - but one that will likely include access to the non-Pro Dataflex data objects).

I'd also expect the term "Teams App" to include some of the core Dataflex features which actually will solve some of the issues w/ PowerApps embedded Teams interfaces.  As well, to put in some low-end features that are both basic, but helpful to end-user facing interactions within Teams.  Also, to make it simpler to automate and embed business rules directly into the Dataflex UI (vs. being forced to operate inside Power Automate).  What's I'd guess is the generic term "Teams App" will ultimately reflect more of a licensing model vs. a technical one (beyond it just being restricted to operating within Teams).

NOTE: Again, after meeting w/ Microsoft, this is 100% true.  Do I secretly work for Microsoft while I sleep?!

I did just have a meeting w/ some of the Microsoft staff and my statements above are likely truer than I even guessed.  That "PowerApps" will now become more of a "general PowerApp" and a "Teams PowerApp".  Where if you agree to limit your PowerApp to working only with Teams, you get free access to this data store.  Move it outside, and you lose this capability (unless you pay $).  Starts to also show you why (perhaps) certain obvious features don't work if/when you try to embed PowerApps into Teams.  Mainly because they want to push you into putting your business rules (not just your data) into their data store.

Some technical aspects are outlined on a blog here.  Read through that and ponder what it all means.  They are collapsing these into their own environments that are also all based on a single Team.  Teams generally represent an individual department or a cross-departmental process.  It sure would be simple to feed an AI engine a series of business rules that have been built into the same data structures already pre-defined into a single process.  I wonder where Microsoft thinks the value (for them) exists here...or don't..in fact...wonder at all.  Because it's fairly obvious why they'd want to know the rules about how/why/where each of our business processes make decisions and have them pre-formatted within their own structured data store.

Final Thoughts

These aren't so much final as I'd say they're where I stand as of this moment.  I'm not completely against attempting to utilize the Dataflex components within Teams.  There are a lot of quirky "how do I" kind of steps within Teams to capture the user experience in a more simplified UI.  

There remain questions around Adaptive Cards and best practices & methods for hosting dynamic content and functions to populate them.  However, it seems like Microsoft is acknowledging this black hole and this might be the beginning of them solving for that.  By including/embedding the data source within the Teams space, it might simplify the deployment for this.

I am not nearly so concerned w/ changing licenses around the specific item of Dataflex, but more of the ancillary "Teams Apps" that is still very nebulous.  There clearly are some licensing requirements around the Virtual Agents that need to be reviewed and how/where they make sense to include within processes.  Again, after a discussion w/ Microsoft about this, the term "Teams Apps" is going to mainly be a licensing discussion over a purely technical one.  That certain apps can/will exist in/out of Teams and the licensing of them will likely be different (more free in Teams, less free outside of Teams).

I'd like to see where the holes are within this model per developing an end-to-end solution and storing the data within Dataflex.  I think it will smooth out the SharePoint List hack to some degree, but it's clear that this is intended to move things even further along.  So I'd like to understand how much of this road is built and how much of this is planned vs. how much of this is pure vapor.