Engineering—May 13, 2020

Smartype: Proper event collection at run time

Smartype, a data quality product that translates any data model into type-safe code to help developers ensure proper event collection at run time. Smartype generates personalized SDKs, based on any data model, providing automated code completion and improving data collection and quality at scale. Now available in beta.

Overview

Great products (and user experiences) are based on great customer data, but ensuring your data is consistently correct, clean, and valid can be a major challenge—especially as organizations grow and their data scales.

Every development project starts with the best of intentions, but even the most disciplined development teams inevitably take shortcuts to meet project deadlines or make inadvertent implementation mistakes. You may find an attribute being collected as FirstName from one data source, as First_Name from another, and even as FirsBame from a third source—as one of our customers found out a few months after their go-live. Over time, small shortcuts, mistakes, and diverging data structuring can snowball into technical debt that overtakes developers’ time and makes it difficult to stay on the roadmap or determine if it is the correct roadmap. Meanwhile, product, analytics, and marketing teams are stuck cobbling data from multiple sources and systems into customer profiles that end up being outdated, incomplete, or just outright inaccurate.

Cross-team collaboration with Data Master

To help organizations establish a source of reliable customer data, earlier this year, we introduced Data Master, a set of UI features that help product, marketing, and engineering teams collaboratively design their data plans, and establish and enforce the desired data schema with validation to ensure accuracy and hygiene.

Data Plans instrumentation

But once all the teams have gone through the effort of creating a plan, developers still need a foolproof way to implement the plan. For developers manually referencing the data plan UI is no better than referencing a spreadsheet—both are slow and error-prone and do not accomplish change management.

Introducing Smartype

What if your data plan were represented in the form of code (classes, constants, functions)?

A code representation can naturally prevent data quality issues through static typing and by maintaining the set of constants called for in the data plan (event names, attribute names, enum values).

This is exactly why, today, we are introducing Smartype for developers—now in beta.

Smartype prevents data quality issues by eliminating incorrect instrumentation of identifiers (event names and attribute names) in your code. Smartype translates a Data Plan (or more generally, your Data Model) into usable code to improve your implementations. This means that each developer, or specifically each data plan, can have its own customized SDK, which has been specifically tailored for your data model(s).

Smartype is built for developers to naturally prevent data quality issues by giving them the following features:

Data Plans as code
All constants called for in the Data Plan (event names, attribute names, enum values) available as a machine-readable JSON Schema.
Data Planning API
Programmatically perform all CRUD operations on Data Plans to decrease the time-to-data-quality and time to implement.
Compile-time feedback
Feedback developers can use to conform to a Data Plan - thereby limiting the likelihood of an implementation issue.
Generate typesafe libraries
Programmatically generate a strongly typed client library across a variety of languages from a centrally defined Data Model.
Multiplatform code sharing
Built on Kotlin so that you can share your common Data Plan implementations across platforms (Android, iOS, web).
Managed within codebase
Any changes to Data Plans result in a regenerated typesafe library that is always in-sync with the latest Data Plan.

How it works

Smartype is a data quality feature set that helps you collaborate against a data model programmatically, implement it correctly at development time, report validations in time to fix them, and transform data points in the plan on an ongoing basis.

Every customer engagement at mParticle begins with a data planning exercise. A Data Plan is a set of expectations about the extent and shape of your data collected with mParticle.

Data plan hierarchy in mParticle

Data Plan API lets developers store their Data Plans in their source code, and lets them use their own SDLC and approval processes to define a data model. You can also perform all CRUD (Create, read, update and delete) operations on Data Plans programmatically and also include Data Planning tools to perform compile-time and runtime data quality verification.

Built on Kotlin, Smartype provides a mechanism for expected and actual type-safe declarations so that a common data model in JSON can define expected declarations, and a platform specific module can provide actual declarations mapped to the expected ones.

Here's a simple example of how Smartype turns any JSON schema into code:

Let's look at an Android implementation of an mParticle data plan:

val api = SmartypeApi(listOf(MParticleReceiver(), this))
val options = MParticleOptions.builder(this)
    .credentials("REPLACE WITH KEY", "REPLACE WITH SECRET")
    //SmartypeApi surfaces the data plan ID and version of the underlying models
    .dataplan(api.dataPlanId, api.dataPlanVersion)
    .build()
val message = api.chooseItem(
    ChooseItemData(
        ChooseItemDataCustomAttributes(
            quantity = 5.0,
            milk = true,
            item = ChooseItemDataCustomAttributesItem.CORTADO
        )
    )
)
//the message object will now be sent to mParticle's SDK
api.send(message)