Data Profiling Is Just The Start

Data profiling is the periodic X-ray of tables in a database to gather extensive information about the contents of each column. Results are stored in a standard table in DataOps TestGen. This table is available for direct review and is used to derive downstream rules.

Examples include:
• Averages
• Column & Table Types & Names
• Date Characteristics
• Min/Max Value
• Numeric Counts:
• Percentiles
• Positions
• Unique Values

Once data profiling is complete, Data Hygiene Detection Tests automatically confirm how closely data structures and assumptions match the actual contents of each column. Results can be used to assist the Data Engineer in refining data structure definitions and target the addition of data ‘patching’ steps, which help to generate a more usable, analyzable dataset. 

Examples Include:

• Invalid Zip Code Format
• Leading Spaces
• Mostly Dates In String
• Mostly not null, empty, or filled values.
• Multiple Data Types Per Column Name
• No Column Values Present
• Non-standard Blank Values

The goal of Automatically Generated Data Tests is to cast a wide net for data problems that can’t be predicted by targeted testing devised in advance.  It’s the same way you might set up a burglar alarm in your home by deploying sensors at all possible entrances to catch a burglar who would only try one window.  Your goal in refining these tests is to maintain maximum sensitivity to real problems while minimizing false positives that are not worth following up on.   

Examples of Test Are:

• Alpha Truncation
• Average Shift
• Constant Value Present
• Daily Record Count
• Value present in List-of-Values
• Distinct Value Change
• Value present in List-of-Values
• Future Date
• Incremental Average Shift

Business Rule Configurable Data Tests allow you to configure data quality validation tests that can’t be gleaned automatically from prior data. It is faster and easier to set up Business Rule Configurable Data Tests than to program custom SQL. Business Rule and Data Test logic are already programmed, tested, and verified to work. To collaborate on rules and documentation, they can be configured and shared with business users in the UI.  User-created configurable Data Tests let you create reusable data quality validation tests tailored to your data sets and customers.

Examples include:

• Data Match
• Prior Match
• Aggregate Match No Drops

Data quality testing is the start, not the end. DataKitchen Open Source DataOps Observability monitors every data journey-from source to customer value-so that problems are detected, localized, and understood immediately..

DataKitchen: Predictable & Sustainable
• Transparent Pricing: The Enterprise version is just $100 per user per connection. Predictable costs that scale reasonably.  Typical Vendors: Opaque Pricing & Price Increases
• No Venture-Backed Uncertainty: We won't suddenly pivot pricing models or sunset features. We are a profitable, stable, reliable partner committed to customer success.