There has been a great deal of interest of late in data governance and in the identification and definition of data elements of material business importance and in the stewardship of those elements so as to define the meaning, structure, and format of the data elements.

We have gone so far as to install data governance software to memorialize our definitions and provide applications and data structure designers with their content to be used in the development of edit and validation routines.  What we have often overlooked is creating an approach and methodology for defining the order in which the rules should be applied.

Edit and validation rules are the rules we instantiate in the code base to ensure that the data that we are using to create and update shared data structures are accurate, complete, authorized, timely and support our needs for being an ongoing concern (e.g., the well-defined, accepted and understood control objectives for data).  The problem has become more complicated as multiple applications update the same data structure(s).  The edit and validation code base must instantiate the edit and validation rules, the rules defined by the data governance function, in the same or at least analogous. It must be noted that each of the specific languages used to build the applications may have different syntaxes and hence require different logic structures to accomplish the same tasks.

To this end, we have created a taxonomy of edit and validation rules which defines the order in which they should be applied, an order that represents the greatest intrinsic value.  The taxonomy will aid in our understanding of how edit and validation are instantiated in an operational/transactional applications:

1. Syntactical – the data elements is evaluated regarding its data type, structure, and adherence to the standard established and documented in the data governance repository:
   • Standard Data Types – in Third and some Fourth generation compiled languages Strong Data Typing is used (this is done so the Compiler can establish the Symbol Table, allocate storage and most important know how to execute logic and arithmetic calculations). Examples of traditional data types are:
     1. CHAR – Character, with the additional need to define the size, e.g., CHAR 50, Fifty characters long; fixed, blank filled left justified, it could also be VAR – Variable and Right justified; the programmer must define the syntactical specifications
     2. Numeric – size, and some decimals, e.g., NUMERIC 20.5 which would be 20 positions in length with 5 of them being to the right of the decimal point.

• Vectors, Arrays, and other non-scaler structures

• Abstract Data Typing – a new concept that allows the programmer to define new types of data structures and operations; for example, to create as n-dimensional Matrix of images or sounds. Other types of abstract data typing are unstructured data (e.g., movies, documents, sound).  These unstructured data types require edit and validation rules, methods and techniques that are very different than structured data.

2. Semantical –
   • Intra-transactional Semantical – Semantic Editing as compared to Syntactical or Data Type editing uses the value or meaning of the data that has been input, the Occurrence Data. A few examples may make this more understandable:
     1. Zip Code Plus 4 provides, via a look up to a file provided by the Post Office, the correct address, including accepted standard abbreviations (e.g., RD for Road, PL for Place)
     2. Code Look Ups for company specific abbreviations and codes, e.g., SKU, catalog numbers and acceptable dimensions for shipping

• Codes published by governmental agencies and standards groups

1. Economic order and ship quantities and related standard references

• Inter- or Historical Inference – in this case, the Semantic correctness of the input is evaluated based on historical behaviors. For example; if the customer has ordered the same SKU 20 time in the last 6 months and has consistently ordered a dozen of that SKU, and the order now comes in for 10 dozen (dozen being the order qauntity established by 2 above) the edit and validation algorithm, based on looking up prior behavior can make the inference that the customer wanted 1 dozen and asked for confirmation at the time of data entry.  This inference editing has become a standard differentiator for several internet providers; they remember your purchases and make recommendations or check the semantic of your order against their historical records.  It is also the basis for fraud detection in online transactions and credit cards, you cannot purchase in Tokyo and five minutes later make one in New York, the historical and geographical constraints do not work.

In summary, data governance function provides the rules needed to determine what the correct and acceptable definition, at the syntactic and semantic level of a data element, edit and validation code is how the rules get instantiated in the operational ecosystem.  Inconsistencies in instantiation result in the degradation of the accuracy, completeness, authorization, timeliness, and ability to support an ongoing concern requirement.  Or, to put it more colloquially, if the rules are not consistently applied across the ecosystem and in agreement with the data governance definitions the data is WRONG, and the business is at risk.

To resolve these conflicts organizations have established Methods, Standards, Design Techniques, Procedures and implemented Architectural Paradigms to enforce data governance definitions and result in edit and validation code that is consistent across multiple applications and data structures.

Like what Philip Teplitzky has to say? He has a monthly column on Fintech Today, which you can subscribe to here.