- I’m currently working at Caixa Econômica Federal as a mid-level Data Engineer.
- The tools I use in my day-to-day work are Databricks,SQL Server, SSIS, Power BI, and Python.
- I have a bachelor's degree in Economics from the Federal University of Pernambuco(UFPE) and an MBA in data engineering.
- I know Python, SQL, and Java languages.
- I’m interested in all the Data stack, Financial Markets, and Homebrewing.
- How to reach me: www.linkedin.com/in/stéfano-vivas
New features proposed:
- Add ability to profile and generate rules on subset of the input data by introducing a sql expression filter (#569)
- Improve summary stats report for string datatype columns (#670)
- Extend summary metrics with per-check-name breakdowns (#943)
- Introduces a set of checks called is_null, is_empty, and is_null_or_empty to allow checks that confirm whether a given column is null, empty, or both (#965)
- Added profile build for "has_no_outliers" check (#977)
- Extended
dq_generate_min_maxmethod to support Python'sDecimaltype in addition tointandfloatfor min/max validation checks (#1013)
New features implemented:
- Introduces a new data quality check to detect outliers for numeric columns using the Median Absolute Deviation method (MAD) (#359)
- Add ability to profile and generate rules on subset of the input data by introducing a sql expression filter (#569)
- Added rule_fingerprint, rule_set_fingerprint, and created_at to checks storage, so now it can be versioned (#672)
- Introduces a new dataset-level quality check, aggr_matches_dataset, to validate ingestion correctness by comparing an aggregate metric (row count by default, or other curated/built-in aggregates) computed on the checked DataFrame against the same metric computed on an dataset reference (#1309)