Data Nessie Roadmap

This is the Data Nessie Roadmap for 2020 to date but is subject to change. Get in touch if you need a missing feature or would like a different prioritisation. We have no shareholders or banks to please, just our customers!

Release Notes

  • Feature (1.0.1): Accept/Reject Disclaimer
  • Feature (1.0.2): All meta data to be stored in local postgres database
  • Feature (1.0.3): Event logging to database table
  • Feature (1.0.4): All connection details in AWS secret manager
  • Feature (1.0.5): Passwords to be encrypted before using in ETL pipeline
  • Feature (1.0.6): No data stored on the AMI, everything processed in flight
  • Feature (1.0.8): Data sets copied directly to S3
  • Feature (1.0.9): Support MySQL + Serverless
  • Feature (1.0.11): Postgres is the database for the meta data
  • Feature (1.0.12): Customer will need to upload the MySQL driver via s3 to Data Nessie for data source.
  • Feature (1.0.13): Customer will be able to select from a list of input/output driver versions
  • Feature (1.0.14): Backup/Restore of Meta Data to/from S3 for Upgrade
  • Feature (1.0.15): Move to S3 sync to AWS CLI  to avoid S3 PDI 2GB limit
  • Feature (1.0.16): Glue Generation 
  • Feature (1.0.17): Automated Testing of Data Nessie pipelines
  • Feature (1.0.18): GUI Test page for access e.g. policies 
  • Feature (1.0.20): Log SQL against source system
  • Feature (1.0.21): Make date available as a row version column, as with strings and numbers
  • Feature (1.0.22): GUI check if data nessie bucket is in use with option to change.
  • Feature (1.0.23): GUI checks if bucket region is same as crawler region
  • Feature (1.0.25): GUI sets the root s3 location
  • Feature (1.0.26): GUI initialisation
  • Feature (1.0.28): Ship with default MySQL driver from MariaDB
  • Feature (1.0.29): Handle overlapping runs from the same pipeline
  • Feature (1.0.30): Only supporting v5 of MySQL Driver
  • Feature (1.0.32): Add Source Row Hash for change detection; datanessie_md5
  • Feature (1.0.33): Parallel running of pipelines
  • Feature (1.1.19): Output format parquet
  • Feature (1.1.1): Error Handling – Athena missing database error in log. 
  • Feature (1.1.2): Driver changes for MySQL 
  • Feature (1.1.4): Get load limit into the SQL statement to handle huge tables
  • Feature (1.1.5): Trigger pipeline via rest request 
  • Feature (1.1.6): Automated Spot Generation 
  • Feature (1.1.7):  Automated Reconciliation 
  • Feature (1.1.9):  Update Amazon AWS Linux 
  • Feature (1.1.10):  Encrypted S3 drive 
  • Feature (1.1.11):  Alerts to AWS notification service 
  • Feature (1.1.12):  Handle MS SQL Server 
  • Feature (1.1.13):  Handle Rowversion column type for changes 
  • Feature (1.1.14):  Add next runtime to the pipeline 
  • Feature (1.1.16):  Tidy the metadata logging tables 
  • Feature (1.1.18):  Line level regular expression to remove/release illegal characters that break Athena from all columns 
  • Feature (1.2.1): AWS Kinesis support for deltas 
  • Feature (1.2.2): Spot position time travel 
  • Feature (1.2.3): Update Amazon AWS Linux 
  • Feature (1.2.4): Handle deletes 
  • Feature (1.2.5): Reconcile between spot and operational positions 

Copyright 2020 Millersoft Ltd, all rights reserved. Terms of Use

Data Nessie