Data Nessie Roadmap
This is the Data Nessie Roadmap for 2020 to date but is subject to change. Get in touch if you need a missing feature or would like a different prioritisation. We have no shareholders or banks to please, just our customers!
Release Notes
- Feature (1.0.1): Accept/Reject Disclaimer
- Feature (1.0.2): All meta data to be stored in local postgres database
- Feature (1.0.3): Event logging to database table
- Feature (1.0.4): All connection details in AWS secret manager
- Feature (1.0.5): Passwords to be encrypted before using in ETL pipeline
- Feature (1.0.6): No data stored on the AMI, everything processed in flight
- Feature (1.0.8): Data sets copied directly to S3
- Feature (1.0.9): Support MySQL + Serverless
- Feature (1.0.11): Postgres is the database for the meta data
- Feature (1.0.12): Customer will need to upload the MySQL driver via s3 to Data Nessie for data source.
- Feature (1.0.13): Customer will be able to select from a list of input/output driver versions
- Feature (1.0.14): Backup/Restore of Meta Data to/from S3 for Upgrade
- Feature (1.0.15): Move to S3 sync to AWS CLI to avoid S3 PDI 2GB limit
- Feature (1.0.16): Glue Generation
- Feature (1.0.17): Automated Testing of Data Nessie pipelines
- Feature (1.0.18): GUI Test page for access e.g. policies
- Feature (1.0.20): Log SQL against source system
- Feature (1.0.21): Make date available as a row version column, as with strings and numbers
- Feature (1.0.22): GUI check if data nessie bucket is in use with option to change.
- Feature (1.0.23): GUI checks if bucket region is same as crawler region
- Feature (1.0.25): GUI sets the root s3 location
- Feature (1.0.26): GUI initialisation
- Feature (1.0.28): Ship with default MySQL driver from MariaDB
- Feature (1.0.29): Handle overlapping runs from the same pipeline
- Feature (1.0.30): Only supporting v5 of MySQL Driver
- Feature (1.0.32): Add Source Row Hash for change detection; datanessie_md5
- Feature (1.0.33): Parallel running of pipelines
- Feature (1.1.19): Output format parquet
- Feature (1.1.1): Error Handling – Athena missing database error in log.
- Feature (1.1.2): Driver changes for MySQL
- Feature (1.1.4): Get load limit into the SQL statement to handle huge tables
- Feature (1.1.5): Trigger pipeline via rest request
- Feature (1.1.6): Automated Spot Generation
- Feature (1.1.7): Automated Reconciliation
- Feature (1.1.9): Update Amazon AWS Linux
- Feature (1.1.10): Encrypted S3 drive
- Feature (1.1.11): Alerts to AWS notification service
- Feature (1.1.12): Handle MS SQL Server
- Feature (1.1.13): Handle Rowversion column type for changes
- Feature (1.1.14): Add next runtime to the pipeline
- Feature (1.1.16): Tidy the metadata logging tables
- Feature (1.1.18): Line level regular expression to remove/release illegal characters that break Athena from all columns
- Feature (1.2.1): AWS Kinesis support for deltas
- Feature (1.2.2): Spot position time travel
- Feature (1.2.3): Update Amazon AWS Linux
- Feature (1.2.4): Handle deletes
- Feature (1.2.5): Reconcile between spot and operational positions
In planning!