Ideas about data


- Import of data
- Export of data


- Import
    - Generally the Code needs to know the format of the data
    - For a general file-format, like JSON, XML or SQLite, the code needs to know the schema
    - There is the data itself, and there is the schema
    - The schema could possibly be determined from the data or be defined externally to the data
    - The actual format of the data is orthogonal to the schema of the data needed for the code to access it
    - As such the reading of the data could be abstracted to load from a data-source API with
      implementations for JSON, XML, CSV or SQLite as data sources, and the code written to work with a given schema
    - Validation would be preferable - validation the data has the expected schema
    - Instead of writing for a given schema, have the code generated for the schema
    - Is it possible to not generate code at all but load a schema definition and have generic code? I am thinking
      not in C++, but perhaps needs thought

    - Import could be streaming or could be in-memory where entire data set is loaded at once
    - Streaming is required for very large data sets which can't fit in memory at once

- Export
    - Similar to import, orthoganality of format to schema. The output equivelant to a source is a sink.
    - So would have a data-sink API with multiple implementations to write it out

    - As with import, export could be streaming or all at once.
    - For SQLite, it might be journalled.



- Code Generation Example

   ./db-code-gen.sh test.sqlite > test-interface.h