create a parquet file with struct inside it

Guys, I'm trying to create a parquet file which has to contain different informations. The problem I'm facing is parquet let me write just infos taken from tables, instead I want to store inside it a struct.

I try to explain better. For example I have a blf file which I have to read with dedicated command (blfread) and then insert the result of blfread inside parquet. Whats I want is having this blfread file as a struct contained in the parquet file, not a table of thousand rows and lot of columns.

From what I know it's something I might achieve using sqlite to create a proper database, but I received infos about using parquet database to manage these situations because it should be more efficient.

Is it really something which can be done or I just have to use sqlite? Final goal is having as result a file database of one row, where every column must contain different infos. One will cotain blf compressed, another will contain infos i might write in a table, etc..

Answer 1

[Kautuk Raj]

Parquet is a columnar storage format that is optimized for reading and writing large amounts of structured data efficiently. While it is possible to store a struct in a Parquet file, it may not be the most efficient or recommended way to store data.

If you want to store a struct in a Parquet file, you will need to convert the struct to a table or a record batch before writing it to the file. You can convert a struct to a table using the struct2table function in MATLAB, and then write the table to a Parquet file using parquetwrite function.

However, if you have different types of data to store (e.g., binary data, text data, and structured data), it is usually better to store them in separate columns or tables within the Parquet file. You can then use SQL queries to join the tables together or extract the data you need.

Alternatively, as you mentioned, you can use a relational database like SQLite to store your data in a more flexible and scalable way. SQLite allows you to store structured data in tables as well as unstructured data like binary blobs, and you can use SQL queries to retrieve the data in any format you need.

[Massimo]

I'm interested in last thing you said. I have to manage files like blf (the one generated by canalyzer). How to store it inside a sqlite database?

[Kautuk Raj]

You can create a table with columns to store the BLF data as a binary blob (BLOB data type), as well as other columns for the related information. You can achieve this using Python's sqlite3 library or SQLite in MATLAB.

[Massimo]

Thanks for the answer. Could you give me more detailed infos? Because I'm absolutely new to matlab, so when you say i can store BLF as BLOB I don't know how to do it. What is this python you mentioned?

I need to do everything with matlb, not other softwares