I have used a db.table() function with better-sqlite3 successfully. But, I am now trying to implement an async function and get an error
db.table('Foo', {
columns: [ 'col' ],
rows: async function* (text) {
const res = await someCall(text);
for (const col of res) yield [ col ];
}
});
/Users/punkish/Projects/zai/node_modules/better-sqlite3/lib/methods/table.js:53
throw new TypeError(`Virtual table module "${moduleName}" ${verb} a table definition with an invalid "rows" property (should be a generator function)`);
^
TypeError: Virtual table module "Foo" used a table definition with an invalid "rows" property (should be a generator function)
update: I discovered that better-sqlite3 does not support async generator functions in virtual tables. I am trying to write an after-insert trigger on a table that takes text entered into it, (async) splits it into chunks, and (async) generates embeddings from an LLM model running on my computer. Doing it as a trigger would make my data-loading process extremely simple. Any suggestions on how I can work around the better-sqlite3 limitation? The following code, that I would like to implement, is not supported
db.table('SplitText', {
columns: [ 'chunk', 'embedding' ],
rows: async function* (cleanedText) {
const chunks = await splitter.splitText(cleanedText);
const model = 'nomic-embed-text';
const source = { model, input: chunks };
const res = await ollama.embed(source);
const embeddings = res.embeddings;
for (let i = 0, j = embeddings.length; i < j; i++) {
yield [ chunks[i], embeddings[i] ];
}
}
});