Timeline for answer to Bonobo ETL: How to provide global context to all nodes during execution? by Romain
Current License: CC BY-SA 4.0
Post Revisions
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 10, 2019 at 7:21 | comment | added | Romain | I think you'd be better of by having some kind of "file metadata" object created from the file object, handle the file cleanly in one node and only use the (readonly) metadata in the other nodes. Using python file objects with random open/close timings and overall file handle availability in nodes that run . in an unpredictable order would for sure bring you into trouble at some point. | |
| Jun 7, 2019 at 14:59 | comment | added | Finn Andersen | It's because each node is a transformation that may want to use attributes associated with the file object, as well as each record | |
| Jun 6, 2019 at 15:15 | comment | added | Romain | I think the approach work, but not after the record extractor. I can't see why you would want the file object after this place, especially because sharing objects between threads like this can be (well, is) very dangerous. Also, I would tend to pass filenames instead of file objects from one node to another so the node responsible for file operations can handle it correctly (like a context manager, with a finally: close()) | |
| Jun 6, 2019 at 14:28 | comment | added | Finn Andersen | Thanks, looks like the Services approach would work. I dont want the context to be completely global, but just accessible by all nodes. What if the first node was a file generator (yields file objects) then second node is record extractor (yeilds records from file), and I want to be able to access the corresponding file object in every node after the File Generator node? MIght be easiest to have the file-generator loop outside Bonobo graph completely and just have first node as Record Extractor, and provide file as service to graph? | |
| Jun 6, 2019 at 14:04 | history | answered | Romain | CC BY-SA 4.0 |