Add an option to copy files on query deployment
Description
Conclusion
Attachments
Activity

Anthony Fishbeck February 15, 2022 at 6:39 PMEdited
I agree that we need to be able to track the progress of all the copying. As mentioned this isn't currently possible when roxie is copying, and users do complain. My POC is currenly planned to create DFU copy workunits. The final version (after POC?) being a single DFU workunit with all the related files to copy in 1.
Yes. Well what currently gets copied (meta data and then roxie background copies) is the intersection of the packagmap and the set of files used by the currently deployed queries covered by the map. That potentially leaves out many of the files in the packagemap. So there are really 2 levels of improvements that could be made when copying files for the packagemap.
1. When deploying a packagmap, copy all files, not just for published queries.
2. Separately - copy all files in a packagemap without publishing the packagemap.
I suspect users will find both useful.
Looking further ahead we would probably end up wanting some way of figuring out what files are no longer needed and would likely need a way of not reporting "pre-deployed" files as not needed. Either by keeping the packagemaps around even when not available to roxie, or by having the user provide the addtional packagemaps when running the unused file report.

Gavin Halliday February 15, 2022 at 5:57 PM
see linked jira. A couple of items that came up from a discussion.
This would be very useful because when it completes the user would know that all the files are copied. (Currently when the roxie is copying the files it is hard to know when it completes)
It may be useful to have an option to copy the data for a package deployment, but not actually copy the packages. (I'm not sure if this actually makes sense....)
*
Details
Components
Assignee
Anthony FishbeckAnthony FishbeckReporter
Gavin HallidayGavin HallidayPriority
MajorFix versions
Labels
Pull Request URL
Roadmap
Not applicableDue date
Jun 30, 2022
Details
Details
Components
Assignee

Reporter

Currently when publishing a query there is an option to copy the meta information, but no option to copy the physical files. In cloud environments this would probably a useful option. (The option is available in the base calls, but defaults to false.)
The same should go for the other places which add roxie meta data (queryset copying and package map deployment).
Note: Care should be taken to ensure the files are copied efficiently and in parallel. Currently the code would copy them on esp, but it may be better to use a load balanced dafilesrv, or directly use the cloud apis where possible.