Add an option to copy files on query deployment

Description

Currently when publishing a query there is an option to copy the meta information, but no option to copy the physical files.  In cloud environments this would probably a useful option.  (The option is available in the base calls, but defaults to false.)

The same should go for the other places which add roxie meta data (queryset copying and package map deployment).

Note: Care should be taken to ensure the files are copied efficiently and in parallel.  Currently the code would copy them on esp, but it may be better to use a load balanced dafilesrv, or directly use the cloud apis where possible.

 

Conclusion

None
100% Done
Loading...

Activity

Show:

Anthony Fishbeck February 15, 2022 at 6:39 PM
Edited

  • I agree that we need to be able to track the progress of all the copying. As mentioned this isn't currently possible when roxie is copying, and users do complain. My POC is currenly planned to create DFU copy workunits. The final version (after POC?) being a single DFU workunit with all the related files to copy in 1.

  • Yes. Well what currently gets copied (meta data and then roxie background copies) is the intersection of the packagmap and the set of files used by the currently deployed queries covered by the map. That potentially leaves out many of the files in the packagemap. So there are really 2 levels of improvements that could be made when copying files for the packagemap.
    1. When deploying a packagmap, copy all files, not just for published queries.
    2. Separately - copy all files in a packagemap without publishing the packagemap.
    I suspect users will find both useful.
    Looking further ahead we would probably end up wanting some way of figuring out what files are no longer needed and would likely need a way of not reporting "pre-deployed" files as not needed. Either by keeping the packagemaps around even when not available to roxie, or by having the user provide the addtional packagemaps when running the unused file report.

Gavin Halliday February 15, 2022 at 5:57 PM

 see linked jira.  A couple of items that came up from a discussion.

  • This would be very useful because when it completes the user would know that all the files are copied.  (Currently when the roxie is copying the files it is hard to know when it completes)

  • It may be useful to have an option to copy the data for a package deployment, but not actually copy the packages.  (I'm not sure if this actually makes sense....)
    *

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Components

Assignee

Reporter

Priority

Fix versions

Labels

Pull Request URL

Roadmap

Not applicable

Due date

Created May 17, 2021 at 11:02 AM
Updated March 13, 2025 at 2:48 PM
Resolved June 28, 2022 at 9:26 AM