Allow packagemap to override remote storage used for lookup/copying of files

Description

https://hpccsystems.atlassian.net/browse/HPCC-28373#icft=HPCC-28373 adds support for using remote-storage instead of DALI for file lookups/copying when publishing queries or packagemaps.

Using the --remote-storage command line parameter.

The packagemap itself should be able to override the name of the remote storage being used for a particular set of files.

Conclusion

None

Activity

Show:

HPCC JiraBot September 24, 2024 at 10:09 PM

Jirabot Action Result:
Workflow Transition To: Merge Pending
Updated PR

Anantha Venkatachalam July 9, 2024 at 3:05 PM

Today:

For baremetal roxie:

  • package file is deployed via command line where a daliip is passed to tell where to pull the data from

  • we also override the daliip from within the package file

For AKS roxie:

  • we need to have the ability to pull data in hybrid mode meaning by default we use daliip in command line to pull data from baremetal

  • simultaneously we want to override the default dali from within the package file to pull data from data store

for example:

baremetal:

<package id=”testkeys” daliip=”remotedali”>

</package>

for aks roxie (what is needed):

<package id=”testkeys” remote-storage=”datastore”>

</package>

I would prefer the attribute name from within xml to match what is available in commandline what we have in command line is the following:

DFS Options:
--remote-storage Use the given remote storage configuration to locate remote files

Anthony Fishbeck July 9, 2024 at 2:49 PM

Reassigned to . Terrence, I can help you navigate the code and figure out what needs to be done.

The work will mostly be in the ws_packagemapprocess service and the ReferencedFileList utility class. can advise you on what his needs are. Basically, he wants to be able to add “remoteStorage“ attributes to various sections of the packagemap the same way he was able to add daliip attributes in the past.. overriding where the system will look for files within that section of the packagemap.

Let me know when you are ready, and we can discuss.

Jacob Cobbett-Smith July 8, 2024 at 8:30 AM

- is this something you can look at, or suggest who to reassign to?

Anthony Fishbeck September 26, 2023 at 5:19 PM

 

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Components

Assignee

Reporter

Priority

Compatibility

Point

Fix versions

Roadmap

Not applicable

Created September 26, 2023 at 5:17 PM
Updated January 22, 2025 at 8:16 PM
Resolved October 14, 2024 at 3:25 PM

Flag notifications