See the Example: Vector Layers from an ArcGIS REST Feature Server topic for background, since that has similar workflow.
Feature servers tend to be very slow. The way to bypass that slowness is to bite the bullet and to scrape the entire table and drawing from the server into a local table and drawing. You can then do all your selections and such locally without waiting around to see what the server does.
Working with data left on the server to do any sort of analytics, even something as simple as selection, can put slowness on top of slowness, because that asks the server to do more.
When you want to select just a subset of items, Manifold sends a query to the server to find objects matching the selection criterion, like where you clicked or the bounding box of a selection marquee. It's not just a matter of looking at what is in cache, because there might have been a change, or there might be some overlap, or the server might have choked and not sent everything (internally timed out) and so on.
Take a look at the log window, and you might see some surprising results. For example, I connected to the server in the Example topic above, zoomed into a smaller area, and ctrl-clicked one area to select it. That took a long time to select. I then chose Edit - Copy to copy it, and it took forever to get that copy of just one record... why? Because Manifold sent a query to the server for what might have been under that clicked location (could have been more than one area there...), and the server timed out or failed at the query.
Looking at the log window, I saw:
2020-11-16 11:41:28 *** (root)::[kansas]::[Current Kansas Field Production] (Query) The remote server returned an error: (504) Gateway Timeout.
2020-11-16 11:43:33 *** (root)::[kansas]::[Current Kansas Field Production] (Query) The remote server returned an error: (504) Gateway Timeout.
2020-11-16 11:43:54 *** (root)::[kansas] Error (400): Unable to complete operation. ["Unable to perform query. "]
2020-11-16 11:43:54 Render: [Map] (20.607 sec)
2020-11-16 11:45:35 *** (root)::[kansas]::[Current Kansas Field Production] (Query) The remote server returned an error: (504) Gateway Timeout.
2020-11-16 11:45:35 -- Copy: [kansas]::[Current Kansas Field Production Drawing] (1 record(s), 94.533 sec)
Errors like "(504) Gateway Timeout" are entirely server-side. Something within the ESRI server infrastructure timed out.
The log window talks about a query because to copy selected items, there's a query for that since things might have changed since the selection. Manifold is making sure to get the latest version of the object from the server.
I don't know if it makes sense to have a mode where copies and such come only out of whatever is in local cache: one important reason to use servers is that they can provide real time data on what is in the layer (if that is not important, it's easier to just download the data without messing with a server). Looking just at cache is not getting real time data. But then again, doing a copy/paste and thereafter working locally is also not doing that.