Panzer: Cleanup of Workset interface

Created by: seamill

Motivation and Context

Currently the workset is more or less a massive storage container for everything we require, and much of it is stored in structure form (direct access to class members). The goal of this update is to add a function interface to access these calls, and to strip everything out of worksets that isn't required.

The main reason for doing so is to allow for on-demand allocation and filling of geometry/topological data arrays as required by the derived code. Currently the workset allocates and fills everything that could possibly be required by the user, which takes lots of memory, and a bit of compute time. By making these calls on-demand, the total allocations made will be greatly reduced for most applications.

Making the interface on-demand also cleans up the workset construction by not requiring the user to know what Basis/Cubature/Geometry/Topology is required by the derived FEM/FV/FD code during workset construction.

@trilinos/panzer

What do we keep/remove/redesign

Ideally, worksets would be a simple container that would look like:


class Workset
{
public:

  using Scalar = double;
  using Index = int;

  Index
  numDimensions() const;

  Index
  subcellIndex() const;

  Kokkos::View<const Scalar***>
  cellVertices() const;

  const  std::string &
  blockID() const;

  Kokkos::View<const Index*>
  localCellIndexes() const;

  const panzer::SubcellConnectivity & 
  getSubcellConnectivity(const int subcell_dimension) const;

  const panzer::IntegrationValues2<Scalar> & 
  getIntegrationValues(const panzer::IntegrationDescriptor & description) const;

  const panzer::IntegrationRule & 
  getIntegrationRule(const panzer::IntegrationDescriptor & description) const;

  panzer::BasisValues2<Scalar> & 
  getBasisValues(const panzer::BasisDescriptor & basis_description, 
                           const panzer::IntegrationDescriptor & integration_description) const;

  const panzer::BasisValues2<Scalar> & 
  getBasisValues(const panzer::BasisDescriptor & basis_description, 
                           const panzer::IntegrationDescriptor & integration_description) const;

  const panzer::BasisValues2<Scalar> & 
  getBasisValues(const panzer::BasisDescriptor & basis_description, 
                           const panzer::PointDescriptor & point_description) const;

  const panzer::PointValues2<Scalar> & 
  getPointValues(const panzer::PointDescriptor & point_description) const;

  const panzer::PureBasis & 
  getBasis(const panzer::BasisDescriptor & description) const;

  Index
  numCells() const;

  Index 
  numOwnedCells() const;

  Index 
  numGhostCells() const;

  Index 
  numVirtualCells() const;

  std::ostream & 
  operator<<(std::ostream& os, const panzer::Workset& w);

protected:
  ...
};

Naturally, there are a few issues with this. Having 'const' everywhere is required since worksets are passed around as const, but it will make things tricky when adding an on-demand feature. For now we can get away with having a shared pointer to an underlying class that does all the mutable stuff... assuming this doesn't cause issues with CUDA.

It also leaves the question of what to do with the rest of the interface. As far as I can tell, the above is missing the following (most of which probably shouldn't be in the workset):

Teuchos::RCP< std::vector<std::string> > basis_names;
void setNumberOfCells(int o_cells,int g_cells,int v_cells);
void setIdentifier(std::size_t identifier) { identifier_ = identifier; }
std::size_t getIdentifier() const { return identifier_; }

double alpha;
double beta;
double time;
double step_size;
double stage_number;
std::vector<double> gather_seeds; // generic gather seeds
bool evaluate_transient_terms;
Teuchos::RCP<WorksetDetails> other;

class WorksetDetailsAccessor;

I also don't know why there are multiple 'WorksetDetails' available in a workset.

If @eric-c-cyr and @rppawlo could run me though the reasoning behind the additional workset stuff, I would be grateful.

Definition of Done

The updates to workset are straightforward. Updating Drekar, Charon, and the various EMPIRE codes will be a bit tricky so we will break it into a set of phases. Updating Charon and Drekar may require some help from @jmgate. Updates to Empire will probably go smoother with @CamelliaDPG.

Phase 1

Workset interface changes:

Merge Workset and WorksetDetails into a single class.
Clean up and minimize interface based after discussing things with Eric and Roger.
Clean up workset construction interface (currently there are multiple conflicting ways to do this)
Deprecate the old interface
Deprecate old construction scheme
Update Drekar
Update Charon
Update EMPIRE

Phase 2

Workset on-demand changes:

Modify workset so that requesting things using Descriptors will spawn new BasisValues/PointValues/IntegrationRules/etc.
Remove 'old' deprecated interface
Remove alternative construction schemes

Note that we will allow the worksets to be fully allocated/filled if the user requests this using the construction interface (i.e. WorksetNeeds is passed to the construction scheme).

Phase 3

Update BasisValues2 class

Split BasisValues2 into two classes: BasisValues and BasisIntegrationValues
Add function interface to new classes
Implement on-demand calculations of various datasets
Deprecate old interface
Update Drekar
Update Charon
Update EMPIRE

Phase 4

Update IntegrationValues2 class

Phase 5

Update PointValues2 class

Phase 6

Update subcell connectivity interface

Add node/edge/face subcell connectivity to topological mesh analysis in Panzer (lots of questions here)
Add node/edge/face subcell connectivities to workset (on-demand)
Deprecate old interface
Update Drekar
Update Charon
Update EMPIRE

Phase 7

Cleanup

Once everyone stops yelling at me, we can go ahead and remove the deprecated interface