Memory check failing for a Nalu application test
Created by: spdomin
I have a full-up production run now failing using the latest Trilinos version. This test reinitializes the linear solver each and evert tilmestep and, as usual, solves many systems over the simulation before it fails (repeatably) at step 490. A memory check of the test revealed the following suspect memory reports.
==48197== Conditional jump or move depends on uninitialised value(s)
==48197== at 0x1EDD3D1: void Kokkos::parallel_for<Tpetra::KokkosRefactor::Details::UnpackArrayMultiColumn<Kokkos::Experimental::View<double**, Kokkos::LayoutLeft, Kokkos::Serial, void>, Kokkos::Experimental::View<double const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void>, Kokkos::Experimental::View<int const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void>, Tpetra::KokkosRefactor::Details::InsertOp> >(unsigned long, Tpetra::KokkosRefactor::Details::UnpackArrayMultiColumn<Kokkos::Experimental::View<double**, Kokkos::LayoutLeft, Kokkos::Serial, void>, Kokkos::Experimental::View<double const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void>, Kokkos::Experimental::View<int const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void>, Tpetra::KokkosRefactor::Details::InsertOp> const&, std::string const&) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x1EDD4E3: Tpetra::KokkosRefactor::Details::UnpackArrayMultiColumn<Kokkos::Experimental::View<double**, Kokkos::LayoutLeft, Kokkos::Serial, void>, Kokkos::Experimental::View<double const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void>, Kokkos::Experimental::View<int const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void>, Tpetra::KokkosRefactor::Details::InsertOp>::unpack(Kokkos::Experimental::View<double**, Kokkos::LayoutLeft, Kokkos::Serial, void> const&, Kokkos::Experimental::View<double const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void> const&, Kokkos::Experimental::View<int const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void> const&, Tpetra::KokkosRefactor::Details::InsertOp const&, unsigned long) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x1F8CAEB: Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>::unpackAndCombineNew(Kokkos::DualView<int const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void> const&, Kokkos::DualView<double const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void> const&, Kokkos::DualView<unsigned long const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void> const&, unsigned long, Tpetra::Distributor&, Tpetra::CombineMode) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x217967E: Tpetra::DistObject<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>::doTransferNew(Tpetra::SrcDistObject const&, Tpetra::CombineMode, unsigned long, Kokkos::DualView<int const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void> const&, Kokkos::DualView<int const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void> const&, Kokkos::DualView<int const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void> const&, Kokkos::DualView<int const*, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, void, void> const&, Tpetra::Distributor&, Tpetra::DistObject<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>::ReverseOption, bool) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x2176C82: Tpetra::DistObject<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>::doTransfer(Tpetra::SrcDistObject const&, Tpetra::CombineMode, unsigned long, Teuchos::ArrayView<int const> const&, Teuchos::ArrayView<int const> const&, Teuchos::ArrayView<int const> const&, Teuchos::ArrayView<int const> const&, Tpetra::Distributor&, Tpetra::DistObject<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>::ReverseOption) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x2175AFB: Tpetra::DistObject<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>::doExport(Tpetra::SrcDistObject const&, Tpetra::Import<int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const&, Tpetra::CombineMode) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xF39239: Xpetra::TpetraMultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::doExport(Xpetra::DistObject<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const&, Xpetra::Import<int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const&, Xpetra::CombineMode) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x1555EF5: MueLu::Hierarchy<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::Iterate(Xpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const&, Xpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >&, MueLu::Hierarchy<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::ConvData, bool, int) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x1555BD2: MueLu::Hierarchy<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::Iterate(Xpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const&, Xpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >&, MueLu::Hierarchy<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::ConvData, bool, int) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x165D2F8: MueLu::TpetraOperator<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::apply(Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false> const&, Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>&, Teuchos::ETransp, double, double) const (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xFF84F8: Belos::LinearProblem<double, Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>, Tpetra::Operator<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >::apply(Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false> const&, Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>&) const (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xFFD34F: Belos::PseudoBlockGmresIter<double, Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>, Tpetra::Operator<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >::iterate() (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x100FEA5: Belos::PseudoBlockGmresSolMgr<double, Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>, Tpetra::Operator<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >::solve() (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xEF64E1: sierra::nalu::TpetraLinearSolver::solve(Teuchos::RCP<Tpetra::Vector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false> >, int&, double&) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x128DFBF: sierra::nalu::TpetraLinearSystem::solve(stk::mesh::FieldBase*) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x11442D2: sierra::nalu::EquationSystem::assemble_and_solve(stk::mesh::FieldBase*) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x118EC7E: sierra::nalu::LowMachEquationSystem::solve_and_update() (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xEDBBDC: sierra::nalu::EquationSystems::solve_and_update() (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xEA6D68: sierra::nalu::Realm::advance_time_step() (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xE6828F: sierra::nalu::TimeIntegrator::integrate_realm() (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xE61ACE: main (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== Uninitialised value was created by a heap allocation
==48197== at 0x4A068FE: malloc (vg_replace_malloc.c:270)
==48197== by 0x265DD35: Kokkos::HostSpace::allocate(unsigned long) const (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x265E7A9: Kokkos::Experimental::Impl::SharedAllocationRecord<Kokkos::HostSpace, void>::SharedAllocationRecord(Kokkos::HostSpace const&, std::string const&, unsigned long, void (*)(Kokkos::Experimental::Impl::SharedAllocationRecord<void, void>*)) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x1F7C149: Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>::classic>::dual_view_type (anonymous namespace)::allocDualView<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >(unsigned long, unsigned long, bool) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x1F8101E: Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>::MultiVector(Teuchos::RCP<Tpetra::Map<int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const> const&, unsigned long, bool) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xF7D20D: Xpetra::MultiVectorFactory<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::Build(Teuchos::RCP<Xpetra::Map<int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const> const&, unsigned long, bool) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x1555EC8: MueLu::Hierarchy<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::Iterate(Xpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const&, Xpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >&, MueLu::Hierarchy<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::ConvData, bool, int) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x1555BD2: MueLu::Hierarchy<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::Iterate(Xpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > const&, Xpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >&, MueLu::Hierarchy<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::ConvData, bool, int) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x165D2F8: MueLu::TpetraOperator<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> >::apply(Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false> const&, Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>&, Teuchos::ETransp, double, double) const (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xFF84F8: Belos::LinearProblem<double, Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>, Tpetra::Operator<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >::apply(Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false> const&, Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>&) const (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xFFD34F: Belos::PseudoBlockGmresIter<double, Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>, Tpetra::Operator<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >::iterate() (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x100FEA5: Belos::PseudoBlockGmresSolMgr<double, Tpetra::MultiVector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false>, Tpetra::Operator<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace> > >::solve() (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xEF64E1: sierra::nalu::TpetraLinearSolver::solve(Teuchos::RCP<Tpetra::Vector<double, int, long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial, Kokkos::HostSpace>, false> >, int&, double&) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x128DFBF: sierra::nalu::TpetraLinearSystem::solve(stk::mesh::FieldBase*) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x11442D2: sierra::nalu::EquationSystem::assemble_and_solve(stk::mesh::FieldBase*) (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0x118EC7E: sierra::nalu::LowMachEquationSystem::solve_and_update() (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xEDBBDC: sierra::nalu::EquationSystems::solve_and_update() (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xEA6D68: sierra::nalu::Realm::advance_time_step() (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xE6828F: sierra::nalu::TimeIntegrator::integrate_realm() (in /home/spdomin/gitHubWork/Nalu/build/naluX)
==48197== by 0xE61ACE: main (in /home/spdomin/gitHubWork/Nalu/build/naluX)