2812 Commits

Author SHA1 Message Date
Joerg Roedel
22bb182c83 iommu: Set default domain type at runtime
Set the default domain-type at runtime, not at compile-time.
This keeps default domain type setting in one place when we
have to change it at runtime.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-23 10:11:28 +02:00
Joerg Roedel
5fa9e7c5fa iommu: Print default domain type on boot
Introduce a subsys_initcall for IOMMU code and use it to
print the default domain type at boot.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-23 10:11:28 +02:00
Joerg Roedel
6b9a7d3a46 iommu/vt-d: Request passthrough mode from IOMMU core
Get rid of the iommu_pass_through variable and request
passthrough mode via the new iommu core function.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-23 10:09:58 +02:00
Joerg Roedel
cc7c8ad973 iommu/amd: Request passthrough mode from IOMMU core
Get rid of the iommu_pass_through variable and request
passthrough mode via the new iommu core function.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-23 10:09:58 +02:00
Joerg Roedel
adab0b07cb iommu: Use Functions to set default domain type in iommu_set_def_domain_type()
There are functions now to set the default domain type which
take care of updating other necessary state. Don't open-code
it in iommu_set_def_domain_type() and use those functions
instead.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-23 10:09:58 +02:00
Joerg Roedel
8a69961c7f iommu: Add helpers to set/get default domain type
Add a couple of functions to allow changing the default
domain type from architecture code and a function for iommu
drivers to request whether the default domain is
passthrough.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-23 10:09:58 +02:00
Joerg Roedel
faf1498993 iommu: Remember when default domain type was set on kernel command line
Introduce an extensible concept to remember when certain
configuration settings for the IOMMU code have been set on
the kernel command line.

This will be used later to prevent overwriting these
settings with other defaults.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-23 10:09:58 +02:00
Will Deacon
a91bcc2b65 Revert "iommu/arm-smmu-v3: Disable detection of ATS and PRI"
This reverts commit b5e86196b83fd68e065a7c811ab8925fb0dc3893.

Now that ATC invalidation is performed in the correct places and without
incurring a locking overhead for non-ATS systems, we can re-enable the
corresponding SMMU feature detection.

Signed-off-by: Will Deacon <will@kernel.org>
2019-08-22 18:16:19 +01:00
Will Deacon
cdb8a3c346 iommu/arm-smmu-v3: Avoid locking on invalidation path when not using ATS
When ATS is not in use, we can avoid taking the 'devices_lock' for the
domain on the invalidation path by simply caching the number of ATS
masters currently attached. The fiddly part is handling a concurrent
->attach() of an ATS-enabled master to a domain that is being
invalidated, but we can handle this using an 'smp_mb()' to ensure that
our check of the count is ordered after completion of our prior TLB
invalidation.

This also makes our ->attach() and ->detach() flows symmetric wrt ATS
interactions.

Acked-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-22 18:16:11 +01:00
Will Deacon
353e3cf859 iommu/arm-smmu-v3: Fix ATC invalidation ordering wrt main TLBs
When invalidating the ATC for an PCIe endpoint using ATS, we must take
care to complete invalidation of the main SMMU TLBs beforehand, otherwise
the device could immediately repopulate its ATC with stale translations.

Hooking the ATC invalidation into ->unmap() as we currently do does the
exact opposite: it ensures that the ATC is invalidated *before*  the
main TLBs, which is bogus.

Move ATC invalidation into the actual (leaf) invalidation routines so
that it is always called after completing main TLB invalidation.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-21 17:58:54 +01:00
Will Deacon
bfff88ec1a iommu/arm-smmu-v3: Rework enabling/disabling of ATS for PCI masters
To prevent any potential issues arising from speculative Address
Translation Requests from an ATS-enabled PCIe endpoint, rework our ATS
enabling/disabling logic so that we enable ATS at the SMMU before we
enable it at the endpoint, and disable things in the opposite order.

Reviewed-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-21 17:58:42 +01:00
Will Deacon
7314ca8699 iommu/arm-smmu-v3: Don't issue CMD_SYNC for zero-length invalidations
Calling arm_smmu_tlb_inv_range() with a size of zero, perhaps due to
an empty 'iommu_iotlb_gather' structure, should be a NOP. Elide the
CMD_SYNC when there is no invalidation to be performed.

Signed-off-by: Will Deacon <will@kernel.org>
2019-08-21 17:58:41 +01:00
Will Deacon
f75d8e33df iommu/arm-smmu-v3: Remove boolean bitfield for 'ats_enabled' flag
There's really no need for this to be a bitfield, particularly as we
don't have bitwise addressing on arm64.

Signed-off-by: Will Deacon <will@kernel.org>
2019-08-21 17:58:40 +01:00
Will Deacon
b5e86196b8 iommu/arm-smmu-v3: Disable detection of ATS and PRI
Detecting the ATS capability of the SMMU at probe time introduces a
spinlock into the ->unmap() fast path, even when ATS is not actually
in use. Furthermore, the ATC invalidation that exists is broken, as it
occurs before invalidation of the main SMMU TLB which leaves a window
where the ATC can be repopulated with stale entries.

Given that ATS is both a new feature and a specialist sport, disable it
for now whilst we fix it properly in subsequent patches. Since PRI
requires ATS, disable that too.

Cc: <stable@vger.kernel.org>
Fixes: 9ce27afc0830 ("iommu/arm-smmu-v3: Add support for PCI ATS")
Acked-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-21 17:58:12 +01:00
Will Deacon
05cbaf4ddd iommu/arm-smmu-v3: Document ordering guarantees of command insertion
It turns out that we've always relied on some subtle ordering guarantees
when inserting commands into the SMMUv3 command queue. With the recent
changes to elide locking when possible, these guarantees become more
subtle and even more important.

Add a comment documented the barrier semantics of command insertion so
that we don't have to derive the behaviour from scratch each time it
comes up on the list.

Signed-off-by: Will Deacon <will@kernel.org>
2019-08-21 15:01:53 +01:00
Christoph Hellwig
90ae409f9e dma-direct: fix zone selection after an unaddressable CMA allocation
The new dma_alloc_contiguous hides if we allocate CMA or regular
pages, and thus fails to retry a ZONE_NORMAL allocation if the CMA
allocation succeeds but isn't addressable.  That means we either fail
outright or dip into a small zone that might not succeed either.

Thanks to Hillf Danton for debugging this issue.

Fixes: b1d2dc009dec ("dma-contiguous: add dma_{alloc,free}_contiguous() helpers")
Reported-by: Tobias Klausmann <tobias.johannes.klausmann@mni.thm.de>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Tested-by: Tobias Klausmann <tobias.johannes.klausmann@mni.thm.de>
2019-08-21 07:14:10 +09:00
Robin Murphy
d720e64150 iommu/arm-smmu: Ensure 64-bit I/O accessors are available on 32-bit CPU
As part of the grand SMMU driver refactoring effort, the I/O register
accessors were moved into 'arm-smmu.h' in commit 6d7dff62afb0
("iommu/arm-smmu: Move Secure access quirk to implementation").

On 32-bit architectures (such as ARM), the 64-bit accessors are defined
in 'linux/io-64-nonatomic-hi-lo.h', so include this header to fix the
build.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-20 12:48:37 +01:00
Will Deacon
4b67f1ddcf iommu/arm-smmu: Make private implementation details static
Many of the device-specific implementation details in 'arm-smmu-impl.c'
are exposed to other compilation units. Whilst we may require this in
the future, let's make it all 'static' for now so that we can expose
things on a case-by-case basic.

Signed-off-by: Will Deacon <will@kernel.org>
2019-08-20 10:58:03 +01:00
Joerg Roedel
fe427e373d Merge branch 'for-joerg/batched-unmap' of git://git.kernel.org/pub/scm/linux/kernel/git/will/linux into core 2019-08-20 11:09:43 +02:00
Robin Murphy
ba7e4a08bb iommu/arm-smmu: Add context init implementation hook
Allocating and initialising a context for a domain is another point
where certain implementations are known to want special behaviour.
Currently the other half of the Cavium workaround comes into play here,
so let's finish the job to get the whole thing right out of the way.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:48 +01:00
Robin Murphy
62b993a36e iommu/arm-smmu: Add reset implementation hook
Reset is an activity rife with implementation-defined poking. Add a
corresponding hook, and use it to encapsulate the existing MMU-500
details.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:48 +01:00
Robin Murphy
3995e18689 iommu/arm-smmu: Add configuration implementation hook
Probing the ID registers and setting up the SMMU configuration is an
area where overrides and workarounds may well be needed. Indeed, the
Cavium workaround detection lives there at the moment, so let's break
that out.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:48 +01:00
Robin Murphy
6d7dff62af iommu/arm-smmu: Move Secure access quirk to implementation
Move detection of the Secure access quirk to its new home, trimming it
down in the process - time has proven that boolean DT flags are neither
ideal nor necessarily sufficient, so it's highly unlikely we'll ever add
more, let alone enough to justify the frankly overengineered parsing
machinery.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:48 +01:00
Robin Murphy
fc058d37b3 iommu/arm-smmu: Add implementation infrastructure
Add some nascent infrastructure for handling implementation-specific
details outside the flow of the architectural code. This will allow us
to keep mutually-incompatible vendor-specific hooks in their own files
where the respective interested parties can maintain them with minimal
chance of conflicts. As somewhat of a template, we'll start with a
general place to collect the relatively trivial existing quirks.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:47 +01:00
Robin Murphy
c5fc64881f iommu/arm-smmu: Rename arm-smmu-regs.h
We're about to start using it for more than just register definitions,
so generalise the name.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:47 +01:00
Robin Murphy
00320ce650 iommu/arm-smmu: Abstract GR0 accesses
Clean up the remaining accesses to GR0 registers, so that everything is
now neatly abstracted. This folds up the Non-Secure alias quirk as the
first step towards moving it out of the way entirely. Although GR0 does
technically contain some 64-bit registers (sGFAR and the weird SMMUv2
HYPC and MONC stuff), they're not ones we have any need to access.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:47 +01:00
Robin Murphy
19713fd40d iommu/arm-smmu: Abstract context bank accesses
Context bank accesses are fiddly enough to deserve a number of extra
helpers to keep the callsites looking sane, even though there are only
one or two of each.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:47 +01:00
Robin Murphy
aadbf2143a iommu/arm-smmu: Abstract GR1 accesses
Introduce some register access abstractions which we will later use to
encapsulate various quirks. GR1 is the easiest page to start with.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:47 +01:00
Robin Murphy
6100576284 iommu/arm-smmu: Get rid of weird "atomic" write
The smmu_write_atomic_lq oddity made some sense when the context
format was effectively tied to CONFIG_64BIT, but these days it's
simpler to just pick an explicit access size based on the format
for the one-and-a-half times we actually care.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:47 +01:00
Robin Murphy
71e8a8cdaf iommu/arm-smmu: Split arm_smmu_tlb_inv_range_nosync()
Since we now use separate iommu_gather_ops for stage 1 and stage 2
contexts, we may as well divide up the monolithic callback into its
respective stage 1 and stage 2 parts.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:47 +01:00
Robin Murphy
490325e0c1 iommu/arm-smmu: Rework cb_base handling
To keep register-access quirks manageable, we want to structure things
to avoid needing too many individual overrides. It seems fairly clean to
have a single interface which handles both global and context registers
in terms of the architectural pages, so the first preparatory step is to
rework cb_base into a page number rather than an absolute address.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:47 +01:00
Robin Murphy
620565a76b iommu/arm-smmu: Convert context bank registers to bitfields
Finish the final part of the job, once again updating some names to
match the current spec.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:47 +01:00
Robin Murphy
5114e96cb2 iommu/arm-smmu: Convert GR1 registers to bitfields
As for GR0, use the bitfield helpers to make GR1 usage a little cleaner,
and use it as an opportunity to audit and tidy the definitions. This
tweaks the handling of CBAR types to match what we did for S2CR a while
back, and fixes a couple of names which didn't quite match the latest
architecture spec (IHI0062D.c).

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:47 +01:00
Robin Murphy
0caf5f4e84 iommu/arm-smmu: Convert GR0 registers to bitfields
FIELD_PREP remains a terrible name, but the overall simplification will
make further work on this stuff that much more manageable. This also
serves as an audit of the header, wherein we can impose a consistent
grouping and ordering of the offset and field definitions

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:47 +01:00
Robin Murphy
a5b396ce50 iommu/qcom: Mask TLBI addresses correctly
As with arm-smmu from whence this code was borrowed, the IOVAs passed in
here happen to be at least page-aligned anyway, but still; oh dear.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:47 +01:00
Robin Murphy
353b325047 iommu/arm-smmu: Mask TLBI address correctly
The less said about "~12UL" the better. Oh dear.

We get away with it due to calling constraints that mean IOVAs are
implicitly at least page-aligned to begin with, but still; oh dear.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will@kernel.org>
2019-08-19 16:52:47 +01:00
Suman Anna
24ce0bab26 iommu/omap: Use the correct type for SLAB_HWCACHE_ALIGN
The macro SLAB_HWCACHE_ALIGN is of type slab_flags_t, but is currently
assigned in the OMAP IOMMU driver using a unsigned long variable. This
generates a sparse warning around the type check. Fix this by defining
the variable flags using the correct type.

Signed-off-by: Suman Anna <s-anna@ti.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-19 15:07:22 +02:00
Christoph Hellwig
df41017eaf ia64: remove support for machvecs
The only thing remaining of the machvecs is a few checks if we are
running on an SGI UV system.  Replace those with the existing
is_uv_system() check that has been rewritten to simply check the
OEM ID directly.

That leaves us with a generic kernel that is as fast as the previous
DIG/ZX1/UV kernels, but can support all hardware.  Support for UV
and the HP SBA IOMMU is now optional based on new config options.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Link: https://lkml.kernel.org/r/20190813072514.23299-27-hch@lst.de
Signed-off-by: Tony Luck <tony.luck@intel.com>
2019-08-16 14:32:26 -07:00
Linus Torvalds
e83b009c5c dma-mapping fixes for 5.3-rc
- fix the handling of the bus_dma_mask in dma_get_required_mask, which
    caused a regression in this merge window (Lucas Stach)
  - fix a regression in the handling of DMA_ATTR_NO_KERNEL_MAPPING (me)
  - fix dma_mmap_coherent to not cause page attribute mismatches on
    coherent architectures like x86 (me)
 -----BEGIN PGP SIGNATURE-----
 
 iQI/BAABCgApFiEEgdbnc3r/njty3Iq9D55TZVIEUYMFAl1UFhILHGhjaEBsc3Qu
 ZGUACgkQD55TZVIEUYOjexAAjPKLo4WGBGO1nd0btwXcI9A7jQTQlXrokmorDVzx
 5++GmTUBeEgvUJath5D3qpQTRZXo9Wb9oGMdS5U6bWJB+SbWtErM304t905TJoDM
 Cs7xcB1ZQeG/5OrQ+qGPgQCo6WO1dOl9FpaIptjNm4dn+OYhyO/YA+dgrJDwgkiA
 140RYUWa+Zhq3df4YqP4M4EnezLN1c4uE80wUxVQKDcq59sxCJek0QT0pUAMbdmQ
 /cUd2XSU113o1llmIRUh0Oj6VSEhWKHb+bdb8JfGndLzxvDcXZKl60tikWe6xpy2
 Ue0kkHRk6OPVRIxWkRjt8D+mlrCyNqN6HWx6eBmVnRKHxZ4ia2hYOFuYN9FFLLK+
 kCUlu5P/HUabBedKIxk4rbWITUqcRSviPD2WdnH2RWblvXNSDoSAufYuJ/9IGSoL
 P6a43DVKFesVF/MxeH9Ko8bnxMUO9Zn97GHcQIUplRwaqrnrCEPlvLVf/teswSQG
 C13rTnouZ0FA4z/uV96G6HfGIj87MLe/RovmLCMTeiSKrDpbcO7szP037Km73M+V
 UBmatoYCioVLxBjw3NkxCRc9UpDPdRUu31uVHrAarh4tutUASEWLrb6s9vFlGyED
 zis9IHWtIAYP3VfFtkXdZ7oDlqC/3KdEErHZuT+z4PK3Wj/QtQVfQ8SB79xFMneD
 V2E=
 =Jzmo
 -----END PGP SIGNATURE-----

Merge tag 'dma-mapping-5.3-4' of git://git.infradead.org/users/hch/dma-mapping

Pull dma-mapping fixes from Christoph Hellwig:

 - fix the handling of the bus_dma_mask in dma_get_required_mask, which
   caused a regression in this merge window (Lucas Stach)

 - fix a regression in the handling of DMA_ATTR_NO_KERNEL_MAPPING (me)

 - fix dma_mmap_coherent to not cause page attribute mismatches on
   coherent architectures like x86 (me)

* tag 'dma-mapping-5.3-4' of git://git.infradead.org/users/hch/dma-mapping:
  dma-mapping: fix page attributes for dma_mmap_*
  dma-direct: don't truncate dma_required_mask to bus addressing capabilities
  dma-direct: fix DMA_ATTR_NO_KERNEL_MAPPING
2019-08-14 10:31:11 -07:00
Marek Szyprowski
7991eb39ee iommu/exynos: Remove __init annotation from exynos_sysmmu_probe()
Exynos SYSMMU driver supports deferred probe. It happens when clocks
needed for this driver are not yet available. Typically next calls to
driver ->probe() happen before init section is free, but this is not
really guaranteed. To make if safe, remove __init annotation from
exynos_sysmmu_probe() function.

Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-14 10:50:50 +02:00
Christoph Hellwig
33dcb37cef dma-mapping: fix page attributes for dma_mmap_*
All the way back to introducing dma_common_mmap we've defaulted to mark
the pages as uncached.  But this is wrong for DMA coherent devices.
Later on DMA_ATTR_WRITE_COMBINE also got incorrect treatment as that
flag is only treated special on the alloc side for non-coherent devices.

Introduce a new dma_pgprot helper that deals with the check for coherent
devices so that only the remapping cases ever reach arch_dma_mmap_pgprot
and we thus ensure no aliasing of page attributes happens, which makes
the powerpc version of arch_dma_mmap_pgprot obsolete and simplifies the
remaining ones.

Note that this means arch_dma_mmap_pgprot is a bit misnamed now, but
we'll phase it out soon.

Fixes: 64ccc9c033c6 ("common: dma-mapping: add support for generic dma_mmap_* calls")
Reported-by: Shawn Anastasio <shawn@anastas.io>
Reported-by: Gavin Li <git@thegavinli.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Catalin Marinas <catalin.marinas@arm.com> # arm64
2019-08-10 19:52:45 +02:00
Tero Kristo
1432ebbd60 iommu/omap: remove pm_runtime_irq_safe flag for OMAP IOMMUs
This is not needed for anything, and prevents proper PM transitions for
parent devices which is bad in case of ti-sysc; this effectively kills
PM completely. Thus, remove the flag.

Signed-off-by: Tero Kristo <t-kristo@ti.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-09 17:37:10 +02:00
Tero Kristo
604629bcb5 iommu/omap: add support for late attachment of iommu devices
Current implementation of OMAP IOMMU enforces strict ordering of device
probe, initiated by iommu and followed by remoteproc later. This doesn't
work too well with the new setup done with ti-sysc changes which may
have the devices probed at pretty much any order. To overcome this limitation,
if iommu has not been probed yet when a consumer tries to attach to it,
add the device to orphan device list which will be parsed during iommu
probe to see if any orphan devices should be attached.

Signed-off-by: Tero Kristo <t-kristo@ti.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-09 17:37:10 +02:00
Suman Anna
d9c4d8a6cc iommu/omap: introduce new API for runtime suspend/resume control
This patch adds the support for the OMAP IOMMUs to be suspended
during the auto suspend/resume of the OMAP remoteproc devices. The
remote processors are auto suspended after a certain time of idle
or inactivity period. This is done by introducing two new API,
omap_iommu_domain_deactivate() and omap_iommu_domain_activate()
to allow the client users/master devices of the IOMMU devices to
deactivate & activate the IOMMU devices from their runtime
suspend/resume operations. There is no API exposed by the IOMMU
layer at present, and so these new API are added directly in the
OMAP IOMMU driver to minimize framework changes.

The API simply decrements and increments the runtime usage count
of the IOMMU devices and let the context be saved/restored using
the existing runtime pm callbacks.

Signed-off-by: Suman Anna <s-anna@ti.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-09 17:37:10 +02:00
Suman Anna
c4206c4e19 iommu/omap: Add system suspend/resume support
The MMU registers for the remote processors lose their context
in Open Switch Retention (OSWR) or device OFF modes. Hence, the
context of the IOMMU needs to be saved before it is put into any
of these lower power state (OSWR/OFF) and restored before it is
powered up to ON again. The IOMMUs need to be active as long as
the client devices that are present behind the IOMMU are active.

This patch adds the dev_pm_ops callbacks to provide the system
suspend/resume functionality through the appropriate runtime
PM callbacks. The PM runtime_resume and runtime_suspend callbacks
are already used to enable, configure and disable the IOMMUs during
the attaching and detaching of the client devices to the IOMMUs,
and the new PM callbacks reuse the same code by invoking the
pm_runtime_force_suspend() and pm_runtime_force_resume() API. The
functionality in dev_pm_ops .prepare() checks if the IOMMU device
was already runtime suspended, and skips invoking the suspend/resume
PM callbacks. The suspend/resume PM callbacks are plugged in through
the 'late' pm ops to ensure that the IOMMU devices will be suspended
only after its master devices (remoteproc devices) are suspended and
restored before them.

NOTE:
There are two other existing API, omap_iommu_save_ctx() and
omap_iommu_restore_ctx(). These are left as is to support
suspend/resume of devices on legacy OMAP3 SoC.

Signed-off-by: Suman Anna <s-anna@ti.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-09 17:37:10 +02:00
Suman Anna
c3b44a063d iommu/omap: add logic to save/restore locked TLBs
The MMUs provide a mechanism to lock TLB entries to avoid
eviction and fetching of frequently used page table entries.
These TLBs lose context when the MMUs are turned OFF. Add the
logic to save and restore these locked TLBS during suspend
and resume respectively. There are no locked TLBs during
initial power ON, and they need not be saved during final
shutdown.

Signed-off-by: Suman Anna <s-anna@ti.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-09 17:37:09 +02:00
Suman Anna
db8918f61d iommu/omap: streamline enable/disable through runtime pm callbacks
The OMAP IOMMU devices are typically present within the respective
client processor subsystem and have their own dedicated hard-reset
line. Enabling an IOMMU requires the reset line to be deasserted
and the clocks to be enabled before programming the necessary IOMMU
registers. The IOMMU disable sequence follow the reverse order of
enabling. The OMAP IOMMU driver programs the reset lines through
pdata ops to invoke the omap_device_assert/deassert_hardreset API.
The clocks are managed through the pm_runtime framework, and the
callbacks associated with the device's pm_domain, implemented in
the omap_device layer.

Streamline the enable and disable sequences in the OMAP IOMMU
driver by implementing all the above operations within the
runtime pm callbacks. All the OMAP devices have device pm_domain
callbacks plugged in the omap_device layer for automatic runtime
management of the clocks. Invoking the reset management functions
within the runtime pm callbacks in OMAP IOMMU driver therefore
requires that the default device's pm domain callbacks in the
omap_device layer be reset, as the ordering sequence for managing
the reset lines and clocks from the pm_domain callbacks don't gel
well with the implementation in the IOMMU driver callbacks. The
omap_device_enable/omap_device_idle functions are invoked through
the newly added pdata ops.

Consolidating all the device management sequences within the
runtime pm callbacks allows the driver to easily support both
system suspend/resume and runtime suspend/resume using common
code.

Signed-off-by: Suman Anna <s-anna@ti.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-09 17:37:09 +02:00
Suman Anna
3846a3b951 iommu/omap: fix boot issue on remoteprocs with AMMU/Unicache
Support has been added to the OMAP IOMMU driver to fix a boot hang
issue on OMAP remoteprocs with AMMU/Unicache, caused by an improper
AMMU/Unicache state upon initial deassertion of the processor reset.
The issue is described in detail in the next three paragraphs.

All the Cortex M3/M4 IPU processor subsystems in OMAP SoCs have a
AMMU/Unicache IP that dictates the memory attributes for addresses
seen by the processor cores. The AMMU/Unicache is configured/enabled
by the SCACHE_CONFIG.BYPASS bit - a value of 1 enables the cache and
mandates all addresses accessed by M3/M4 be defined in the AMMU. This
bit is not programmable from the host processor. The M3/M4 boot
sequence starts out with the AMMU/Unicache in disabled state, and
SYS/BIOS programs the AMMU regions and enables the Unicache during
one of its initial boot steps. This SCACHE_CONFIG.BYPASS bit is
however enabled by default whenever a RET reset is applied to the IP,
irrespective of whether it was previously enabled or not. The AMMU
registers lose their context whenever this reset is applied. The reset
is effective as long as the MMU portion of the subsystem is enabled
and clocked. This behavior is common to all the IPU and DSP subsystems
that have an AMMU/Unicache.

The IPU boot sequence involves enabling and programming the MMU, and
loading the processor and releasing the reset(s) for the processor.
The PM setup code currently sets the target state for most of the
power domains to RET. The L2 MMU can be enabled, programmed and
accessed properly just fine with the domain in hardware supervised
mode, while the power domain goes through a RET->ON->RET transition
during the programming sequence. However, the ON->RET transition
asserts a RET reset, and the SCACHE_CONFIG.BYPASS bit gets auto-set.
An AMMU fault is thrown immediately when the M3/M4 core's reset is
released since the first instruction address itself will not be
defined in any valid AMMU regions. The ON->RET transition happens
automatically on the power domain after enabling the iommu due to
the hardware supervised mode.

This patch adds and invokes the .set_pwrdm_constraint pdata ops, if
present, during the OMAP IOMMU enable and disable functions to resolve
the above boot hang issue. The ops will allow to invoke a mach-omap2
layer API pwrdm_set_next_pwrst() in a multi-arch kernel environment.
The ops also returns the current power domain state while enforcing
the constraint so that the driver can store it and use it to set back
the power domain state while releasing the constraint. The pdata ops
implementation restricts the target power domain to ON during enable,
and back to the original power domain state during disable, and thereby
eliminating the conditions for the boot issue. The implementation is
effective only when the original power domain state is either RET or
OFF, and is a no-op when it is ON or INACTIVE.

The .set_pwrdm_constraint ops need to be plugged in pdata-quirks
for the affected remote processors to be able to boot properly.

Note that the current issue is seen only on kernels with the affected
power domains programmed to enter RET. For eg., IPU1 on DRA7xx is in a
separate domain and is susceptible to this bug, while the IPU2 subsystem
is within CORE power domain, and CORE RET is not supported on this SoC.
IPUs on OMAP4 and OMAP5 are also susceptible since they are in CORE power
domain, and CORE RET is a valid power target on these SoCs.

Signed-off-by: Suman Anna <s-anna@ti.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-09 17:37:09 +02:00
Lu Baolu
3a18844dcf iommu/vt-d: Fix possible use-after-free of private domain
Multiple devices might share a private domain. One real example
is a pci bridge and all devices behind it. When remove a private
domain, make sure that it has been detached from all devices to
avoid use-after-free case.

Cc: Ashok Raj <ashok.raj@intel.com>
Cc: Jacob Pan <jacob.jun.pan@linux.intel.com>
Cc: Kevin Tian <kevin.tian@intel.com>
Cc: Alex Williamson <alex.williamson@redhat.com>
Fixes: 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")
Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
Tested-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-09 17:35:25 +02:00
Lu Baolu
ae23bfb68f iommu/vt-d: Detach domain before using a private one
When the default domain of a group doesn't work for a device,
the iommu driver will try to use a private domain. The domain
which was previously attached to the device must be detached.

Cc: Ashok Raj <ashok.raj@intel.com>
Cc: Jacob Pan <jacob.jun.pan@linux.intel.com>
Cc: Kevin Tian <kevin.tian@intel.com>
Cc: Alex Williamson <alex.williamson@redhat.com>
Fixes: 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")
Reported-by: Alex Williamson <alex.williamson@redhat.com>
Link: https://lkml.org/lkml/2019/8/2/1379
Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
Tested-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2019-08-09 17:35:25 +02:00