2005-04-17 02:20:36 +04:00
/*
2005-11-02 06:58:39 +03:00
* Copyright ( c ) 2000 - 2005 Silicon Graphics , Inc .
* All Rights Reserved .
2005-04-17 02:20:36 +04:00
*
2005-11-02 06:58:39 +03:00
* This program is free software ; you can redistribute it and / or
* modify it under the terms of the GNU General Public License as
2005-04-17 02:20:36 +04:00
* published by the Free Software Foundation .
*
2005-11-02 06:58:39 +03:00
* This program is distributed in the hope that it would be useful ,
* but WITHOUT ANY WARRANTY ; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE . See the
* GNU General Public License for more details .
2005-04-17 02:20:36 +04:00
*
2005-11-02 06:58:39 +03:00
* You should have received a copy of the GNU General Public License
* along with this program ; if not , write the Free Software Foundation ,
* Inc . , 51 Franklin St , Fifth Floor , Boston , MA 02110 - 1301 USA
2005-04-17 02:20:36 +04:00
*/
# ifndef __XFS_BUF_H__
# define __XFS_BUF_H__
# include <linux/list.h>
# include <linux/types.h>
# include <linux/spinlock.h>
# include <linux/mm.h>
# include <linux/fs.h>
2015-09-09 00:58:40 +03:00
# include <linux/dax.h>
2005-04-17 02:20:36 +04:00
# include <linux/buffer_head.h>
# include <linux/uio.h>
2013-08-28 04:18:05 +04:00
# include <linux/list_lru.h>
2005-04-17 02:20:36 +04:00
/*
* Base types
*/
2006-01-11 07:39:08 +03:00
# define XFS_BUF_DADDR_NULL ((xfs_daddr_t) (-1LL))
typedef enum {
XBRW_READ = 1 , /* transfer into target memory */
XBRW_WRITE = 2 , /* transfer from target memory */
XBRW_ZERO = 3 , /* Zero target memory */
} xfs_buf_rw_t ;
2012-08-10 22:01:51 +04:00
# define XBF_READ (1 << 0) /* buffer intended for reading from device */
# define XBF_WRITE (1 << 1) /* buffer intended for writing to device */
# define XBF_READ_AHEAD (1 << 2) /* asynchronous read-ahead */
2016-07-20 04:13:43 +03:00
# define XBF_NO_IOACCT (1 << 3) /* bypass I/O accounting (non-LRU bufs) */
2012-08-10 22:01:51 +04:00
# define XBF_ASYNC (1 << 4) /* initiator will not wait for completion */
# define XBF_DONE (1 << 5) /* all pages in the buffer uptodate */
# define XBF_STALE (1 << 6) /* buffer has been staled, do not find it */
xfs: abort metadata writeback on permanent errors
If we are doing aysnc writeback of metadata, we can get write errors
but have nobody to report them to. At the moment, we simply attempt
to reissue the write from io completion in the hope that it's a
transient error.
When it's not a transient error, the buffer is stuck forever in
this loop, and we cannot break out of it. Eventually, unmount will
hang because the AIL cannot be emptied and everything goes downhill
from them.
To solve this problem, only retry the write IO once before aborting
it. We don't throw the buffer away because some transient errors can
last minutes (e.g. FC path failover) or even hours (thin
provisioned devices that have run out of backing space) before they
go away. Hence we really want to keep trying until we can't try any
more.
Because the buffer was not cleaned, however, it does not get removed
from the AIL and hence the next pass across the AIL will start IO on
it again. As such, we still get the "retry forever" semantics that
we currently have, but we allow other access to the buffer in the
mean time. Meanwhile the filesystem can continue to modify the
buffer and relog it, so the IO errors won't hang the log or the
filesystem.
Now when we are pushing the AIL, we can see all these "permanent IO
error" buffers and we can issue a warning about failures before we
retry the IO. We can also catch these buffers when unmounting an
issue a corruption warning, too.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Ben Myers <bpm@sgi.com>
2013-12-12 09:34:38 +04:00
# define XBF_WRITE_FAIL (1 << 24) /* async writes have failed on this buffer */
2011-07-08 16:36:32 +04:00
/* I/O hints for the BIO layer */
2012-08-10 22:01:51 +04:00
# define XBF_SYNCIO (1 << 10) /* treat this buffer as synchronous I/O */
# define XBF_FUA (1 << 11) /* force cache write through mode */
# define XBF_FLUSH (1 << 12) /* flush the disk cache before a write */
2005-04-17 02:20:36 +04:00
2010-06-24 05:49:12 +04:00
/* flags used only as arguments to access routines */
2012-08-10 22:01:51 +04:00
# define XBF_TRYLOCK (1 << 16) /* lock requested, but do not wait */
# define XBF_UNMAPPED (1 << 17) /* do not map the buffer */
2005-04-17 02:20:36 +04:00
2010-06-24 05:49:12 +04:00
/* flags used only internally */
2012-08-10 22:01:51 +04:00
# define _XBF_PAGES (1 << 20) /* backed by refcounted pages */
# define _XBF_KMEM (1 << 21) /* backed by heap memory */
# define _XBF_DELWRI_Q (1 << 22) /* buffer on a delwri queue */
# define _XBF_COMPOUND (1 << 23) /* compound buffer */
xfs: track and serialize in-flight async buffers against unmount
Newly allocated XFS metadata buffers are added to the LRU once the hold
count is released, which typically occurs after I/O completion. There is
no other mechanism at current that tracks the existence or I/O state of
a new buffer. Further, readahead I/O tends to be submitted
asynchronously by nature, which means the I/O can remain in flight and
actually complete long after the calling context is gone. This means
that file descriptors or any other holds on the filesystem can be
released, allowing the filesystem to be unmounted while I/O is still in
flight. When I/O completion occurs, core data structures may have been
freed, causing completion to run into invalid memory accesses and likely
to panic.
This problem is reproduced on XFS via directory readahead. A filesystem
is mounted, a directory is opened/closed and the filesystem immediately
unmounted. The open/close cycle triggers a directory readahead that if
delayed long enough, runs buffer I/O completion after the unmount has
completed.
To address this problem, add a mechanism to track all in-flight,
asynchronous buffers using per-cpu counters in the buftarg. The buffer
is accounted on the first I/O submission after the current reference is
acquired and unaccounted once the buffer is returned to the LRU or
freed. Update xfs_wait_buftarg() to wait on all in-flight I/O before
walking the LRU list. Once in-flight I/O has completed and the workqueue
has drained, all new buffers should have been released onto the LRU.
Signed-off-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
2016-07-20 04:15:28 +03:00
# define _XBF_IN_FLIGHT (1 << 25) /* I/O in flight, for accounting purposes */
2008-05-19 10:34:42 +04:00
2010-06-24 05:49:12 +04:00
typedef unsigned int xfs_buf_flags_t ;
2005-04-17 02:20:36 +04:00
2009-12-15 02:14:59 +03:00
# define XFS_BUF_FLAGS \
{ XBF_READ , " READ " } , \
{ XBF_WRITE , " WRITE " } , \
2011-07-08 16:36:32 +04:00
{ XBF_READ_AHEAD , " READ_AHEAD " } , \
2016-11-28 06:57:42 +03:00
{ XBF_NO_IOACCT , " NO_IOACCT " } , \
2009-12-15 02:14:59 +03:00
{ XBF_ASYNC , " ASYNC " } , \
{ XBF_DONE , " DONE " } , \
{ XBF_STALE , " STALE " } , \
xfs: abort metadata writeback on permanent errors
If we are doing aysnc writeback of metadata, we can get write errors
but have nobody to report them to. At the moment, we simply attempt
to reissue the write from io completion in the hope that it's a
transient error.
When it's not a transient error, the buffer is stuck forever in
this loop, and we cannot break out of it. Eventually, unmount will
hang because the AIL cannot be emptied and everything goes downhill
from them.
To solve this problem, only retry the write IO once before aborting
it. We don't throw the buffer away because some transient errors can
last minutes (e.g. FC path failover) or even hours (thin
provisioned devices that have run out of backing space) before they
go away. Hence we really want to keep trying until we can't try any
more.
Because the buffer was not cleaned, however, it does not get removed
from the AIL and hence the next pass across the AIL will start IO on
it again. As such, we still get the "retry forever" semantics that
we currently have, but we allow other access to the buffer in the
mean time. Meanwhile the filesystem can continue to modify the
buffer and relog it, so the IO errors won't hang the log or the
filesystem.
Now when we are pushing the AIL, we can see all these "permanent IO
error" buffers and we can issue a warning about failures before we
retry the IO. We can also catch these buffers when unmounting an
issue a corruption warning, too.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Ben Myers <bpm@sgi.com>
2013-12-12 09:34:38 +04:00
{ XBF_WRITE_FAIL , " WRITE_FAIL " } , \
2011-07-08 16:36:32 +04:00
{ XBF_SYNCIO , " SYNCIO " } , \
{ XBF_FUA , " FUA " } , \
{ XBF_FLUSH , " FLUSH " } , \
2012-08-10 22:01:51 +04:00
{ XBF_TRYLOCK , " TRYLOCK " } , /* should never be set */ \
2012-04-23 09:59:07 +04:00
{ XBF_UNMAPPED , " UNMAPPED " } , /* ditto */ \
2009-12-15 02:14:59 +03:00
{ _XBF_PAGES , " PAGES " } , \
2011-03-26 01:16:45 +03:00
{ _XBF_KMEM , " KMEM " } , \
2012-06-22 12:50:08 +04:00
{ _XBF_DELWRI_Q , " DELWRI_Q " } , \
xfs: track and serialize in-flight async buffers against unmount
Newly allocated XFS metadata buffers are added to the LRU once the hold
count is released, which typically occurs after I/O completion. There is
no other mechanism at current that tracks the existence or I/O state of
a new buffer. Further, readahead I/O tends to be submitted
asynchronously by nature, which means the I/O can remain in flight and
actually complete long after the calling context is gone. This means
that file descriptors or any other holds on the filesystem can be
released, allowing the filesystem to be unmounted while I/O is still in
flight. When I/O completion occurs, core data structures may have been
freed, causing completion to run into invalid memory accesses and likely
to panic.
This problem is reproduced on XFS via directory readahead. A filesystem
is mounted, a directory is opened/closed and the filesystem immediately
unmounted. The open/close cycle triggers a directory readahead that if
delayed long enough, runs buffer I/O completion after the unmount has
completed.
To address this problem, add a mechanism to track all in-flight,
asynchronous buffers using per-cpu counters in the buftarg. The buffer
is accounted on the first I/O submission after the current reference is
acquired and unaccounted once the buffer is returned to the LRU or
freed. Update xfs_wait_buftarg() to wait on all in-flight I/O before
walking the LRU list. Once in-flight I/O has completed and the workqueue
has drained, all new buffers should have been released onto the LRU.
Signed-off-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
2016-07-20 04:15:28 +03:00
{ _XBF_COMPOUND , " COMPOUND " } , \
{ _XBF_IN_FLIGHT , " IN_FLIGHT " }
2013-08-28 04:18:06 +04:00
xfs: abort metadata writeback on permanent errors
If we are doing aysnc writeback of metadata, we can get write errors
but have nobody to report them to. At the moment, we simply attempt
to reissue the write from io completion in the hope that it's a
transient error.
When it's not a transient error, the buffer is stuck forever in
this loop, and we cannot break out of it. Eventually, unmount will
hang because the AIL cannot be emptied and everything goes downhill
from them.
To solve this problem, only retry the write IO once before aborting
it. We don't throw the buffer away because some transient errors can
last minutes (e.g. FC path failover) or even hours (thin
provisioned devices that have run out of backing space) before they
go away. Hence we really want to keep trying until we can't try any
more.
Because the buffer was not cleaned, however, it does not get removed
from the AIL and hence the next pass across the AIL will start IO on
it again. As such, we still get the "retry forever" semantics that
we currently have, but we allow other access to the buffer in the
mean time. Meanwhile the filesystem can continue to modify the
buffer and relog it, so the IO errors won't hang the log or the
filesystem.
Now when we are pushing the AIL, we can see all these "permanent IO
error" buffers and we can issue a warning about failures before we
retry the IO. We can also catch these buffers when unmounting an
issue a corruption warning, too.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Ben Myers <bpm@sgi.com>
2013-12-12 09:34:38 +04:00
2013-08-28 04:18:06 +04:00
/*
* Internal state flags .
*/
# define XFS_BSTATE_DISPOSE (1 << 0) /* buffer being discarded */
2009-12-15 02:14:59 +03:00
xfs: allow logical-sector sized O_DIRECT
Some time ago, mkfs.xfs started picking the storage physical
sector size as the default filesystem "sector size" in order
to avoid RMW costs incurred by doing IOs at logical sector
size alignments.
However, this means that for a filesystem made with i.e.
a 4k sector size on an "advanced format" 4k/512 disk,
512-byte direct IOs are no longer allowed. This means
that XFS has essentially turned this AF drive into a hard
4K device, from the filesystem on up.
XFS's mkfs-specified "sector size" is really just controlling
the minimum size & alignment of filesystem metadata.
There is no real need to tightly couple XFS's minimal
metadata size to the minimum allowed direct IO size;
XFS can continue doing metadata in optimal sizes, but
still allow smaller DIOs for apps which issue them,
for whatever reason.
This patch adds a new field to the xfs_buftarg, so that
we now track 2 sizes:
1) The metadata sector size, which is the minimum unit and
alignment of IO which will be performed by metadata operations.
2) The device logical sector size
The first is used internally by the file system for metadata
alignment and IOs.
The second is used for the minimum allowed direct IO alignment.
This has passed xfstests on filesystems made with 4k sectors,
including when run under the patch I sent to ignore
XFS_IOC_DIOINFO, and issue 512 DIOs anyway. I also directly
tested end of block behavior on preallocated, sparse, and
existing files when we do a 512 IO into a 4k file on a
4k-sector filesystem, to be sure there were no unexpected
behaviors.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Ben Myers <bpm@sgi.com>
2014-01-22 02:46:23 +04:00
/*
* The xfs_buftarg contains 2 notions of " sector size " -
*
* 1 ) The metadata sector size , which is the minimum unit and
* alignment of IO which will be performed by metadata operations .
* 2 ) The device logical sector size
*
* The first is specified at mkfs time , and is stored on - disk in the
* superblock ' s sb_sectsize .
*
* The latter is derived from the underlying device , and controls direct IO
* alignment constraints .
*/
2005-04-17 02:20:36 +04:00
typedef struct xfs_buftarg {
2006-01-11 07:39:08 +03:00
dev_t bt_dev ;
struct block_device * bt_bdev ;
2010-09-22 04:47:20 +04:00
struct xfs_mount * bt_mount ;
2014-01-22 02:45:52 +04:00
unsigned int bt_meta_sectorsize ;
size_t bt_meta_sectormask ;
xfs: allow logical-sector sized O_DIRECT
Some time ago, mkfs.xfs started picking the storage physical
sector size as the default filesystem "sector size" in order
to avoid RMW costs incurred by doing IOs at logical sector
size alignments.
However, this means that for a filesystem made with i.e.
a 4k sector size on an "advanced format" 4k/512 disk,
512-byte direct IOs are no longer allowed. This means
that XFS has essentially turned this AF drive into a hard
4K device, from the filesystem on up.
XFS's mkfs-specified "sector size" is really just controlling
the minimum size & alignment of filesystem metadata.
There is no real need to tightly couple XFS's minimal
metadata size to the minimum allowed direct IO size;
XFS can continue doing metadata in optimal sizes, but
still allow smaller DIOs for apps which issue them,
for whatever reason.
This patch adds a new field to the xfs_buftarg, so that
we now track 2 sizes:
1) The metadata sector size, which is the minimum unit and
alignment of IO which will be performed by metadata operations.
2) The device logical sector size
The first is used internally by the file system for metadata
alignment and IOs.
The second is used for the minimum allowed direct IO alignment.
This has passed xfstests on filesystems made with 4k sectors,
including when run under the patch I sent to ignore
XFS_IOC_DIOINFO, and issue 512 DIOs anyway. I also directly
tested end of block behavior on preallocated, sparse, and
existing files when we do a 512 IO into a 4k file on a
4k-sector filesystem, to be sure there were no unexpected
behaviors.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Ben Myers <bpm@sgi.com>
2014-01-22 02:46:23 +04:00
size_t bt_logical_sectorsize ;
size_t bt_logical_sectormask ;
2006-01-11 07:39:08 +03:00
2010-11-30 09:27:57 +03:00
/* LRU control structures */
struct shrinker bt_shrinker ;
2013-08-28 04:18:05 +04:00
struct list_lru bt_lru ;
xfs: track and serialize in-flight async buffers against unmount
Newly allocated XFS metadata buffers are added to the LRU once the hold
count is released, which typically occurs after I/O completion. There is
no other mechanism at current that tracks the existence or I/O state of
a new buffer. Further, readahead I/O tends to be submitted
asynchronously by nature, which means the I/O can remain in flight and
actually complete long after the calling context is gone. This means
that file descriptors or any other holds on the filesystem can be
released, allowing the filesystem to be unmounted while I/O is still in
flight. When I/O completion occurs, core data structures may have been
freed, causing completion to run into invalid memory accesses and likely
to panic.
This problem is reproduced on XFS via directory readahead. A filesystem
is mounted, a directory is opened/closed and the filesystem immediately
unmounted. The open/close cycle triggers a directory readahead that if
delayed long enough, runs buffer I/O completion after the unmount has
completed.
To address this problem, add a mechanism to track all in-flight,
asynchronous buffers using per-cpu counters in the buftarg. The buffer
is accounted on the first I/O submission after the current reference is
acquired and unaccounted once the buffer is returned to the LRU or
freed. Update xfs_wait_buftarg() to wait on all in-flight I/O before
walking the LRU list. Once in-flight I/O has completed and the workqueue
has drained, all new buffers should have been released onto the LRU.
Signed-off-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
2016-07-20 04:15:28 +03:00
struct percpu_counter bt_io_count ;
2005-04-17 02:20:36 +04:00
} xfs_buftarg_t ;
struct xfs_buf ;
2006-01-11 07:39:08 +03:00
typedef void ( * xfs_buf_iodone_t ) ( struct xfs_buf * ) ;
2005-04-17 02:20:36 +04:00
2012-11-12 15:54:01 +04:00
2006-01-11 07:39:08 +03:00
# define XB_PAGES 2
2005-04-17 02:20:36 +04:00
2012-06-22 12:50:08 +04:00
struct xfs_buf_map {
xfs_daddr_t bm_bn ; /* block number for I/O */
int bm_len ; /* size of I/O */
} ;
2012-06-22 12:50:09 +04:00
# define DEFINE_SINGLE_BUF_MAP(map, blkno, numblk) \
struct xfs_buf_map ( map ) = { . bm_bn = ( blkno ) , . bm_len = ( numblk ) } ;
2012-11-14 10:54:40 +04:00
struct xfs_buf_ops {
2016-01-04 08:10:19 +03:00
char * name ;
2012-11-14 10:54:40 +04:00
void ( * verify_read ) ( struct xfs_buf * ) ;
void ( * verify_write ) ( struct xfs_buf * ) ;
} ;
2005-04-17 02:20:36 +04:00
typedef struct xfs_buf {
2010-09-24 13:59:15 +04:00
/*
* first cacheline holds all the fields needed for an uncontended cache
* hit to be fully processed . The semaphore straddles the cacheline
* boundary , but the counter and lock sits on the first cacheline ,
* which is the only bit that is touched if we hit the semaphore
* fast - path on locking .
*/
2016-12-07 09:36:36 +03:00
struct rhash_head b_rhash_head ; /* pag buffer hash node */
2012-06-22 12:50:08 +04:00
xfs_daddr_t b_bn ; /* block number of buffer */
2012-04-23 09:58:51 +04:00
int b_length ; /* size of buffer in BBs */
2010-09-24 13:59:15 +04:00
atomic_t b_hold ; /* reference count */
2010-12-02 08:30:55 +03:00
atomic_t b_lru_ref ; /* lru reclaim ref count */
2010-09-24 13:59:15 +04:00
xfs_buf_flags_t b_flags ; /* status flags */
2006-01-11 07:39:08 +03:00
struct semaphore b_sema ; /* semaphore for lockables */
2010-09-24 13:59:15 +04:00
2012-08-10 22:01:51 +04:00
/*
* concurrent access to b_lru and b_lru_flags are protected by
* bt_lru_lock and not by b_sema
*/
2010-12-02 08:30:55 +03:00
struct list_head b_lru ; /* lru list */
2013-08-28 04:18:06 +04:00
spinlock_t b_lock ; /* internal state lock */
unsigned int b_state ; /* internal state flags */
2014-10-02 03:04:31 +04:00
int b_io_error ; /* internal IO error state */
2006-01-11 07:39:08 +03:00
wait_queue_head_t b_waiters ; /* unpin waiters */
struct list_head b_list ;
2010-09-24 13:59:04 +04:00
struct xfs_perag * b_pag ; /* contains rbtree root */
2006-01-11 07:39:08 +03:00
xfs_buftarg_t * b_target ; /* buffer target (device) */
void * b_addr ; /* virtual address of buffer */
2014-12-04 01:43:17 +03:00
struct work_struct b_ioend_work ;
struct workqueue_struct * b_ioend_wq ; /* I/O completion wq */
2006-01-11 07:39:08 +03:00
xfs_buf_iodone_t b_iodone ; /* I/O completion function */
2008-08-13 10:36:11 +04:00
struct completion b_iowait ; /* queue for I/O waiters */
2006-01-11 07:39:08 +03:00
void * b_fspriv ;
2011-07-13 15:43:49 +04:00
struct xfs_trans * b_transp ;
2006-01-11 07:39:08 +03:00
struct page * * b_pages ; /* array of page pointers */
struct page * b_page_array [ XB_PAGES ] ; /* inline pages */
2012-06-22 12:50:09 +04:00
struct xfs_buf_map * b_maps ; /* compound buffer map */
2012-12-05 03:18:02 +04:00
struct xfs_buf_map __b_map ; /* inline compound buffer map */
2012-06-22 12:50:09 +04:00
int b_map_count ;
2012-04-23 09:58:52 +04:00
int b_io_length ; /* IO size in BBs */
2010-09-24 13:59:15 +04:00
atomic_t b_pin_count ; /* pin count */
atomic_t b_io_remaining ; /* #outstanding I/O requests */
unsigned int b_page_count ; /* size of page array */
unsigned int b_offset ; /* page offset in first page */
2014-06-25 08:58:08 +04:00
int b_error ; /* error code on I/O */
2016-05-18 04:08:15 +03:00
/*
* async write failure retry count . Initialised to zero on the first
* failure , then when it exceeds the maximum configured without a
* success the write is considered to be failed permanently and the
* iodone handler will take appropriate action .
*
* For retry timeouts , we record the jiffie of the first failure . This
* means that we can change the retry timeout for buffers already under
* I / O and thus avoid getting stuck in a retry loop with a long timeout .
*
* last_error is used to ensure that we are getting repeated errors , not
* different errors . e . g . a block device might change ENOSPC to EIO when
* a failure timeout occurs , so we want to re - initialise the error
* retry behaviour appropriately when that happens .
*/
int b_retries ;
unsigned long b_first_retry_time ; /* in jiffies */
int b_last_error ;
2012-11-14 10:54:40 +04:00
const struct xfs_buf_ops * b_ops ;
2012-11-12 15:54:19 +04:00
2006-01-11 07:39:08 +03:00
# ifdef XFS_BUF_LOCK_TRACKING
int b_last_holder ;
2005-04-17 02:20:36 +04:00
# endif
} xfs_buf_t ;
/* Finding and Reading Buffers */
2012-06-22 12:50:09 +04:00
struct xfs_buf * _xfs_buf_find ( struct xfs_buftarg * target ,
struct xfs_buf_map * map , int nmaps ,
xfs_buf_flags_t flags , struct xfs_buf * new_bp ) ;
static inline struct xfs_buf *
xfs_incore (
struct xfs_buftarg * target ,
xfs_daddr_t blkno ,
size_t numblks ,
xfs_buf_flags_t flags )
{
DEFINE_SINGLE_BUF_MAP ( map , blkno , numblks ) ;
return _xfs_buf_find ( target , & map , 1 , flags , NULL ) ;
}
struct xfs_buf * _xfs_buf_alloc ( struct xfs_buftarg * target ,
struct xfs_buf_map * map , int nmaps ,
xfs_buf_flags_t flags ) ;
static inline struct xfs_buf *
xfs_buf_alloc (
struct xfs_buftarg * target ,
xfs_daddr_t blkno ,
size_t numblks ,
xfs_buf_flags_t flags )
{
DEFINE_SINGLE_BUF_MAP ( map , blkno , numblks ) ;
return _xfs_buf_alloc ( target , & map , 1 , flags ) ;
}
2005-04-17 02:20:36 +04:00
2012-06-22 12:50:10 +04:00
struct xfs_buf * xfs_buf_get_map ( struct xfs_buftarg * target ,
struct xfs_buf_map * map , int nmaps ,
xfs_buf_flags_t flags ) ;
struct xfs_buf * xfs_buf_read_map ( struct xfs_buftarg * target ,
struct xfs_buf_map * map , int nmaps ,
2012-11-14 10:54:40 +04:00
xfs_buf_flags_t flags ,
const struct xfs_buf_ops * ops ) ;
2012-06-22 12:50:10 +04:00
void xfs_buf_readahead_map ( struct xfs_buftarg * target ,
2012-11-12 15:54:01 +04:00
struct xfs_buf_map * map , int nmaps ,
2012-11-14 10:54:40 +04:00
const struct xfs_buf_ops * ops ) ;
2012-06-22 12:50:10 +04:00
static inline struct xfs_buf *
xfs_buf_get (
struct xfs_buftarg * target ,
xfs_daddr_t blkno ,
size_t numblks ,
xfs_buf_flags_t flags )
{
DEFINE_SINGLE_BUF_MAP ( map , blkno , numblks ) ;
return xfs_buf_get_map ( target , & map , 1 , flags ) ;
}
static inline struct xfs_buf *
xfs_buf_read (
struct xfs_buftarg * target ,
xfs_daddr_t blkno ,
size_t numblks ,
2012-11-12 15:54:01 +04:00
xfs_buf_flags_t flags ,
2012-11-14 10:54:40 +04:00
const struct xfs_buf_ops * ops )
2012-06-22 12:50:10 +04:00
{
DEFINE_SINGLE_BUF_MAP ( map , blkno , numblks ) ;
2012-11-14 10:54:40 +04:00
return xfs_buf_read_map ( target , & map , 1 , flags , ops ) ;
2012-06-22 12:50:10 +04:00
}
static inline void
xfs_buf_readahead (
struct xfs_buftarg * target ,
xfs_daddr_t blkno ,
2012-11-12 15:54:01 +04:00
size_t numblks ,
2012-11-14 10:54:40 +04:00
const struct xfs_buf_ops * ops )
2012-06-22 12:50:10 +04:00
{
DEFINE_SINGLE_BUF_MAP ( map , blkno , numblks ) ;
2012-11-14 10:54:40 +04:00
return xfs_buf_readahead_map ( target , & map , 1 , ops ) ;
2012-06-22 12:50:10 +04:00
}
2012-04-23 09:58:49 +04:00
void xfs_buf_set_empty ( struct xfs_buf * bp , size_t numblks ) ;
int xfs_buf_associate_memory ( struct xfs_buf * bp , void * mem , size_t length ) ;
struct xfs_buf * xfs_buf_get_uncached ( struct xfs_buftarg * target , size_t numblks ,
int flags ) ;
2014-10-02 03:05:32 +04:00
int xfs_buf_read_uncached ( struct xfs_buftarg * target , xfs_daddr_t daddr ,
size_t numblks , int flags , struct xfs_buf * * bpp ,
const struct xfs_buf_ops * ops ) ;
2012-04-23 09:58:49 +04:00
void xfs_buf_hold ( struct xfs_buf * bp ) ;
2005-04-17 02:20:36 +04:00
/* Releasing Buffers */
2006-01-11 07:39:08 +03:00
extern void xfs_buf_free ( xfs_buf_t * ) ;
extern void xfs_buf_rele ( xfs_buf_t * ) ;
2005-04-17 02:20:36 +04:00
/* Locking and Unlocking Buffers */
2011-07-08 16:36:19 +04:00
extern int xfs_buf_trylock ( xfs_buf_t * ) ;
2006-01-11 07:39:08 +03:00
extern void xfs_buf_lock ( xfs_buf_t * ) ;
extern void xfs_buf_unlock ( xfs_buf_t * ) ;
2011-07-08 16:36:19 +04:00
# define xfs_buf_islocked(bp) \
( ( bp ) - > b_sema . count < = 0 )
2005-04-17 02:20:36 +04:00
/* Buffer Read and Write Routines */
2011-08-23 12:28:07 +04:00
extern int xfs_bwrite ( struct xfs_buf * bp ) ;
2014-10-02 03:04:22 +04:00
extern void xfs_buf_ioend ( struct xfs_buf * bp ) ;
2006-01-11 07:39:08 +03:00
extern void xfs_buf_ioerror ( xfs_buf_t * , int ) ;
2011-10-10 20:52:49 +04:00
extern void xfs_buf_ioerror_alert ( struct xfs_buf * , const char * func ) ;
2014-10-02 03:05:14 +04:00
extern void xfs_buf_submit ( struct xfs_buf * bp ) ;
extern int xfs_buf_submit_wait ( struct xfs_buf * bp ) ;
2010-01-20 02:47:39 +03:00
extern void xfs_buf_iomove ( xfs_buf_t * , size_t , size_t , void * ,
2006-01-11 07:39:08 +03:00
xfs_buf_rw_t ) ;
2010-10-06 22:41:18 +04:00
# define xfs_buf_zero(bp, off, len) \
xfs_buf_iomove ( ( bp ) , ( off ) , ( len ) , NULL , XBRW_ZERO )
2006-01-11 07:39:08 +03:00
2005-04-17 02:20:36 +04:00
/* Buffer Utility Routines */
2015-06-22 02:44:29 +03:00
extern void * xfs_buf_offset ( struct xfs_buf * , size_t ) ;
2016-02-10 07:01:11 +03:00
extern void xfs_buf_stale ( struct xfs_buf * bp ) ;
2005-04-17 02:20:36 +04:00
/* Delayed Write Buffer Routines */
2017-04-21 22:40:44 +03:00
extern void xfs_buf_delwri_cancel ( struct list_head * ) ;
xfs: on-stack delayed write buffer lists
Queue delwri buffers on a local on-stack list instead of a per-buftarg one,
and write back the buffers per-process instead of by waking up xfsbufd.
This is now easily doable given that we have very few places left that write
delwri buffers:
- log recovery:
Only done at mount time, and already forcing out the buffers
synchronously using xfs_flush_buftarg
- quotacheck:
Same story.
- dquot reclaim:
Writes out dirty dquots on the LRU under memory pressure. We might
want to look into doing more of this via xfsaild, but it's already
more optimal than the synchronous inode reclaim that writes each
buffer synchronously.
- xfsaild:
This is the main beneficiary of the change. By keeping a local list
of buffers to write we reduce latency of writing out buffers, and
more importably we can remove all the delwri list promotions which
were hitting the buffer cache hard under sustained metadata loads.
The implementation is very straight forward - xfs_buf_delwri_queue now gets
a new list_head pointer that it adds the delwri buffers to, and all callers
need to eventually submit the list using xfs_buf_delwi_submit or
xfs_buf_delwi_submit_nowait. Buffers that already are on a delwri list are
skipped in xfs_buf_delwri_queue, assuming they already are on another delwri
list. The biggest change to pass down the buffer list was done to the AIL
pushing. Now that we operate on buffers the trylock, push and pushbuf log
item methods are merged into a single push routine, which tries to lock the
item, and if possible add the buffer that needs writeback to the buffer list.
This leads to much simpler code than the previous split but requires the
individual IOP_PUSH instances to unlock and reacquire the AIL around calls
to blocking routines.
Given that xfsailds now also handle writing out buffers, the conditions for
log forcing and the sleep times needed some small changes. The most
important one is that we consider an AIL busy as long we still have buffers
to push, and the other one is that we do increment the pushed LSN for
buffers that are under flushing at this moment, but still count them towards
the stuck items for restart purposes. Without this we could hammer on stuck
items without ever forcing the log and not make progress under heavy random
delete workloads on fast flash storage devices.
[ Dave Chinner:
- rebase on previous patches.
- improved comments for XBF_DELWRI_Q handling
- fix XBF_ASYNC handling in queue submission (test 106 failure)
- rename delwri submit function buffer list parameters for clarity
- xfs_efd_item_push() should return XFS_ITEM_PINNED ]
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Mark Tinguely <tinguely@sgi.com>
Signed-off-by: Ben Myers <bpm@sgi.com>
2012-04-23 09:58:39 +04:00
extern bool xfs_buf_delwri_queue ( struct xfs_buf * , struct list_head * ) ;
extern int xfs_buf_delwri_submit ( struct list_head * ) ;
extern int xfs_buf_delwri_submit_nowait ( struct list_head * ) ;
2005-04-17 02:20:36 +04:00
/* Buffer Daemon Setup Routines */
2006-01-11 07:39:08 +03:00
extern int xfs_buf_init ( void ) ;
extern void xfs_buf_terminate ( void ) ;
2005-04-17 02:20:36 +04:00
2012-06-22 12:50:08 +04:00
/*
* These macros use the IO block map rather than b_bn . b_bn is now really
* just for the buffer cache index for cached buffers . As IO does not use b_bn
* anymore , uncached buffers do not use b_bn at all and hence must modify the IO
* map directly . Uncached buffers are not allowed to be discontiguous , so this
* is safe to do .
*
* In future , uncached buffers will pass the block number directly to the io
* request function and hence these macros will go away at that point .
*/
2012-12-05 03:18:02 +04:00
# define XFS_BUF_ADDR(bp) ((bp)->b_maps[0].bm_bn)
# define XFS_BUF_SET_ADDR(bp, bno) ((bp)->b_maps[0].bm_bn = (xfs_daddr_t)(bno))
2006-01-11 07:39:08 +03:00
2011-10-10 20:52:45 +04:00
static inline void xfs_buf_set_ref ( struct xfs_buf * bp , int lru_ref )
2010-12-02 08:31:13 +03:00
{
atomic_set ( & bp - > b_lru_ref , lru_ref ) ;
}
2006-01-11 07:39:08 +03:00
2011-07-23 03:40:27 +04:00
static inline int xfs_buf_ispinned ( struct xfs_buf * bp )
{
return atomic_read ( & bp - > b_pin_count ) ;
}
2006-01-11 07:39:08 +03:00
static inline void xfs_buf_relse ( xfs_buf_t * bp )
2005-04-17 02:20:36 +04:00
{
xfs: fix error handling for synchronous writes
If we get an IO error on a synchronous superblock write, we attach an
error release function to it so that when the last reference goes away
the release function is called and the buffer is invalidated and
unlocked. The buffer is left locked until the release function is
called so that other concurrent users of the buffer will be locked out
until the buffer error is fully processed.
Unfortunately, for the superblock buffer the filesyetm itself holds a
reference to the buffer which prevents the reference count from
dropping to zero and the release function being called. As a result,
once an IO error occurs on a sync write, the buffer will never be
unlocked and all future attempts to lock the buffer will hang.
To make matters worse, this problems is not unique to such buffers;
if there is a concurrent _xfs_buf_find() running, the lookup will grab
a reference to the buffer and then wait on the buffer lock, preventing
the reference count from ever falling to zero and hence unlocking the
buffer.
As such, the whole b_relse function implementation is broken because it
cannot rely on the buffer reference count falling to zero to unlock the
errored buffer. The synchronous write error path is the only path that
uses this callback - it is used to ensure that the synchronous waiter
gets the buffer error before the error state is cleared from the buffer
by the release function.
Given that the only sychronous buffer writes now go through xfs_bwrite
and the error path in question can only occur for a write of a dirty,
logged buffer, we can move most of the b_relse processing to happen
inline in xfs_buf_iodone_callbacks, just like a normal I/O completion.
In addition to that we make sure the error is not cleared in
xfs_buf_iodone_callbacks, so that xfs_bwrite can reliably check it.
Given that xfs_bwrite keeps the buffer locked until it has waited for
it and checked the error this allows to reliably propagate the error
to the caller, and make sure that the buffer is reliably unlocked.
Given that xfs_buf_iodone_callbacks was the only instance of the
b_relse callback we can remove it entirely.
Based on earlier patches by Dave Chinner and Ajeet Yadav.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reported-by: Ajeet Yadav <ajeet.yadav.77@gmail.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Alex Elder <aelder@sgi.com>
2011-01-07 16:02:23 +03:00
xfs_buf_unlock ( bp ) ;
2006-01-11 07:39:08 +03:00
xfs_buf_rele ( bp ) ;
2005-04-17 02:20:36 +04:00
}
2014-02-27 08:17:27 +04:00
static inline int
xfs_buf_verify_cksum ( struct xfs_buf * bp , unsigned long cksum_offset )
{
return xfs_verify_cksum ( bp - > b_addr , BBTOB ( bp - > b_length ) ,
cksum_offset ) ;
}
2014-02-27 08:18:23 +04:00
static inline void
xfs_buf_update_cksum ( struct xfs_buf * bp , unsigned long cksum_offset )
{
xfs_update_cksum ( bp - > b_addr , BBTOB ( bp - > b_length ) ,
cksum_offset ) ;
}
2005-04-17 02:20:36 +04:00
/*
* Handling of buftargs .
*/
2010-09-22 04:47:20 +04:00
extern xfs_buftarg_t * xfs_alloc_buftarg ( struct xfs_mount * ,
2014-04-14 13:01:00 +04:00
struct block_device * ) ;
2009-03-03 22:48:37 +03:00
extern void xfs_free_buftarg ( struct xfs_mount * , struct xfs_buftarg * ) ;
2005-04-17 02:20:36 +04:00
extern void xfs_wait_buftarg ( xfs_buftarg_t * ) ;
2014-04-14 13:00:29 +04:00
extern int xfs_setsize_buftarg ( xfs_buftarg_t * , unsigned int ) ;
2010-02-02 02:13:42 +03:00
2006-01-11 07:39:08 +03:00
# define xfs_getsize_buftarg(buftarg) block_size((buftarg)->bt_bdev)
# define xfs_readonly_buftarg(buftarg) bdev_read_only((buftarg)->bt_bdev)
2005-04-17 02:20:36 +04:00
# endif /* __XFS_BUF_H__ */
