1616 Commits

Author SHA1 Message Date
Pranith Kumar K
9ead9a6690 statedump: Prevent duplicate statedump for master and active
Csaba found that statedump of fusemount has two instances of
each of the mem-accounting information. On debugging, I realized
that statedump is called for both ctx->master and ctx->active.
Since ctx->active is a sub-graph of ctx->master, there are duplicate
entries. Fixed this part to prevent duplication in this patch.

fixes bz#1578721
BUG: 1578721
Change-Id: I5a63b4f5933d4d720ac010c58e6dee3b27067d42
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
2018-05-16 14:57:53 +05:30
Varsha Rao
021b60858c libglusterfs/stack: Fix coverity issue
Fix ARRAY_VS_SINGLETON coverity issue.

Change-Id: I9e70be026e80011dfc8fb0f06c4563895eb6daf4
fixes: bz#1558574
Signed-off-by: Varsha Rao <varao@redhat.com>
2018-05-11 05:54:06 +00:00
Kaleb S. KEITHLEY
2b2410acb4 core: FreeBSD has pthread_set_name_np() (versus pthread_setname_np())
And has had it since at least FreeBSD 9.0

Reported-by: Roman Serbski <mefystofel@gmail.com>

Change-Id: I52cfde7f2f7a82d0e66465ac392ed7e201e1653b
updates: bz#1193929
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-05-09 08:58:34 -04:00
ShyamsundarR
38fa8b983b glusterd: Update op-version for master
Updated the op-version on master to the next release
op-version, for any future options appearing on master.

Fixes: bz#1575742
Change-Id: I861d489573360e6679c6f5ffe127835ea71f2dda
Signed-off-by: ShyamsundarR <srangana@redhat.com>
2018-05-08 03:46:14 +00:00
Kotresh HR
f8fe0771f0 posix/ctime: posix hook to set ctime xattr in relevant fops
This patch uses the ctime posix APIs to set consistent
time across replica on disk. It also stores the time
attributes in the inode context.

Credits: Rafi KC <rkavunga@redhat.com>
Updates: #208
Change-Id: I1a8d74d1e251f1d6d142f066fc99258025c0bcdd
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2018-05-06 07:09:08 +05:30
Kotresh HR
3e24848f1e posix: APIs in posix to get and set time attributes
This is part of the effort to provide consistent time
across distribute and replica set for time attributes
(ctime, atime, mtime) of the object. This patch contains
the APIs to set and get the attributes from on disk
and in inode context.

Credits: Rafi KC <rkavunga@redhat.com>
Updates: #208
Change-Id: I5d3cba53eef90ac252cb8299c0da42ebab3bde9f
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2018-05-06 01:32:30 +00:00
Csaba Henk
2ac79ed804 fuse: add support for kernel writeback cache
- Added kernel-writeback-cache command line and xlator
  option for requesting utilisation of the writeback
  cache of the kernel in FUSE_INIT (see [1]).
- Added attr-times-granularity command line and xlator
  option via which granularity of the {a,m,c}time in
  stat (attr) data that we support can be indicated to
  kernel. This is a means to avoid divergence of the
  attr times between kernel and userspace that could
  occur with writeback-cache, while still maintaining
  maximum time precision the FUSE server is capable of
  (see [2]).
- Handling FATTR_CTIME flag in FUSE_SETATTR that
  indicates presence of ctime in setattr payload.
  Currently we cannot associate arbitrary ctimes to
  files on backend, so we just touch them to update
  their ctimes to current time. Having ctimes in setattr
  payload is also a side effect of writeback cache
  (see [3] and [4]).

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4d99ff8,
     "fuse: Turn writeback cache on"
[2]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e27c9d3,
     "fuse: fuse: add time_gran to INIT_OUT"
[3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1e18bda,
     "fuse: add .write_inode"
[4]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ab9e13f,
     "fuse: allow ctime flushing to userspace"

Updates: #435
Change-Id: Id174c8e0c815c4456c35f8c53e41a6a507d91855
Signed-off-by: Csaba Henk <csaba@redhat.com>
2018-05-04 17:42:12 +00:00
Soumya Koduri
b04066721b gfapi : RECALL_LEASE implementation
Right now there are two types of upcalls
* poll method
* registering callback

But callback can be registered per fs and same callback fn shall be used
for any lease recall with object handle as argument as done for cache
invalidation.

TODO: RECALL LEASE for each glfd (for future reference)
 (may be needed fo Samba as they do not deal with
  object handles.

    In case of RECALL_LEASE, we could associate separate
     cbk function for each glfd either by
       - extending pub_glfs_lease to accept new args (recall_cbk_fn, cookie)
       - or by defining new API "glfs_register_recall_cbk_fn (glfd, recall_cbk_fn, cookie)
       . In such cases, flag it and instead of calling below upcall functions, define
        a new one to go through the glfd list and invoke each of theirs recall_cbk_fn.

Plus added following as well
* passed lease id to dict in required arguments
* added flag check in pub_glfs_open

Updates: #350
Change-Id: I07a971f0f26ec6aae0b9f9a5613504317dee153b
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
Signed-off-by: Poornima G <pgurusid@redhat.com>
Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
2018-05-04 17:12:19 +00:00
Raghavendra Bhat
b42a048d3a features/bitrot: print the path of the corrupted objects
Currently "gluster volume bitrot <volume name> scrub status"
gives the list of the corrupted objects (files as of now).
But only the gfids of those corrupted objects are seen and
one has to do getfattr, find etc operations to get the actual
path of those objects for removal etc.

This change makes an attempt to print the path of those files
as much as possible.

* Try to get the path using the on disk gfid2path xattr.

* If the above operation fails, then go for in memory path
(provided that the object has its dentry
properly created and linked in the inode table of the brick where
the corrupted object is present) So the gfid to path resolution is
a soft resolution, i.e. based on the inode and dentry cache in the
brick's memory. If the path cannot be obtained via inode table also,
then only gfid is printed.

Change-Id: Ie9a30307f43a49a2a9225821803c7d40d231de68
fixes: bz#1570962
Signed-off-by: Raghavendra Bhat <raghavendra@redhat.com>
2018-05-04 11:13:57 +00:00
Ashish Pandey
112048652a protocol/server : unwind as per op version
Change-Id: Id6717640ac14881b490e512c4682e45ffffa7f5b
fixes: bz#1570538
BUG: 1570538
Signed-off-by: Ashish Pandey <aspandey@redhat.com>
2018-05-03 07:16:21 +00:00
Kaleb S. KEITHLEY
4e7b423d3c core/various: python3 compat, prepare for python2 -> python3
see https://review.gluster.org/#/c/19788/

use print fn from __future__

Change-Id: If5075d8d9ca9641058fbc71df8a52aa35804cda4
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-05-02 11:28:46 +00:00
karthik-us
aa98153145 libglusterfs: Capture the dict response in syncop_xattrop_cbk
Problem:
Currently it is not possible to capture the xattrs values which
are set on the bricks by calling syncop_(f)xattrop, because the
response dict is not being assigned to any of the dictionaries.

Fix:
In the xattrop callback capture the response dict and send it
back to the caller if it is requested.

Change-Id: I9de9bcd97d6008091c9b060bcca3676cb9ae8ef9
fixes: bz#1572076
Signed-off-by: karthik-us <ksubrahm@redhat.com>
2018-04-27 11:57:47 +05:30
Ashish Pandey
69c35dbb4b feature/thin-arbiter: Implement thin-arbiter translator
Updates #352

Change-Id: I3d8caa6479dc8e48bec62a09b056971bb061f0cf
Signed-off-by: Ashish Pandey <aspandey@redhat.com>
2018-04-25 14:34:39 +05:30
Pranith Kumar K
93464ffd6d libglusterfs/syncop: Handle barrier_{init/destroy} in error cases
BUG: 1568521
updates: bz#1568521
Change-Id: I53e60cfcaa7f8edfa5eca47307fa99f10ee64505
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
2018-04-23 21:24:31 +05:30
Mohit Agrawal
408a6d07ab server: fix unresolved symbols by moving them to libglusterfs
Problem: glusterd2 build is failed due to undefined symbol
         (xlator_mem_cleanup , glusterfsd_ctx) in server.so

Solution: To resolve the same done below two changes
          1) Move xlator_mem_cleanup code from glusterfsd-mgmt.c
             to xlator.c to be part of libglusterfs.so
          2) replace glusterfsd_ctx to this->ctx because symbol
             glusterfsd_ctx is not part of server.so

BUG: 1544090
Change-Id: Ie5e6fba9ed458931d08eb0948d450aa962424ae5
fixes: bz#1544090
Signed-off-by: Mohit Agrawal <moagrawa@redhat.com>
2018-04-20 13:25:29 +05:30
Prashanth Pai
9e7082b756 Make glusterfsd binary print statedump & xlator dir
The glusterd2 needs following options, some of which are provided by
gluster CLI today:

--print-xlatordir
--print-statedumpdir
--print-logdir

However, the CLI package need not be present on the machine running
glusterd2. This change adds the above CLI options to glusterfsd binary
which glusterd2 depends on.

Reverts 9a1ae47c8d60836ae0628a04a153f28c1085c0e8

Related changes:
https://review.gluster.org/#/c/19882/
https://github.com/gluster/glusterd2/pull/663

Updates: bz#1193929
Change-Id: I18c123b0d3350d2bd4f2400783e3b94e402a4e29
Signed-off-by: Prashanth Pai <ppai@redhat.com>
2018-04-19 15:18:40 +05:30
Mohit Agrawal
0043c63f70 gluster: Sometimes Brick process is crashed at the time of stopping brick
Problem: Sometimes brick process is getting crashed at the time
         of stop brick while brick mux is enabled.

Solution: Brick process was getting crashed because of rpc connection
          was not cleaning properly while brick mux is enabled.In this patch
          after sending GF_EVENT_CLEANUP notification to xlator(server)
          waits for all rpc client connection destroy for specific xlator.Once rpc
          connections are destroyed in server_rpc_notify for all associated client
          for that brick then call xlator_mem_cleanup for for brick xlator as well as
          all child xlators.To avoid races at the time of cleanup introduce
          two new flags at each xlator cleanup_starting, call_cleanup.

BUG: 1544090
Signed-off-by: Mohit Agrawal <moagrawa@redhat.com>

Note: Run all test-cases in separate build (https://review.gluster.org/#/c/19700/)
      with same patch after enable brick mux forcefully, all test cases are
      passed.

Change-Id: Ic4ab9c128df282d146cf1135640281fcb31997bf
updates: bz#1544090
2018-04-19 04:31:51 +00:00
hari gowtham
be26b0da2f glusterd: volume inode/fd status broken with brick mux
Problem:
The values for inode/fd was populated from the ctx received
from the server xlator.
Without brickmux, every brick from a volume belonged to a
single brick from the volume.
So searching the server and populating it worked.

With brickmux, a number of bricks can be confined to a single
process. These bricks can be from different volumes too (if
we use the max-bricks-per-process option).
If they are from different volumes, using the server xlator
to populate causes problem.

Fix:
Use the brick to validate and populate the inode/fd status.

Signed-off-by: hari gowtham <hgowtham@redhat.com>

Change-Id: I2543fa5397ea095f8338b518460037bba3dfdbfd
fixes: bz#1566067
2018-04-19 02:54:50 +00:00
Pranith Kumar K
a7525c507e cluster/afr: Make sure latency-arg is passed to afr
xlator_notify doesn't pass the extra arguments that come in the
input function, so XLATOR_NOTIFY macro should be used instead
to pass the extra arguments to the function.

BUG: 1567881
fixes bz#1567881
Change-Id: Ic15b6c446638cbacf3149693147a754219037c47
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
2018-04-18 13:55:49 +00:00
Xavi Hernandez
52d1b36e37 libglusterfs: fix comparison of a NULL dict with a non-NULL dict
Function are_dicts_equal() had a bug when the first argument was NULL and
the second one wasn't NULL. In this case it incorrectly returned that the
dicts were different when they could be equal.

Fixes: bz#1566732
Change-Id: I0fc245c2e7d1395865a76405dbd05e5d34db3273
Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
2018-04-18 08:34:20 +00:00
Kaleb S. KEITHLEY
29024cfdd5 core/build/various: python3 compat, prepare for python2 -> python3
Note 1) we're not supposed to be using #!/usr/bin/env python, see
https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#Shebang_lines

Note 2) we're also not supposed to be using "!/usr/bin/python,
see https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build#Quick_Opt-Out

The previous patch (https://review.gluster.org/19767) tried to do too
much in one patch, so it was abandoned.

This patch does two things:
1) minor cleanup of configure(.ac) to explicitly use python2
2) change all the shebang lines to #!/usr/bin/python2 and add them
where they were missing based on warnings emitted during rpmbuild.

In a follow-up patch python2 will eventually be changed to python3.

Before that python2-isms (e.g. print, string.join(), etc.) need to be
converted to python3. Some of those can be rewritten in version agnostic
python. E.g. print statements become print() with "from __future_ import
print_function". The python 2to3 utility will be used for some of those.
Also Aravinda has given guidance in the comments to the first patch for
changes.

updates: #411
Change-Id: I471730962b2526022115a1fc33629fb078b74338
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-04-12 11:04:27 +00:00
Susant Palai
48623a33a0 experimental/cloudsync: Download xlator for archival feature
spec-files:
https://review.gluster.org/#/c/18854/

Overview:
* Cloudsync maintains three file states in it's inode-ctx i.e
  1 - LOCAL,
  2 - REMOTE,
  3 - DOWNLOADING.

* A data modifying fop is allowed only if the state is LOCAL.
  If the state is REMOTE or DOWNLOADING, client will download
  or wait for the download to finish initiated by other client.

* Multiple download and upload from different clients are synchronized
  by inodelk.

* In POSIX a state check is done (part of different commit)before
  allowing the fop to continue. If the state is remote/downloading the
  fop is unwound with EREMOTE. The client will then download the file
  and continue with the fop again.

* Basic Algo for fop (let's say write fop):
  - If LOCAL -> resume fop
  - If REMOTE ->
	- INODELK
	- STAT (this gets state and heal the state if needed)
	- DOWNLOAD
	- resume fop

Note:
* Developers will need to write plugins for download, based on the
remote store they choose. In phase-1, support will be added for
one remote store per volume. In future, more options for multiple
remote stores will be explored.

TODOs:
 - Implement stat/lookup/readdirp to return size info from xattr
 - Make plugins configurable
 - Implement unlink fop
 - Add metrics collection
 - Add sharding support

Design Contributions:
Aravinda V K <avishwan@redhat.com>
Amar Tumballi <amarts@redhat.com>
Ram Ankireddypalle <areddy@commvault.com>
Susant Palai <spalai@redhat.com>

updates: #387
Change-Id: Iddf711ee7ab4e946ae3e472ff62791a7b85e6d4b
Signed-off-by: Susant Palai <spalai@redhat.com>
2018-04-10 01:09:29 +00:00
Krutika Dhananjay
08fadcc2a7 mount/fuse: Add support for multi-threaded fuse readers
Usage: Use 'reader-thread-count=<NUM>' as command line option to
set the thread count at the time of mounting the volume.

Next task is to make these threads auto-scale based on the load,
instead of having the user remount the volume everytime to change
the thread count.

Updates #412

Change-Id: I94aa1505e5ae6a133683d473e0e4e0edd139b76b
Signed-off-by: Krutika Dhananjay <kdhananj@redhat.com>
2018-04-02 06:10:30 +00:00
Sanoj Unnikrishnan
04ede2e163 Quota: heal directory on newly added bricks when quota limit is reached
Problem: if a lookup is done on a newly added brick for a path on which limit
has been reached, the lookup fails to heal the directory tree due to quota.

Solution: Tag the lookup as an internal fop and ignore it in quota.
Since marking internal fop does not usually give enough contextual information.
Introducing new flags to pass the contextual info.

Adding dict_check_flag and dict_set_flag to aid flag operations.
A flag is a single bit in a bit array (currently limited to 256 bits).

Change-Id: Ifb6a68bcaffedd425dd0f01f7db24edd5394c095
fixes: bz#1505355
BUG: 1505355
Signed-off-by: Sanoj Unnikrishnan <sunnikri@redhat.com>
2018-03-28 04:07:12 +00:00
Pranith Kumar K
2da6650dfa storage/posix: Add active-fd-count option in gluster
Problem:
when dd happens on sharded replicate volume all the writes on shards happen
through anon-fd. When the writes don't come quick enough, old anon-fd closes
and new fd gets created to serve the new writes. open-fd-count is decremented
only after the fd is closed as part of fd_destroy(). So even when one fd is on
the way to be closed a new fd will be created and during this short period it
appears as though there are multiple fds opened on the file. AFR thinks another
application opened the same file and switches off eager-lock leading to
extra latency.

Fix:
Have a different option called active-fd whose life cycle starts at
fd_bind() and ends just before fd_destroy()

BUG: 1557932
Change-Id: I2e221f6030feeedf29fbb3bd6554673b8a5b9c94
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
2018-03-21 10:36:31 +05:30
Mohit Agrawal
cf06dd5440 glusterd: TLS verification fails while using intermediate CA
Problem: TLS verification fails while using intermediate CA
         if mgmt SSL is enabled.

Solution: There are two main issue of TLS verification failing
          1) not calling ssl_api to set cert_depth
          2) The current code does not allow to set certificate depth
             while MGMT SSL is enabled.
          After apply this patch to set certificate depth user
          need to set parameter option transport.socket.ssl-cert-depth <depth>
          in /var/lib/glusterd/secure_acccess instead to set in
          /etc/glusterfs/glusterd.vol. At the time of set secure_mgmt in ctx
          we will check the value of cert-depth and save the value of cert-depth
          in ctx.If user does not provide any value in cert-depth in that case
          it will consider default value is 1

BUG: 1555154
Change-Id: I89e9a9e1026e37efb5c20f9ec62b1989ef644f35
Signed-off-by: Mohit Agrawal <moagrawa@redhat.com>
2018-03-19 19:00:03 +00:00
Sven Fischer
de52876407 cleanup: xlator_t structure's 'client_latency' variable is not used
- Removed unused struct member and its one time usage.
  - cleaned up wrong white space

member 'client_latency' was not used otherwise since it was added by

commit 07cc8679cdf3b29680f4f105d0222da168d8bfc1
Author: Kevin Vigor <kvigor@fb.com>
Date:   Tue Mar 21 08:23:25 2017 -0700

    Halo Replication feature for AFR translator

Change-Id: Ibb0ea828d4090bbe8897f6af326b317884162a00
BUG: 1495153
Signed-off-by: Sven Fischer <sven@fischer-abc.de>
2018-03-19 03:30:31 +00:00
ShyamsundarR
ece3f0f669 protocol: Fix 4.0 client, parsing older iatt in dict
In a mixed mode cluster involving 4.0 and older 3.x bricks, if
clients are newer, then the iatt encoded in the dictionary can be
of the older iatt format, which a newer client will map incorrectly
to the newer structure.

This causes failures in FOPs that depend on this iatt for some
functionality (seen in mkdir operations failing as EIO, when DHT
hits its internal setxattr call).

The fix provided is to convert the iatt in the dict, based on which
RPC version is used to communicate with the server.

IOW, this is the reverse of change in commit "b966c7790e"

Tested using a mixed mode cluster (i.e bricks in 3.12 and 4.0 versions)
and a mixed set of clients, 3.12 and 4.0 clients.

There is no regression test provided, as this needs a mixed mode cluster
to test and validate.

Change-Id: I454e54651ca836b9f7c28f45f51d5956106aefa9
BUG: 1554053
Signed-off-by: ShyamsundarR <srangana@redhat.com>
2018-03-10 23:12:48 -05:00
ShyamsundarR
b966c7790e protocol: Added iatt conversion to older format
Added iatt conversion to an older format, when dealing with
older RPC versions. This enables iatt structure conformance
when dealing with older clients.

This helps fix rolling upgrade from 3.x versions to 4.0 version
of gluster by sending the right iatt in the dictionary when DHT
requests the same.

Change-Id: Ieaf925f81f8c7798a8fba1e90a59fa9dec82856c
BUG: 1544699
Signed-off-by: ShyamsundarR <srangana@redhat.com>
2018-03-10 18:08:53 +00:00
Amar Tumballi
940f870f47 core: provide infra to make any xlator pass-through
updates: #304

Change-Id: If6a13d2e56b195390a386d720103a882e077f66c
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-03-09 18:32:56 +00:00
Poornima G
1369f313d1 libglusterfs: Fix coverity issue FORWARD_NULL
Change-Id: I1402046edb232ca9d23346db82a0cfd041c91e70
Signed-off-by: Poornima G <pgurusid@redhat.com>
2018-03-02 19:38:45 +00:00
Kaushal M
fc35d400cb libglusterfs: Fix volume_options_t struct
The volume_options_t struct was modified and a new member was introduced
in the middle of the struct. This caused GD2 to crash when it tried to
read the volume options. The new member has been moved to the end of the
struct to correct this.

And a note has been added to notify developers on how to modify this
struct, and the xlator_api_t struct.

Updates: gluster/glusterfs#302

Change-Id: I2e9899ec10516be29c7e9d574da53be8ec17a99e
Signed-off-by: Kaushal M <kaushal@redhat.com>
2018-03-02 09:00:14 +00:00
Kaleb S. KEITHLEY
bb4343fb1a libglusterfs: move compat RPC/XDR #defines to eliminate warnings
Building with libtirpc (versus legacy glibc rpc) results in many
warnings about xdr macros that are redefined in libtirpc headers
because of the way compat.h and glusterfs.h are usually #included.

And these xdr macros in libglusterfs/src/compat.h - which were copied
from legacy glibc's rpc headers - are different than the same-name macros
in libtirpc. I haven't checked to see that any of the macros are
expanded (incorrectly) between the definition in compat.h and the
redefinition in tirpc/rpc/xdr.h; the risk seems pretty minimal. Regardless
it seems better, from a truth-and-beauty perspective to not have the
old, incorrect definitions in the first place.

Not to mention that any file that #includes compat.h and not glusterfs.h
does not need these xdr macro definitions at all. They're really only
needed when using really old glibc rpc, which would only be evident if
including glusterfs.h and/or glusterfs-fops.h. (Which by the way, nothing
currently #includes glusterfs-fops.h by itself. And maybe nothing ever
should?)

Change-Id: Ic11e4407d6ab7c498a8745a99379cbf4788a24e8
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-02-27 10:55:10 +00:00
N Balachandran
745e522c3a options: framework for options levels
Framework in order to classify options.

Updates gluster/glusterfs#302

Change-Id: I3dd6ae27bd0eb8e0065ffca75838c801e4f3ac91
Signed-off-by: N Balachandran <nbalacha@redhat.com>
2018-02-27 10:05:57 +00:00
Varsha Rao
430bff7dc3 performance/io-threads: nuke everything from a client when it disconnects
> io-threads: nuke everything from a client when it disconnects
> Commit ID: 4d8268d760
> https://review.gluster.org/#/c/18254/
> By Jeff Darcy <jdarcy@fb.com>

This patch is required to forward port io-threads namespace patch.
Updates: #401

Change-Id: I13d3a74862eea3d01e8dbc8736987c3dae6e8b2a
Signed-off-by: Varsha Rao <varao@redhat.com>
2018-02-27 03:45:30 +00:00
Milind Changire
7d641313f4 rpcsvc: scale rpcsvc_request_handler threads
Scale rpcsvc_request_handler threads to match the scaling of event
handler threads.

Please refer to https://bugzilla.redhat.com/show_bug.cgi?id=1467614#c51
for a discussion about why we need multi-threaded rpcsvc request
handlers.

Change-Id: Ib6838fb8b928e15602a3d36fd66b7ba08999430b
Signed-off-by: Milind Changire <mchangir@redhat.com>
2018-02-26 15:14:38 +05:30
Varsha Rao
d0e7177416 xlators/features/namespace: Add namespace xlator and link into brick graph
The following release-3.8-fb branch patch is upstreamed:
> features/namespace: Add namespace xlator and link into brick graph
> Commit ID: dbd30776f26e
> https://review.gluster.org/#/c/18041/
> By Michael Goulet <mgoulet@fb.com>

Changes in this patch:
Removes extra config.h and namespace.h file in namespace.c
Adds default_getspec_cbk to libglusterfs.sym
Rename dict_for_each to dict_foreach_inline
Remove fd.h header file stack.h
Add test case for truncate, open and symlink

This patch is required to forward port io-threads namespace patch.
Updates: #401

Change-Id: Ib88c95b89eecee9b8957df8a4c8712c899c761d1
Signed-off-by: Varsha Rao <varao@redhat.com>
2018-02-21 09:52:17 +00:00
Ravishankar N
6daa653569 posix/afr: handle backward compatibility for rchecksum fop
Added a volume option 'fips-mode-rchecksum' tied to op version 4.
If not set, rchecksum fop will use MD5 instead of SHA256.

updates: #230
Change-Id: Id8ea1303777e6450852c0bc25503cda341a6aec2
Signed-off-by: Ravishankar N <ravishankar@redhat.com>
2018-02-19 14:57:12 +00:00
Amar Tumballi
e80c10d5e6 metrics: set latency min value during xlator init
otherwise, the very first metrics will have all the min as 0.

also no need to print pending-fops if it is 0.

Updates #168

Change-Id: I233de6c92b1a73977bb468ba211ac6ec3c05298f
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-02-16 16:18:11 +00:00
Prashanth Pai
664b946496 Fetch backup volfile servers from glusterd2
Clients will request for a list of volfile servers from glusterd2 by
setting a (optional) flag in GETSPEC RPC call. glusterd2 will check for
the presence of this flag and accordingly return a list of glusterd2
servers in GETSPEC RPC reply. Currently, this list of servers returned
only contains servers which have bricks belonging to the volume.

See:
https://github.com/gluster/glusterd2/issues/382
https://github.com/gluster/glusterfs/issues/351

Updates #351
Change-Id: I0eee3d0bf25a87627e562380ef73063926a16b81
Signed-off-by: Prashanth Pai <ppai@redhat.com>
2018-02-16 16:16:25 +00:00
Raghavendra G
adb266baa1 libglusterfs/syncop: Add syncop_entrylk
Change-Id: Idd86b9f0fa144c2316ab6276e2def28b696ae18a
BUG: 1543279
Signed-off-by: Raghavendra G <rgowdapp@redhat.com>
2018-02-13 12:14:17 +05:30
Kinglong Mee
248152767b gfapi: return pre/post attributes from glfs_ftruncate
Updates: #389
Change-Id: I8faea0828921fb17f05f7321c3cb01747373f21e
Signed-off-by: Kinglong Mee <mijinlong@open-fs.com>
2018-02-12 21:34:46 +00:00
Kinglong Mee
09943beb49 gfapi: return pre/post attributes from glfs_fsync/fdatasync
Updates: #389
Change-Id: I4153df72d5eeecefa7579170899db4c340128bea
Signed-off-by: Kinglong Mee <mijinlong@open-fs.com>
2018-02-12 21:34:46 +00:00
Kinglong Mee
d01f7244e9 gfapi: return pre/post attributes from glfs_pread/pwrite
As nfs-ganesha, a wcc data contains pre/post attributes is return
in read/write rpc reply. nfs-ganesha get those attributes by
two getattr between the real read/write right now.

But, gluster has return pre/post attributes from glusterfsd,
those attributes are skipped in syncop/gfapi, if gfapi return them,
the upper user (nfs-ganesha) can use them directly without any
duplicate getattr.

Updates: #389
Change-Id: I7b643ae4241cfe2aeb17063de00192d81674024a
Signed-off-by: Kinglong Mee <mijinlong@open-fs.com>
2018-02-12 21:34:46 +00:00
Poornima G
bfb66cc535 io-threads: Implement put fop
Updates #353
Change-Id: I8a30b53a52618c6a6c740d2c67b19e5322ce4ddb
Signed-off-by: Poornima G <pgurusid@redhat.com>
2018-02-12 13:57:00 +05:30
Varsha Rao
aa4372bf42 performance/io-threads: expose io-thread queue depths
The following release-3.8-fb branch patch is upstreamed:

> io-stats: Expose io-thread queue depths
> Commit ID: 69509ee7d2
> https://review.gluster.org/#/c/18143/
> By Shreyas Siravara <sshreyas@fb.com>

Changes in this patch:
- Replace iot_pri_t with gf_fop_pri_t
- Replace IOT_PRI_{HI, LO, NORMAL, MAX, LEAST} with
  GF_FOP_PRI_{HI, LO, NORMAL, MAX, LEAST}
- Use dict_unref() instead of dict_destroy()

This patch is required to forward port io-threads namespace patch.
Updates: #401
Change-Id: I1b47a63185a441a30fbc423ca1015df7b36c2518
Signed-off-by: Varsha Rao <varao@redhat.com>
2018-02-08 17:01:12 +00:00
Susant Palai
545a7ce676 cluster/dht: avoid overwriting client writes during migration
For more details on this issue see
https://github.com/gluster/glusterfs/issues/308

Solution:
This is a restrictive solution where a file will not be migrated
if a client writes to it during the migration. This does not
check if the writes from the rebalance and the client actually
do overlap.

If dht_writev_cbk finds that the file is being migrated (PHASE1)
it will set an xattr on the destination file indicating the file
was updated by a non-rebalance client.
Rebalance checks if any other client has written to the dst file
and aborts the file migration if it finds the xattr.

updates gluster/glusterfs#308

Change-Id: I73aec28bc9dbb8da57c7425ec88c6b6af0fbc9dd
Signed-off-by: Susant Palai <spalai@redhat.com>
Signed-off-by: Raghavendra G <rgowdapp@redhat.com>
Signed-off-by: N Balachandran <nbalacha@redhat.com>
2018-02-02 15:24:38 +00:00
Kinglong Mee
85f1d54447 statedump: sanity check of mem_acct and rec for xlator
With memory accounting is disabled, glusterfs crash when doing statedump at,

0  0x00007fe24cff543a in gf_proc_dump_xlator_mem_info_only_in_use (xl=0x7fe23e44dc00) at statedump.c:269
1  0x00007fe24cff6310 in gf_proc_dump_oldgraph_xlator_info (top=0x7fe23e44dc00) at statedump.c:530
2  0x00007fe24cff7114 in gf_proc_dump_info (signum=10, ctx=0x7fe24ac0e000) at statedump.c:845
3  0x00007fe24d4d4bab in glusterfs_sigwaiter (arg=0x7ffc6c080750) at glusterfsd.c:2109
4  0x00007fe24bbd5dc5 in start_thread () from /lib64/libpthread.so.0
5  0x00007fe24b51a73d in clone () from /lib64/libc.so.6

(gdb) p xl->mem_acct
$1 = (struct mem_acct *) 0x0
(gdb) p xl->mem_acct->rec
$2 = 0x10

Change-Id: I10858170431311833ae01224d51c66caaad5e9a3
BUG: 1539603
Signed-off-by: Kinglong Mee <mijinlong@open-fs.com>
2018-01-31 13:05:42 +00:00
ShyamsundarR
b114303ef9 glusterd: Update op-version for master
Updated the op-version on master to the next release
op-version, for any future options appearing on master.

Change-Id: I2ef6f8874c638ade1d97477bdd8ffa1bd1a9f952
BUG: 1540338
Signed-off-by: ShyamsundarR <srangana@redhat.com>
2018-01-30 14:40:35 -05:00
Mohit Agrawal
c142d26e44 rpc: Showing some unusual timer error logs during brick stop
Solution: Update msg condition in gf_timer_call_after function
          to avoid the message

BUG: 1538427
Change-Id: I849e8e052a8259cf977fd5e7ff3aeba52f9b5f27
Signed-off-by: Mohit Agrawal <moagrawa@redhat.com>
2018-01-30 09:34:51 +00:00