360 Commits

Author SHA1 Message Date
Ashish Pandey
3bcf210a5a rpm: add thin-arbiter package
Discussion on thin arbiter volume -
https://github.com/gluster/glusterfs/issues/352#issuecomment-350981148

Main idea of having this rpm package is to deploy thin-arbiter
without glusterd and other commands on a node, and all we need
on that tie-breaker node is to run a single glusterfs command.
Also note that, no other glusterfs installation needs
thin-arbiter.so.

Make sure RPM contains sample vol file, which can work by default,
and a script to configure that volfile, along with translator image.

Change-Id: Ibace758373d8a991b6a19b2ecc60c93b2f8fc489
updates: bz#1672818
Signed-off-by: Amar Tumballi <amarts@redhat.com>
Signed-off-by: Ashish Pandey <aspandey@redhat.com>
(cherry picked from commit ca9bef7f1538beb570fcb190ff94f86f0b8ba38a)
2019-03-13 01:48:26 +00:00
Niels de Vos
fce763a6fc build: include gluster/__init__.* files for python packaging
The gluster/__init__.py file was originaly part of the glupy
installation. This file is required to have the python-gluster package
function correctly, it is expected to provide the 'gluster' namespace
for other python packages (like from the libgfapi-python project).

Because glupy does not exist anymore, this file is now added to in a new
extras/python directory.

Backport of commit b235a56394c09bb1d7308a620053dbe098d15967:
> Change-Id: I14fe959778ee3344d7d54ba342c7928a4d8080a2
> Fixes: c3fcff9ccbfcec1be242fd5cf210c9995586b078
> Fixes: 8293d21280fd6ddfc9bb54068cf87794fc6be207
> Updates: bz#1642810
> Signed-off-by: Niels de Vos <ndevos@redhat.com>

Change-Id: I04e1e0dc4022b53c02053d5452f305b78e56c389
Updates: bz#1680585
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2019-02-25 11:16:43 +00:00
Amar Tumballi
99b3ab0cf3 tests: run nfs tests only if --enable-gnfs is provided
Fixes: bz#1665358
Change-Id: Idbf88ec3ac683733b32c313377eeb72f2819bf0d
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2019-01-24 15:18:00 +00:00
Amar Tumballi
18bbb0f5b5 configure: fix the duplicate CFLAGS options
updates: bz#1193929
Change-Id: I403878719a3f81fb2ea951a951f84880fb54f3cc
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2019-01-11 07:46:19 +00:00
Amar Tumballi
36883e160e configure: add a message about asan option
This added hint helps to get ASan logs for the daemon processes, with
this one can start using asan for regression tests.

updates: bz#1633930
Change-Id: I3b39892d45d29ae514dad8ab10f65703c02003f1
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-12-29 11:12:19 +01:00
Amar Tumballi
088e2cbb5e symlink-cache: remove from the build
symlink-cache was written as an experiment to reduce the load
on 'build' systems, which keep doing symlink resolution to get
the proper header files. But since last 6+ years, there was no
way to add it to the volfile using gluster cli, and hence was
not supported anymore. As it is not maintained, and as announced
on [1], we are planning to remove it from the build system.

[1]- https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html

updates: bz#1635688
Change-Id: Iaa25069bceed04cf65f79a4b4a02c05cee848eb5
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-12-13 06:12:50 +00:00
Raghavendra Bhat
7dadea15c5 copy_file_range support in GlusterFS
* libglusterfs changes to add new fop

    * Fuse changes:
      - Changes in fuse bridge xlator to receive and send responses

    * posix changes to perform the op on the backend filesystem

    * protocol and rpc changes for sending and receiving the fop

    * gfapi changes for performing the fop

    * tools: glfs-copy-file-range tool for testing copy_file_range fop

      - Although, copy_file_range support has been added to the upstream
	    fuse kernel module, no release has been made yet of a kernel
        which contains the support. It is expected to come in the
        upcoming release of linux-4.20

        So, as of now, executing copy_file_range fop on a fused based
        filesystem results in fuse kernel module sending read on the
	    source fd and write on the destination fd.

	    Therefore a small gfapi based tool has been written to be able
        test the copy_file_range fop. This tool is similar (in functionality)
	    to the example program given in copy_file_range man page.

	    So, running regular copy_file_range on a fuse mount point and
	    running gfapi based glfs-copy-file-range tool gives some idea about
	    how fast, the copy_file_range (or reflink) can be.

	    On the local machine this was the result obtained.

	    mount -t glusterfs workstation:new /mnt/glusterfs
	    [root@workstation ~]# cd /mnt/glusterfs/
	    [root@workstation glusterfs]# ls
	    file
	    [root@workstation glusterfs]# cd
	    [root@workstation ~]# time /tmp/a.out /mnt/glusterfs/file /mnt/glusterfs/new
	    real  0m6.495s
	    user  0m0.000s
	    sys   0m1.439s
	    [root@workstation ~]# time glfs-copy-file-range $(hostname) new /tmp/glfs.log /file /rrr
	    OPEN_SRC: opening /file is success
	    OPEN_DST: opening /rrr is success
	    FSTAT_SRC: fstat on /rrr is success
	    copy_file_range successful

        real  0m0.309s
        user  0m0.039s
        sys   0m0.017s

        This tool needs following arguments
         1) hostname
         2) volume name
         3) log file path
         4) source file path (relative to the gluster volume root)
         5) destination file path (relative to the gluster volume root)

        "glfs-copy-file-range <hostname> <volume> <log file path> <source> <destination>"

      - Added a testcase as well to run glfs-copy-file-range tool

    * io-stats changes to capture the fop for profiling

    * NOTE:

      - Added conditional check to see whether the copy_file_range syscall
        is available or not. If not, then return ENOSYS.

      - Added conditional check for kernel minor version in fuse_kernel.h
        and fuse-bridge while referring to copy_file_range. And the kernel
        minor version is kept as it is. i.e. 24. Increment it in future
        when there is a kernel release which contains the support for
        copy_file_range fop in fuse kernel module.

    * The document which contains a writeup on this enhancement can be found at
      https://docs.google.com/document/d/1BSILbXr_knynNwxSyyu503JoTz5QFM_4suNIh2WwrSc/edit

Change-Id: I280069c814dd21ce6ec3be00a884fc24ab692367
updates: #536
Signed-off-by: Raghavendra Bhat <raghavendra@redhat.com>
2018-12-12 15:56:55 +00:00
Amar Tumballi
8b3a1b19fb encryption: remove crypt xlator from build
Based on the proposal to remove few features as they are not
actively maintained [1], removing crypt translator from the build.

[1] - https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html

Crypt xlator helped in on-disk / at-rest encryption of data. But
currently as there are no maintainers for this, planning to remove
it from master codebase. We are planning to host these experimental/
tech-preview xlators in another repository, so people who want to
contribute can still use the bits.

updates: bz#1635688

Change-Id: I7f2453907a595c34f635a88c49aab0845369c6e7
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-12-11 04:16:33 +00:00
Eli Schwartz
82987486a9
configure.ac: fix option to stop automake aborting on pointless error
glusterfs does not follow the GNU coding standards and therefore must
use the "foreign" strictness. Without this, autoreconf -fi would fail to
execute successfully because automake returned non-zero.

This change ensures that people using autoreconf, the GNU preferred
invocation method for the autotools build system, can successfully set
up the build.

Remove the pointless --foreign argument from the autogen.sh invocation
of automake. Not only is configure.ac the preferred way to define such
options (rather than handwritten, piecemeal invocations of every tool in
the autotools toolchain), it was never needed in the autogen.sh as that
script provides no error handling at all and always (incorrectly)
returns successfully as long as autotools itself is installed (no matter
how broken glusterfs itself is).

Change-Id: Ib0246d5368a54594f517a322465cffb9a85c1b49
fixes: bz#1656100
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2018-12-04 12:06:50 -05:00
Niels de Vos
b2a5098982 build: add option to compile with ThreadSanitizer
ThreadSanitizer is a debugging tool that can detect threads that race
for data modifications. These races can result in data corruption and
are difficult to track and fix.

Change-Id: Ibbdaf17c811e30e79cd5bdcf9cd9ff2d0cdb2abb
URL: https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
Reported-by: Yaniv Kaul <ykaul@redhat.com>
Fixes: #543
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2018-11-27 05:21:06 +00:00
Amar Tumballi
a907e468e7 bd: remove from the build
Based on the proposal to remove few features as they are not
actively maintained [1], removed BD (block device) translator
from the build.

[1] - https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html

Updates: bz#1635688
Change-Id: Ia96db406c58a7aef355dde6bc33523bb2492b1a9
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-11-08 07:36:50 +00:00
Amar Tumballi
c3fcff9ccb glupy: remove from the build
Based on the proposal to remove few features as they are not
actively maintained [1], removing 'glupy' translator from the
build.

[1] https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html

This patch aims at clearing the translator from build and tests.
A followup is needed to remove the code from repository.

Updates: bz#1642810
Change-Id: I41d0c1956330c3bbca62c540ccf9ab01bbf3a092
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-11-08 02:46:48 +00:00
Amar Tumballi
e78682bb7c xlators/experimental: remove from the build
Based on the proposal to remove few features as they are not
actively maintained [1], removing all experimental translators
from the build.

[1] https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html

Note that as followup patch, there would be a patch to remove the code.

Updates: bz#1635688
Change-Id: I8ebc256517feb37fc7580104e8bebe27d047f959
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-11-08 02:46:09 +00:00
Amar Tumballi
55a6ba56be tiering: remove the translator from build and glusterd
Based on the proposal to remove few features as they are not
actively maintained [1], removing tier translator from the
build. Also make sure there are no regression tests involving
tiering feature are present.

[1] https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html

Change-Id: I2c177f711f9b54b7b24e1a13525ff3132bd9a9c5
updates: bz#1642807
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-11-02 02:39:35 +00:00
Amar Tumballi
7331395513 stripe: remove the translator from build and glusterd
Based on the proposal to remove few features as they are not
actively maintained [1], removing stripe translator from the
build. Also make sure there are no regression tests involving
stripe translator.

[1] https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html

Note that this patch aims at removing the translator from build, and
a followup patch is needed to remove the code from repository.

Updates: bz#1364707
Change-Id: I235b305338f138e29e9f30cba65bc0dadbebbbd5
Signed-off-by: Amar Tumballi <amarts@redhat.com>
2018-10-31 02:24:49 +00:00
Krishnan Parthasarathi
62faf7d37b socket: use accept4/paccept for nonblocking socket
This reduces the no. of syscalls on Linux systems from 2, accept(2) and
fcntl(2) for setting O_NONBLOCK, to a single accept4(2).  On NetBSD, we
have paccept(2) that does the same, if we leave signal masking aside.

Added sys_accept which accepts an extra flags argument than accept(2).
This would opportunistically use accept4/paccept as available.  It would
fallback to accept(2) and fcntl(2) otherwise.

While at this, the patch sets FD_CLOEXEC flag on the accepted socket fd.

BUG: 1236272
Change-Id: I41e43fd3e36d6dabb07e578a1cea7f45b7b4e37f
fixes: bz#1236272
Signed-off-by: Krishnan Parthasarathi <kparthas@redhat.com>
2018-10-12 02:11:38 +00:00
Kaleb S. KEITHLEY
5b91555a58 python3: assume python3 unless building _packages_ on sys without py3
The jenkins release-new job runs on a CentOS 7 box, which does not
have python3. As a result it runs (autogen.sh and) configure before
producing the dist tar file, converting all the python3 shebangs to
python2 shebangs in the dist tar file.

Then when that tar file is "carried" to, e.g. Fedora koji build
system to build packages, the shebangs are incorrect, despite having
originally been correct in the git repo.

Change-Id: I5154baba3f6d29d3c4823bafc2b57abecbf90e5b
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-09-27 03:12:58 +00:00
Kaleb S. KEITHLEY
f15e948da3 build: cleanup xlator link, --no-undefined, libuuid
While attempting to build a (pre-)5.0 of glusterfs on Ubuntu
bionic and cosmic, it became apparent that there are some gremlins
hiding in the combination of the xlator export-symbols, the newish
addition of -Wl,--no-undefined, and the new switch to libuuid from
the old contrib/uuid.

Note: even though Fedora 28 (and later) and Ubuntu bionic (and
later) have the same nominal version of libtool, the Fedora version
appears to do a better job of recursing through dependencies to
determine the libraries to link with.

Examination of the build logs showed that despite appearing to work
on Fedora, not all xlators and shared libs were linked with -Wl,
--no-undefined, and -luuid. And in the case of the gnfs xlator, it
was not only not linked with -Wl,--no-undefined but alsos not linked
with -lgfxdr and -lgfrpc.

Added GF_XLATOR_LDFLAGS, similar to GF_XLATOR_DEFAULT_LDFLAGS.
GF_XLATOR_DEFAULT_LDFLAGS is for xlators that export/expose the
default or common set of symbols. GF_XLATOR_LDFLAGS is for those
remaining xlators that export/expose non-default symbols, e.g. dht
and glupy. This removes the need in the future to add things like
$(UUID_LIBS) to every xlator's Makefile.am. Just add it to
GF_XLATOR_LDFLAGS and GF_XLATOR_DEFAULT_LDFLAGS in configure.ac
and you're done.

This patch was tested on Fedora 28 (build, rpmbuild), Fedora
Rawhide/30 (rpmbuild), RHEL8 (rpmbuild), CentOS7 (rpmbuild), Fedora
koji --scratch build for f30/rawhide, and a Launchpad build for
Ubuntu cosmic/18.10.

Change-Id: Ieca104fa5c5d3c094e701c8ca4a73754dd0292b0
updates: bz#1193929
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-09-12 14:38:13 +00:00
Niels de Vos
0e399b9e46 rpm: add "--with asan" build option
RHEL-6 does not have libasan, enabling the "--with asan" option is a
no-op there.

RHEL-7 has an earlier version of libasan, and that does not have the
__asan_init symbol. Test for __asan_report_error in confiure.as instead.

Change-Id: I6322e832c5cfbd7d750f5c32c84c28771674ced6
Updates: #492
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2018-09-11 04:20:13 +00:00
Anoop C S
05e64dab56 build: Make building of glupy more obvious
Yet again glupy..!!

Core changes:
* Provide --disable-glupy(instead of --enable-glupy) since we try to
  build it by default.
* AC_SUBST macro is considered/executed even if it is inside an if
  condition. As a result GLUPY_SUBDIR was always substituted which in
  turn would make compilation enter glupy/ translator sources in absence
  of python{2/3}-devel and fails.

Miscellaneous changes:
* Remove an explicit echo for printing PYTHON version.
* Replace hard coded python version in warning message displayed in
  absence of python{2/3}-devel.
* Redirect pushd and popd output to /dev/null

Change-Id: If1ba30a12a8bda5763ef528787fccb2f7946b136
Updates: bz#1193929
Signed-off-by: Anoop C S <anoopcs@redhat.com>
2018-09-10 13:16:06 +00:00
Kaleb S. KEITHLEY
af0d5a9b53 core: python3
see https://review.gluster.org/#/c/19788/,
    https://review.gluster.org/#/c/19871/,
    https://review.gluster.org/#/c/19952/,
    https://review.gluster.org/#/c/20104/,
    https://review.gluster.org/#/c/20162/,
    https://review.gluster.org/#/c/20185/,
    https://review.gluster.org/#/c/20207/,
    https://review.gluster.org/#/c/20227/,
    https://review.gluster.org/#/c/20307/,
    https://review.gluster.org/#/c/20320/,
    https://review.gluster.org/#/c/20332/,
    https://review.gluster.org/#/c/20364/,
    https://review.gluster.org/#/c/20441/, and
    https://review.gluster.org/#/c/20484

shebangs changed from /usr/bin/python2 to /usr/bin/python3.
(Reminder, various distribution packaging guidelines require use
of explicit python version and don't allow '#!/usr/bin/env python',
regardless of how handy that idiom may be.)

glusterfs.spec(.in) package python{2,3}-gluster and python2 or
python3 dependencies as appropriate.

configure(.ac):
+ test for and use python2 or python3 as appropriate. If build
  machine has python2 and python3, use python3. Override by
  setting PYTHON=/usr/bin/python2 when running configure.
+ PYTHONDEV_CPPFLAGS from python[23]-config --includes is a
  better match to the original python sysconfig.get_python_inc().
  All those other extraneous flags breaks the build.
+ Only change the shebangs once. Changing them over and over
  again, e.g., during a `make glusterrpms` in extras/LinuxRPM
  just sends make (is it really make that's looping?) into an
  infinite loop. If you figure out why, let me know.
+ Oldest python2 is python2.6 on CentOS 6 and Debian 8 (Jessie).
  Everything else has 2.7 or 3.x
+ logic from https://review.gluster.org/c/glusterfs/+/21050, which
  needs to be removed/merged after that patch is merged.

Builds on CentOS 6, CentOS 7, Fedora 28, Fedora rawhide, and the
mysterious RHEL > 7.

Change-Id: Idae21d3b6f58b32372e1daa0d234e491e563198f
updates: #411
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-09-03 09:14:44 +00:00
Niels de Vos
44e40404ed build: add --enable-asan configure options
Introduce a `./configure --enable-asan` to build with
`-fsanitize=address -fno-omit-frame-pointer` options. This uses the
libasan.so shared library, so that needs to be available.

While running builds with the ASAN options, several linker issues
surfaced and these have been addressed with this change as well.
Building with --enable-asan has been tested on Fedora 28.

Change-Id: I428a9da70dd8f7d0056cfbe5c398619a571469b2
Updates: #492
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2018-08-30 05:41:26 +00:00
Niels de Vos
fa2f092709 build: use standard PKG_CHECK_MODULES for libxml2 availability
In case the development parts of libxml2 are not installed, it was
required to re-run ./autogen.sh to cleanup the cached values for the
check. This is not nice towards users. By using the standard
PKG_CHECK_MODULES for libxml-2.0 the results of the check are not cached
and will be probed again when running ./configure.

Change-Id: I3c4586e5555a521be5d4fb61bdb873ae0317311a
Fixes: bz#1599219
Reported-by: Sachidananda Urs <surs@redhat.com>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2018-08-16 06:55:21 +00:00
Niels de Vos
379d427960 build: remove bundled arg-standalone
libargp or argp-standalone is available on all commonly used
distributions. There is no need to bundle an unmaintained version of
argp-standalone in this repository anymore.

FreeBSD places the argp.h file in /usr/local/include when
argp-standalone is installed. This path is not added to CPPFLAGS by
default, so thats done in configure.ac as well.

Change-Id: I384a53ab0a008ec9d48fd83afeaf8fbc197e91ee
Fixes: bz#1609337
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2018-07-28 12:54:33 +00:00
Niels de Vos
7f7296d1f5 build: remove uuid from contrib/
Bundling libuuid is not needed anymore, all current distributions
provide it now.

Some OS's provide their own uuid_*() functions in libc. These may not be
fully compatible with libuuid.so found on Linux systems. In that case,
either e2fsprogs-libuuid can be installed, or support for the native
uuid_*() functions can be added to libglusterfs/src/compat-uuid.h.

Change-Id: Icfa48caea81307a3bca549364969c2038911942b
Fixes: bz#1607319
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2018-07-27 17:37:32 +00:00
Niels de Vos
35cfc01fa2 build: rename event.h to gf-event.h
Newer FreeBSD versions (noticed with 10.3-RELEASE) provide a event.h
file that on occasion gets included instead of the libglusterfs file.
When this happens, 'struct event_pool' will not be defined and building
will fail with errors like:

    autoscale-threads.c:18:55: error: incomplete definition of type 'struct event_pool'
            int                      thread_count   = pool->eventthreadcount;
                                                      ~~~~^
    autoscale-threads.c:17:16: note: forward declaration of 'struct event_pool'
            struct event_pool       *pool           = ctx->event_pool;
                   ^

This problem is caused by 'pkg-config --cflags uuid' that adds
/usr/local/include to the GF_CPPFLAGS. The use of libuuid is preferred
so that the contrib/uuid/ directory can be removed.

By renaming event.h to gf-event.h there is no conflict between the
different event.h files anymore and compiling on FreeBSD works without
issues.

Change-Id: Ie69f6b8a4f8f8e9630d39a86693eb74674f0f763
Updates: bz#1607319
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2018-07-27 17:37:32 +00:00
Xavi Hernandez
6dc5dfef81 Fix compile warnings
This patch fixes compile warnings that appear with newer compilers. The
solution applied is only to remove the warnings, but it doesn't always
solve the problem in the best way. It assumes that the problem will never
happen, as the previous code assumed.

Change-Id: I6e8470d6c2e2dbd3bd7d324b5fd2f92ffdc3d6ec
updates: bz#1193929
Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
2018-07-10 16:28:24 +02:00
Susant Palai
62990c30fc cloudsync: Adding s3 plugin for cloudsync
This is a plugin which provides an interface to retrive files from amazon-s3
which are archived in to s3.

Users need to give the above information for cloudsync to retrieve the file
from s3.

TODO:
1- A separate commit in to developer-guide will detail about the usage
of this plugin in more detail.
2- Need to create target file in aws-bucket with "gfid" names. Helps avoiding
name collisions.

Change-Id: I2e4a586f4e3f86164de9178e37673a07f317e7d9
Updates: #387
Signed-off-by: Susant Palai <spalai@redhat.com>
2018-05-30 03:26:25 +00:00
Prashanth Pai
89e1a4e572 build: Disallow unresolved symbol references
In the past, it was often[1] forgotten for xlators to be linked against
the symbols they refer to. This often caused glusterd2 to fail while
loading xlator's shared object (.so) file.

This change adds "--no-undefined" as a linker flag which causes the
linker to treat unresolved symbol references as an error and hence fail
linking.

[1]:
https://review.gluster.org/#/c/19912/
https://review.gluster.org/#/c/19664/
https://review.gluster.org/#/c/19056/
https://review.gluster.org/#/c/17659/
https://bugzilla.redhat.com/show_bug.cgi?id=1532238

Bonus:
Added cloudsync and utime xlator's generated source files to .gitignore

Updates: bz#1193929
Change-Id: I9604a4a87b7313a5fa43bda5fdb37dfa7ef8facd
Signed-off-by: Prashanth Pai <ppai@redhat.com>
2018-05-18 04:22:42 +00:00
Kotresh HR
487aaafaca utime: ctime client side xlator
The client side utime xlator does two things.

1. Update unix epoch time in frame->root->ctime
2. Update the frame->root->flags based on the fop
   which indicates time attributes that should be
   updated for the parent/entry.

Credits: Rafi KC <rkavunga@redhat.com>
Updates: #208
Change-Id: I9cad297040c70798a0a8468a080eb4aeff73138d
Signed-off-by: Kotresh HR <khiremat@redhat.com>
2018-05-06 07:09:18 +05:30
Ashish Pandey
69c35dbb4b feature/thin-arbiter: Implement thin-arbiter translator
Updates #352

Change-Id: I3d8caa6479dc8e48bec62a09b056971bb061f0cf
Signed-off-by: Ashish Pandey <aspandey@redhat.com>
2018-04-25 14:34:39 +05:30
Xie Changlong
b2f94254cd build: exclude '--with-previous-options' to prevent infinite loop
Reproducible Steps:
1. cd glusterfs/; rm -rf *; git reset --hard #clean repo
2. cd extras/LinuxRPM/; ./make_glusterrpms #it's ok here
3. ./make_glusterrpms #infinite loop
4. cd ../../; make distclean #infinite loop

Change-Id: I162953d4576cedea7c6f6c631a77163a5cca023e
updates: #439
Signed-off-by: Xie Changlong <xiechanglong@cmss.chinamobile.com>
2018-04-16 12:50:12 +00:00
Kaleb S. KEITHLEY
29024cfdd5 core/build/various: python3 compat, prepare for python2 -> python3
Note 1) we're not supposed to be using #!/usr/bin/env python, see
https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#Shebang_lines

Note 2) we're also not supposed to be using "!/usr/bin/python,
see https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build#Quick_Opt-Out

The previous patch (https://review.gluster.org/19767) tried to do too
much in one patch, so it was abandoned.

This patch does two things:
1) minor cleanup of configure(.ac) to explicitly use python2
2) change all the shebang lines to #!/usr/bin/python2 and add them
where they were missing based on warnings emitted during rpmbuild.

In a follow-up patch python2 will eventually be changed to python3.

Before that python2-isms (e.g. print, string.join(), etc.) need to be
converted to python3. Some of those can be rewritten in version agnostic
python. E.g. print statements become print() with "from __future_ import
print_function". The python 2to3 utility will be used for some of those.
Also Aravinda has given guidance in the comments to the first patch for
changes.

updates: #411
Change-Id: I471730962b2526022115a1fc33629fb078b74338
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-04-12 11:04:27 +00:00
Susant Palai
48623a33a0 experimental/cloudsync: Download xlator for archival feature
spec-files:
https://review.gluster.org/#/c/18854/

Overview:
* Cloudsync maintains three file states in it's inode-ctx i.e
  1 - LOCAL,
  2 - REMOTE,
  3 - DOWNLOADING.

* A data modifying fop is allowed only if the state is LOCAL.
  If the state is REMOTE or DOWNLOADING, client will download
  or wait for the download to finish initiated by other client.

* Multiple download and upload from different clients are synchronized
  by inodelk.

* In POSIX a state check is done (part of different commit)before
  allowing the fop to continue. If the state is remote/downloading the
  fop is unwound with EREMOTE. The client will then download the file
  and continue with the fop again.

* Basic Algo for fop (let's say write fop):
  - If LOCAL -> resume fop
  - If REMOTE ->
	- INODELK
	- STAT (this gets state and heal the state if needed)
	- DOWNLOAD
	- resume fop

Note:
* Developers will need to write plugins for download, based on the
remote store they choose. In phase-1, support will be added for
one remote store per volume. In future, more options for multiple
remote stores will be explored.

TODOs:
 - Implement stat/lookup/readdirp to return size info from xattr
 - Make plugins configurable
 - Implement unlink fop
 - Add metrics collection
 - Add sharding support

Design Contributions:
Aravinda V K <avishwan@redhat.com>
Amar Tumballi <amarts@redhat.com>
Ram Ankireddypalle <areddy@commvault.com>
Susant Palai <spalai@redhat.com>

updates: #387
Change-Id: Iddf711ee7ab4e946ae3e472ff62791a7b85e6d4b
Signed-off-by: Susant Palai <spalai@redhat.com>
2018-04-10 01:09:29 +00:00
Kaleb S. KEITHLEY
3e3ca8307e build: revert configure --without-ipv6-default behaviour
patch https://review.gluster.org/19692 breaks gluster on systems with
IPv6 enabled but don't have IPv6 reverse DNS. Also it defaulted to
enabling ipv6-default regardless of whether --with-ipv6-default or
--without-ipv6-default were specified in the options to configure.
(Also the patch was merged without review.)

Prefer libtirpc over glibc rpc.

On newer linux with tirpc and without glibc rpc use tirpc (obviously)

on less new linux with both tirpc and glibc rpc default to use tirpc,
unless --without-tirpc is specified, in which case use glibc rpc

On less new linux without tirpc fall back to glib rpc (obviously)

ipv6-default requires libtirpc. It is off by default. It must be
explicitly enabled with --with-ipv6-default. If --with-ipv6-default is
specified, but tirpc is not available, disable it and issue a warning

Change-Id: Ib96a230fafb83ec83a71948fe55af1215a7a6ffa
BUG: 1562052
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-04-02 12:50:38 +00:00
James Le Cuirot
d978ff0e3a build: Fix misleading TIRPC result in configure summary
Requesting ipv6-default even if you explicitly disable libtirpc will
then implicitly enable libtirpc because that is required. That is fine
but the configure summary should not then show TIRPC as disabled when
it is not.

The result has also been made clearer by stating that TIRPC is
"missing" when it has been tried but not found.

BUG: 1553938
Change-Id: I945bd6859aaf3defa682b0d05ee34a9827b9c45f
Signed-off-by: James Le Cuirot <chewi@gentoo.org>
2018-03-24 05:14:05 +00:00
James Le Cuirot
d121b97f9a build: Fix configure --without-ipv6-default behaviour
The current behaviour disables ipv6-default when no switch is given at
all but otherwise checks if libtirpc was requested, regardless of
whether you have given --with-ipv6-default or --without-ipv6-default.

I believe the intention was to enable when libtirpc is requested by
default but otherwise respect the switch given.

This is important because ipv6-default breaks Gluster for systems that
have IPv6 disabled.

BUG: 1553926
Change-Id: I76b91ae2699574b2e5b777453732bb5cbd79bbca
Signed-off-by: James Le Cuirot <chewi@gentoo.org>
2018-03-24 05:14:05 +00:00
Kaleb S. KEITHLEY
0ec482b5d5 build: fix typo, spelling mistake
transistional -> transitional

Change-Id: I1eb7e063288384458c305afea6d6c46a358701ff
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-03-02 19:40:37 +00:00
Varsha Rao
d0e7177416 xlators/features/namespace: Add namespace xlator and link into brick graph
The following release-3.8-fb branch patch is upstreamed:
> features/namespace: Add namespace xlator and link into brick graph
> Commit ID: dbd30776f26e
> https://review.gluster.org/#/c/18041/
> By Michael Goulet <mgoulet@fb.com>

Changes in this patch:
Removes extra config.h and namespace.h file in namespace.c
Adds default_getspec_cbk to libglusterfs.sym
Rename dict_for_each to dict_foreach_inline
Remove fd.h header file stack.h
Add test case for truncate, open and symlink

This patch is required to forward port io-threads namespace patch.
Updates: #401

Change-Id: Ib88c95b89eecee9b8957df8a4c8712c899c761d1
Signed-off-by: Varsha Rao <varao@redhat.com>
2018-02-21 09:52:17 +00:00
Niels de Vos
8d9961e637 build: add --without-server option
With Gluster 4.0 we will not provide the server components for EL6 and
older. At one point Gluster 4.x will get GlusterD2, which requires
Golang tools in the distribution. EL6 does not contain these at the
moment.

With this change, it is possible to `./configure --without-server` which
prevents building glusterd and the xlators for the bricks. Building RPMs
can pass `--without server` and the glusterfs-server sub-package will
not be created.

Change-Id: I97f5ccf9f2c76e60d9af83915fc59fae57ad6d25
BUG: 1074947
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2018-02-19 17:38:55 +00:00
Anoop C S
ec2c95f310 build: Remove unwanted autoconf messages for libtirpc
AC_MSG_RESULT is normally used when it is preceded by AC_MSG_CHECKING
during explicit checking of certain requirements. With PKG_CHECK_MODULES
being used for checking the presence of libtirpc itself generates the
following message:

checking for TIRPC... yes

Change-Id: I3f088a45ef2ced6f6fd9e1524e758c812deecb8f
BUG: 1536186
Signed-off-by: Anoop C S <anoopcs@redhat.com>
2018-02-04 15:53:01 +00:00
Jan Engelhardt
c3647b747a build: restore ability to control verbosity settings
`make V=1` is broken — no commands are printed whatsoever. At the
same time, `make V=0` *also* is broken in that no summary lines
("CC  foo.o") are printed, either. Kill the annoying hardcoded
--quiet in configure.ac, since it seems to override everything that
automake offers.

Change-Id: I377c0e0469619a33586afb4a93dde6d241e7bc21
Fixes: #381
BUG: 1539023
Original-author: Jan Engelhardt <jengelh@inai.de>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
[ndevos: silence rpc/xdr/gen rpgcen]
2018-01-26 12:58:03 +00:00
Kaleb S. KEITHLEY
0c57232ae0 build: glibc has removed rpc headers and rpcgen in Fedora28, use libtirpc
Other Linux distributions are doing the same; some others have already
done so.

Switch to libtirpc(-devel) and unbundled rpcgen packages. For now
rpcgen is still provided by the glibc-rpcgen RPM, but rpcsvc-proto's
rpcgen subpackage is available now but will not be used until
glibc-rpcgen is retired. (note, rpcsvc-proto's rpcgen is just named
rpcgen-...rpm. I.e. not rpcsvc-proto-rpcgen.) Right now either one
will satisfy the BuildRequires: rpcgen.

Also, when a .spec file has
  BuildRequires: foo-devel
it is not necessary to also have:
  BuildRequires: foo
or even:
  BuildRequires: foo foo-devel

The foo-devel package has a dependency on foo, which will install foo
automatically. It's usually also not necessary to have a corresponding
  Requires: foo
as the rpmbuild process will also automatically determine the
install-time dependencies.

And other minor glusterfs.spec.in cleanup of ipv6-default, including
sorting the argument definitions because the comment says "keep them
sorted" (Though nobody appears to have ever sorted them in the first
place.)

Change-Id: I86f847dfda0fef83e22c6e8b761342d652a2d9ba
BUG: 1536186
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
2018-01-24 10:22:14 +00:00
Sakshi Bansal
78868033bc dentry fop serializer: added new server side xlator for dentry fop serialization
Problems addressed by this xlator :

[1]. To prevent race between parallel mkdir,mkdir and lookup etc.

Fops like mkdir/create, lookup, rename, unlink, link that happen on a
particular dentry must be serialized to ensure atomicity.

Another possible case can be a fresh lookup to find existance of a path
whose gfid is not set yet. Further, storage/posix employs a ctime based
heuristic 'is_fresh_file' (interval time is less than 1 second of current
time) to check fresh-ness of file. With serialization of these two fops
(lookup & mkdir), we eliminate the race altogether.

[2]. Staleness of dentries

This causes exponential increase in traversal time for any inode in the
subtree of the directory pointed by stale dentry.

Cause :  Stale dentry is created because of following two operations:

      a. dentry creation due to inode_link, done during operations like
         lookup, mkdir, create, mknod, symlink, create and
      b. dentry unlinking due to various operations like rmdir, rename,
         unlink.

       The reason is __inode_link uses __is_dentry_cyclic, which explores
       all possible path to avoid cyclic link formation during inode
       linkage. __is_dentry_cyclic explores stale-dentry(ies) and its
       all ancestors which is increases traversing time exponentially.

Implementation : To acheive this all fops on dentry must take entry locks
before they proceed, once they have acquired locks, they perform the fop
and then release the lock.

Some documentation from email conversation:
[1] http://www.gluster.org/pipermail/gluster-devel/2015-December/047314.html

[2] http://www.gluster.org/pipermail/gluster-devel/2015-August/046428.html

With this patch, the feature is optional, enable it by running:

 `gluster volume set $volname features.sdfs enable`

Also the feature is tested for a month without issues in the
experiemental branch for all the regression.

Change-Id: I6e80ba3cabfa6facd5dda63bd482b9bf18b6b79b
Fixes: #397
BUG: 1304962
Signed-off-by: Sakshi Bansal <sabansal@redhat.com>
Signed-off-by: Amar Tumballi <amarts@redhat.com>
Signed-off-by: Sunny Kumar <sunkumar@redhat.com>
2018-01-24 05:39:44 +00:00
James Le Cuirot
2155f9926a build: Use pkg-config to get libtirpc flags
This benefits cross-compiling and other exotic setups.

Change-Id: Id9f168728d96264ccab23d2e618fa2b4003455a3
Signed-off-by: James Le Cuirot <chewi@gentoo.org>
2017-11-29 00:44:11 +00:00
James Le Cuirot
373cdd6e76 build: Allow libtirpc to be explicitly requested
Some distributions like Gentoo no longer include the RPC stuff in
their glibc packages.

Change-Id: Ic47065e9c2f5a0ccd860df9d7185eff59990ff10
Signed-off-by: James Le Cuirot <chewi@gentoo.org>
2017-11-22 07:13:12 +00:00
Aravinda VK
705ec05504 geo-rep: Refactoring Config and Arguments parsing
- Fixed Python pep8 issues
- Removed dead code
- Rewritten configuration management
- Rewritten Arguments/subcommands handling
- Added Args upgrade to accommodate all these changes without changing
  glusterd code
- use of md5 removed, which was used to hash the brick path for workdir

Both Master and Slave nodes will have subdir for session in the
format "<mastervol>_<primary_slave_host>_<slavevol>

  $GLUSTER_LOGDIR/geo-replication/<mastervol>_<primary_slave_host>_<slavevol>
  $GLUSTER_LOGDIR/geo-replication-slaves/<mastervol>_<primary_slave_host>_<slavevol>

Log file paths renamed since session info is available with directory
name itself.

  $LOG_DIR_MASTER/
      - gsyncd.log - Gsyncd, Worker monitor logs
      - mnt-<brick-path>.log - Aux mount logs, mounted by each worker
      - changes-<brick-path>.log - Changelog related logs(One per brick)

  $LOG_DIR_SLAVE/
      - gsyncd.log - Slave Gsyncd logs
      - mnt-<master-node>-<master-brick-path>.log - Aux mount logs,
        mounted for each connection from master-node:master-brick
      - mnt-mbr-<master-node>-<master-brick-path>.log - Same as above,
        but mountbroker setup

Fixes: #73
Change-Id: I2ec2a21e4e2a92fd92899d026e8543725276f021
Signed-off-by: Aravinda VK <avishwan@redhat.com>
2017-11-15 05:20:08 +00:00
Xavier Hernandez
3f8d118e48 libglusterfs/atomic: Improved atomic support
This patch solves a detection problem in configure.ac that prevented
that compilation detects builtin __atomic or __sync functions.

It also adds more atomic types and support for other atomic functions.

An special case has been added to support 64-bit atomics on 32-bit
systems. The solution is to fallback to the mutex solution only for
64-bit atomics, but smaller atomic types will still take advantage
of builtins if available.

Change-Id: I6b9afc7cd6e66b28a33278715583552872278801
BUG: 1510397
Signed-off-by: Xavier Hernandez <jahernan@redhat.com>
2017-11-14 05:22:00 +00:00
Kevin Vigor
1260ee53b1 gluster: IPv6 single stack support
Summary:
- This diff changes all locations in the code to prefer inet6 family
  instead of inet.  This will allow change GlusterFS to operate
  via IPv6 instead of IPv4 for all internal operations while still
  being able to serve (FUSE or NFS) clients via IPv4.
- The changes apply to NFS as well.
- This diff ports D1892990, D1897341 & D1896522 to the 3.8 branch.

Test Plan: Prove tests!

Reviewers: dph, rwareing

Signed-off-by: Shreyas Siravara <sshreyas@fb.com>

Change-Id: I34fdaaeb33c194782255625e00616faf75d60c33
BUG: 1406898
Reviewed-on-3.8-fb: http://review.gluster.org/16059
Reviewed-by: Shreyas Siravara <sshreyas@fb.com>
Tested-by: Shreyas Siravara <sshreyas@fb.com>
2017-10-24 09:28:08 +00:00
Niels de Vos
a413c6353d build: make it possible to build cleanly 2x in a row
'make clean' does not cleanup everything, and some of the files get
cleaned too eagerly. Several files are being packaged in a 'make dist'
tarball, that get rebuild each time anyway.

Specifically, this change prevents
 - libglusterfs/src/generator.pyc from laying around
 - keeping rpc/xdr/gen/*.x symlinks
 - modifying tests/basic/{fuse,gfapi}/Makefile each run
 - including tests/env.rc and events/src/eventtypes.py in the tarball

Change-Id: I774dd1abf3a9d3b6a89b938cf6ee7d7792c59a82
BUG: 1501317
Reported-by: Patrick Matthäi <pmatthaei@debian.org>
Signed-off-by: Niels de Vos <ndevos@redhat.com>
2017-10-13 16:28:55 +00:00