42 Commits

Author SHA1 Message Date
Milind Changire
d95535bae2 tools/glusterfind: unquote DELETE path before further processing
Problem:
DELETE path is quoted before it reaches glusterfind. This wasn't handled
in the glusterfind code leading to double quoting of path separator
'%2F' to '%252F' i.e. the '%' character in '%2F' itself was quoted to
'%25'

Solution:
unquote the the deleted path before further processing

Change-Id: I2dfbbd7792dc0f9da5c8e02093b0f1c031ff344a
BUG: 1465024
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: https://review.gluster.org/17629
Smoke: Gluster Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2017-06-30 07:40:51 +00:00
Milind Changire
3011af6773 tools/glusterfind: initialize variable 'end'
Problem:
syntax error due to uninitialized variable 'end'

Solution:
set end = -1

Change-Id: Ia8b500e9dd5304d0832e5f594abca0b5b2ab357e
BUG: 1462241
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: https://review.gluster.org/17557
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2017-06-19 10:10:32 +00:00
Milind Changire
dbdf56ae3f tools/glusterfind: add --end-time option
Add optional --end-time argument to be used with --since-time
when using the "query" command.

"start" and "end" times are passed in the command-line to
changelog.py only if --full has not been specified on command-line.
-1 is passed to changelog.py as end time if user has not supplied
--end-time on command-line.

brickfind.py:
Remove unused "start" command-line argument.

Also:
Minor indentation changes to keep flake8-2.7 happy.

Change-Id: I063ef5459916f711503881ade5c4fc32374edad0
BUG: 1453151
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: https://review.gluster.org/17439
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2017-06-12 05:14:08 +00:00
Milind Changire
64588d4f6d tools/glusterfind: add --field-separator option
Problem:
Default field separator is a space character.
This gets in the way if the file name itself has embedded spaces.

Solution:
Add --field-separator option to "pre" and "query" commands.
The field separator string will be used to separate strings in the
output lines that get written to the output file.

eg.
old output:
NEW file1.txt
RENAME file2 Copy.txt file3.txt

with --field-separator as "==="
new output:
NEW===file1.txt
RENAME===file2 Copy.txt===file3.txt

Change-Id: I71e878fed58ba1113d97044ac9f6404ee66227c7
BUG: 1453151
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: https://review.gluster.org/17481
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2017-06-12 04:48:55 +00:00
Kaleb S. KEITHLEY
fdabb97d57 build/packaging: Debian and Ubuntu don't have /usr/libexec
GLUSTERFS_LIBEXECDIR is effectively hard-coded to /usr/libexec/glusterfs
in configure(.ac)

Debian-based distributions don't have a /usr/libexec/ directory

This issues is partially mitigated by the use of $libexecdir in
some of the Makefile.am files, but even so the incorrectly defined
GLUSTERFS_LIBEXECDIR results in various things such as gsyncd,
glusterfind, eventsd, etc., trying to invoke other scripts and
programs from a location that doesn't exist.

And once we correctly define GLUSTERFS_LIBEXECDIR, then we might as
well use it appropriatedly.

Change-Id: If5219cadc51ae316f7ba2e2831d739235c77902d
BUG: 1430841
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Reviewed-on: https://review.gluster.org/16880
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Joe Julian <me@joejulian.name>
Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
2017-03-13 10:18:07 -04:00
Milind Changire
355c195503 tools/glusterfind: delete temporary folder
Problem:
Run specific temporary folder created under
/usr/var/lib/misc/glusterfsd/glusterfind/<session>/<volume>/
remains undeleted.

Solution:
Delete the temporary folder from all nodes.

Change-Id: I0edaf868aebb01b15c489434bbb26fe853351384
BUG: 1413526
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: https://review.gluster.org/16416
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2017-01-30 00:27:25 -05:00
Milind Changire
2b4b928ed3 tools/glusterfind: avoid deleting keys directory
Problem:
gluster volume delete mistakenly deletes the .keys directory under
/var/lib/glusterd/glusterfind.

Solution:
Check for ".keys" directory and avoid deleting it.

Change-Id: Ia595c8bf3f423c1ad5d6faa183a29598c07a11f9
BUG: 1402369
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/16052
Reviewed-by: Aravinda VK <avishwan@redhat.com>
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
2016-12-07 22:05:48 -08:00
Milind Changire
592fd52d58 tools/glusterfind: xml parsing fix for tiered volumes
gluster volume info <vol> --xml for non-tiered volumes have
'bricks/brick' elements under the 'volInfo/volumes/volume' element.
However, tiered volumes have a 'bricks/hotBricks/brick' and
'bricks/coldBricks/brick' elements under the 'volInfo/volumes/volume'
element.

Fix main.py::get_nodes()

BUG: 1389481
Change-Id: I2f4465bfa8a55e7fa87917d3ec3e69b05d5241b9
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/15746
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2016-11-17 03:30:54 -08:00
Milind Changire
feea851fad tools/glusterfind: kill remote processes and separate run-time directories
Problem #1:
Hitting CTRL+C leaves stale processes on remote nodes if glusterfind pre
has been initiated.

Solution #1:
Adding "-t -t" to ssh command-line forces pseudo-terminal to be assigned
to remote process. When local process receives Keyboard Interrupt,
SIGHUP is immediately conveyed to the remote terminal causing remote
changelog.py process to terminate immediately.

Problem #2:
Concurrent glusterfind pre runs are not possible on the same glusterfind
session in case of a runaway process.

Solution #2:
glusterfind pre runs now add random directory name to the working
directory to store and manage temporary database and changelog
processing.
If KeyboardInterrupt is received, the function call
run_cmd_nodes("cleanup", args, tmpfilename=gtmpfilename)
cleans up the remote run specific directory.

Patch:
7571380 cli/xml: Fix wrong XML format in volume get command
broke "gluster volume get <vol> changelog.rollover-time --xml"
Now fixed function utils.py::get_changelog_rollover_time()

Fixed spurious trailing space getting written if second path is empty in
main.py::write_output()
Fixed repetitive changelog processing in changelog.py::get_changes()

Change-Id: Ia8d96e2cd47bf2a64416bece312e67631a1dbf29
BUG: 1382236
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/15609
Smoke: Gluster Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2016-10-25 05:00:54 -07:00
Aravinda VK
db4f4f1500 packaging: Remove ".py" extension from symlink target
S57glusterfind-delete-post.py* is packaged in
$LIBEXEC/glusterfs/glusterfind directory, symlink created to
/var/lib/glusterd/hooks directory

ln -s $(libexecdir)/glusterfs/glusterfind/S57glusterfind-delete-post.py \
$(DESTDIR)$(GLUSTERD_WORKDIR)/hooks/1/delete/post/S57glusterfind-delete-post.py

".py" should be removed from symlink target so that build system will
not try to compile the symlink files and generate .pyc/.pyo files.

BUG: 1356868
Change-Id: I9155378aa72eb8559ba8af76d91a4985f69f38f9
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/14928
Smoke: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
2016-07-15 06:41:16 -07:00
Milind Changire
16a3f0d020 tools/glusterfind: add --full option to query command
The --full option will use brickfind.py to list all files in the volume.
The output file will contain url-encoded file names prefixed with the
tag string to indicate that all files should be considered as new.
The default tag string for the --full option is "NEW". This can be
changed with the --tag-for-full-find command-line option.

Change-Id: Ic85ba5db062e19df13ae9dc2de8a08eacb5c9792
BUG: 1286279
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/12779
Smoke: Gluster Build System <jenkins@build.gluster.com>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2016-03-30 21:51:32 -07:00
Aravinda VK
48a0a38fad tools/glusterfind: Handling Unicode file names
Unicode filenames handled cleanly with this patch. Changelog
files and output files are opened with utf-8 encoding using codecs.open.

urllib.quote_plus and unquote_plus will not handle Unicode so, encode
Unicode to 8-bit string version before calling unquote. urllib.quote_plus
requires 8-bit string itself so do not decode to Unicode if we need to use
quote_plus(when --no-encode=false). Decode to unicode in --no-encode is set.

BUG: 1319717
Change-Id: If5561c749ab5529445650d322c831eb4da22b65a
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/13798
Smoke: Gluster Build System <jenkins@build.gluster.com>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
2016-03-30 06:37:18 -07:00
Aravinda VK
62047c80f5 tools/glusterfind: Do not fail cleanup script on volume delete
If glusterfind dir is not present inside GLUSTERD_WORKDIR, then
cleanup script Post Volume delete was failing. This patch safely
handles the absence of glusterfind directory in GLUSTERD_WORKDIR

BUG: 1289935
Change-Id: I9f7e6035e3dc6079927803df942b0f921218c79a
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/12923
Smoke: Gluster Build System <jenkins@build.gluster.com>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
2016-03-08 01:58:10 -08:00
Aravinda VK
1b897c39ba tools/glusterfind: New option --no-encode
New option added to skip encoding path in output file. Also handled
Unicode strings.

File paths can have newline characters, to differentiate between each
path patch is encoded according to
RFC3986(https://www.ietf.org/rfc/rfc3986.txt). Due to this consumer
applications have to decode the path before consuming it. With this
option Paths are not encoded, can be directly consumed by
applications.

Unicode encoding is handled automatically

BUG: 1310080
Change-Id: I83d59831997dbd1264b48e9b1aa732c7dfc700b5
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/13477
Smoke: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2016-02-26 02:11:08 -08:00
Milind Changire
a56e32e197 tools/glusterfind: add query command to list files
When session information is maintained outside Gluster, there needs to
be some mechanism to list files starting from a time-stamp. This patch
implements the feature via the "query" command-line option.

The only caveat is that the first time the query command is run for the
volume, it will likely report that "historical changelogs are not
available". This is due to the fact that changelogs had not been turned
on for the volume so far. So the volume options need to be turned on
outside glusterfind or the since_time need to be greater than the
current time when the query command is run for the very first time for
the volume. The query command turns on the required volume options for
collecting changelogs.

Change-Id: I6cb7a57a5ecd166210e2eb4deede06d40ccfa996
BUG: 1272006
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/12362
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2015-11-24 22:29:07 -08:00
Aravinda VK
d47323d0e6 tools/glusterfind: StrictHostKeyChecking=no for ssh/scp verification
Also do not use scp command in case copy file from local
node.

Change-Id: Ie78c77eb0252945867173937391b82001f29c3b0
Signed-off-by: Aravinda VK <avishwan@redhat.com>
BUG: 1260918
Reviewed-on: http://review.gluster.org/12124
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
2015-11-23 09:29:18 -08:00
Aravinda VK
6c3895fd13 tools/glusterfind: Handle Keyboard interrupt
Do not print Python traceback when glusterfind command is interrupted

Change-Id: I67383534f965e410fef7ce09798e9d435ef738ae
Signed-off-by: Aravinda VK <avishwan@redhat.com>
BUG: 1243812
Reviewed-on: http://review.gluster.org/11698
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
2015-11-18 20:05:11 -08:00
Niels de Vos
4ac2ff18db build: MKDIR_P is not defined for Makefiles, use mkdir_p instead
Change-Id: Id6d5263eb7b1c53e72a7668e716e9cc4e34b82cd
Reported-by: Milind Changire <mchangir@redhat.com>
BUG: 1198849
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/12553
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
2015-11-10 00:57:44 -08:00
Aravinda VK
cf3d6f14ae tools/glusterfind: Do not show session corrupted if no status file
When a glusterfind session is created it creates session directories
in all the nodes which are part of the Volume. But session status
file only present in initiated node.

Show Session corrupted only if status file exists and invalid content.

Change-Id: I8443c0335b872645b54c0aa77d9893dbe2589c92
Signed-off-by: Aravinda VK <avishwan@redhat.com>
BUG: 1243838
Reviewed-on: http://review.gluster.org/11699
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
2015-08-27 03:08:25 -07:00
Saravanakumar Arumugam
0c47ef608b tools/glusterfind: Prepend prefix in case of delete
In case of delete operation alone, output prefix adding
was not handled earlier. 
Output prefix is added now.

Change-Id: Ia91444dddbff501b26a864f6185ca4c0aaf4c802
BUG: 1244144
Signed-off-by: Saravanakumar Arumugam <sarumuga@redhat.com>
Reviewed-on: http://review.gluster.org/11712
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
2015-08-26 23:21:23 -07:00
Niels de Vos
454bd09b8b rpm: include required directory for glusterfind
The directory was marked as %ghost, which causes the following
installation failure:

  Error unpacking rpm package glusterfs-server-3.8dev-0.446.git45e13fe.el7.centos.x86_64
  error: unpacking of archive failed on file /var/lib/glusterd/hooks/1/delete/post/S57glusterfind-delete-post.py;5581f20e: cpio: open

Also, *all* Python files should be part of the RPM package. This
includes generated .pyc and .pyo files.

BUG: 1225465
Change-Id: Iee74905b101912c4a845257742c470c3fe42ce2a
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/11298
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
2015-08-19 03:33:14 -07:00
Milind Changire
9361e2bf05 tools/glusterfind: password prompts for peer nodes on delete
SSH keys getting deleted on local node caused password prompts for peer
nodes on a delete operation.

This race for session directory cleanup on local node is now addressed
by avoiding the delete path for local node in node_cmd()
The session directory tree is deleted anyway in mode_delete()

Change-Id: I51c4baf4f9c1ed3caa319d4163bef343bd621429
BUG: 1243785
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/11693
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2015-08-04 23:26:37 -07:00
Aravinda VK
adc704557a tools/glusterfind: RENAME and MODIFY issues
If Modification happens before RENAME, GFID to Path Conversion
converts it into New Path. Delete Modify Entry and insert again
So that MODIFY <NEW NAME> comes after RENAME.

Default value of pgfids and basenames changed to "" instead of NULL
Also fixed RENAME issue of displaying "RENAME <NEW NAME> <NEW NAME>".
Also fixed RENAME followed by missing MODIFY

Change-Id: I8202f6e6ec33f7bd921e71da38677f2ee2dab87a
BUG: 1236270
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/11443
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
2015-07-05 23:09:10 -07:00
Kotresh HR
a0e9c7d52d tools/glusterfind: Fail glusterfind creation if volume is offline
Following two fixes are done.

1. Fail glusterfind session creation if volume is not online
   even before session directories are created. This avoids
   'glusterfind list' to pick the session directories and show
   status as 'Session Corrupted'.

2. Check of '!Started' instead of wether the volume is 'Stopped'.
   It covers all the cases.

Change-Id: I3e9cb384d978ada28f508c07e37d6ceb2272a731
BUG: 1232729
Signed-off-by: Kotresh HR <khiremat@redhat.com>
Reviewed-on: http://review.gluster.org/11278
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2015-06-18 23:04:04 -07:00
Milind Changire
7d8b53d5fc tools/glusterfind: print message for good cases
Added messages for printing status of good or default cases
for 'create', 'list', 'post' and 'delete' commands

Change-Id: Ie983bff7f5a66f118de06c225166ff6bee870438
BUG: 1230017
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/11208
Reviewed-by: Aravinda VK <avishwan@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
2015-06-13 21:51:33 -07:00
Milind Changire
b0edf64f91 tools/glusterfind: ignoring deleted files
OSError and IOError exceptions were being thrown if files
were deleted after session was created and a subsequent
glusterfind pre was attmepted. glusterfind now detects
this scenario and safely ignores these changes to the
file-system. We also avoid recording deleted file paths
into database in the case where gfid to path resolution
cannot be performed for deleted files.

Also, we now turn on volume option to capture delete paths.

Change-Id: Iaf3883463f5e64643497bd77229ad80a9b2d5da5
BUG: 1228111
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/11194
Reviewed-by: Aravinda VK <avishwan@redhat.com>
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
2015-06-12 07:13:18 -07:00
Aravinda VK
9e72abb08f tools/glusterfind: Cleanup session dir after delete
Sessions directories are created in
/var/lib/glusterd/glusterfind/<SESSION>/<VOLUME>, only <VOLUME>
dir is created since same session name can exists for two volumes.

Now cleanup is added, <SESSION> dir will be removed if <SESSION>
contains only one Volume.

Change-Id: I236393659e63eb98342570e0255739f4adea4491
Signed-off-by: Aravinda VK <avishwan@redhat.com>
BUG: 1230111
Reviewed-on: http://review.gluster.org/11157
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Vijay Bellur <vbellur@redhat.com>
2015-06-12 03:07:47 -07:00
Aravinda VK
076e869c65 tools/glusterfind: Fix GFID to Path conversion for dir
For Directories, GFID to Path conversion was fine, but wrong
filter was used while updating to db.

Change-Id: Id588adbb25ff5fd357f8cc2d5485bd2eb4a52b65
Signed-off-by: Aravinda VK <avishwan@redhat.com>
BUG: 1230127
Reviewed-on: http://review.gluster.org/11158
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
2015-06-11 12:28:12 -07:00
Milind Changire
53e84fe1ee tools/glusterfind: verifying volume is online
Volume needs to be online (Started) to perform meaningful
operations i.e. for "create" and "pre". This is now verified.

Change-Id: I207cf1f5e30be0f67a776b866349b59d70a13984
BUG: 1225564
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/10955
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2015-06-11 12:05:56 -07:00
Milind Changire
c2898f0409 tools/glusterfind: verifying volume presence
An invalid volume name passed to "pre", "post" and  "delete"
operations throws up unfriendly errors. This patch presents
friendly error messages.

Change-Id: I233809653379b80b34c2e74c0b6f4fe502c8961e
BUG: 1225424
Signed-off-by: Milind Changire <mchangir@redhat.com>
Reviewed-on: http://review.gluster.org/10945
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
2015-06-04 00:08:48 -07:00
Aravinda VK
e88837ed0f tools/glusterfind: GFID to Path conversion using Changelog
Records fop information collected from Changelogs in sqlite database.
This is only working database, not required after processing.

After post processing, output file is generated by reading these
database files.

This is applicable only in incremental run, When a changelog is
parsed, all the details are saved in Db. GFID to Path is converted
to those files for which information is available in Changelogs.

For all the failed cases, it tries to convert to Path using Pgfid,
if not found GFID to Path is done using find.

BUG: 1201284
Change-Id: I53f168860dae15a0149004835e67f97aebd822be
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/10463
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
2015-05-08 21:59:10 -07:00
Aravinda VK
e561935d01 tools/glusterfind: Partial Find
Enabled by default, if one node fails Glusterfind will not fail
to return list of files from other nodes. This behavior can be changed
using --disable-partial

Now session is maintained in each nodes as well as in initiator node.
Every pre command will pick the status file from respective node and
start collecting list of changes happened after the status time.

--reset-session-time, new option to force reset the session time.
Next incremental run will start from this time.

Change-detector argument is removed since Changelog mode is required
to detect deletes and Renames.

Change-Id: I0b83bc7c0e1b30b13de772b2d21fe968db4ff964
Signed-off-by: Aravinda VK <avishwan@redhat.com>
BUG: 1201289
Reviewed-on: http://review.gluster.org/10320
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
2015-05-08 21:19:06 -07:00
Aravinda VK
20353cc323 tools/glusterfind: New option to pre --regenerate-outfile
When pre command is run twice, it overwrites the outfile.
Now pre command will fail when executed twice. To force the
regeneration use --regenerate-outfile

Change-Id: I0cf7a139522812ece4decdfbcba667a05ce5c35e
Signed-off-by: Aravinda VK <avishwan@redhat.com>
BUG: 1207028
Reviewed-on: http://review.gluster.org/10418
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
2015-05-04 02:27:26 -07:00
Aravinda VK
0fc310accb tools/glusterfind: Do not fail to list if Session is invalid
If session is invalid, Show the session time as "Session Corrupted"

BUG: 1209843
Change-Id: I10f7fb79f6a3a2cabab3f21bce86f2a2f424e8b6
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/10164
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Tested-by: NetBSD Build System
2015-04-23 06:56:06 -07:00
Emmanuel Dreyfus
1fbcecb72e Fix case mistake for MKDIR_P in Makefiles
Some makefiles used $(mkdir_p) instead of the corectly defined $(MKDIR_P).
The former is substituted as an empty string, leading to possible failures
depending of the user shell tolerance. NetBSD's /bin/sh seems to choke
more easily than Linux's /bin/bash, but if the later does not fail, it
does not created the intended directories anyway.

BUG: 1129939
Change-Id: I8caed4000f3c91cb3a685453848fb854793945ed
Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org>
Reviewed-on: http://review.gluster.org/10276
Tested-by: NetBSD Build System
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
2015-04-20 04:29:26 -07:00
Aravinda VK
5cb5d70292 tools/glusterfind: Prevent ssh public key overwrite issue
Same ssh key was used for all the sessions, when multiple
sessions created in Cluster, public keys get overwritten by
newest sessions. Moved ssh keys to respective session dir.

BUG: 1206547
Change-Id: I3d8fac9b24bc7c71445c7b4deae83104693e7dab
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/10150
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Tested-by: Venky Shankar <vshankar@redhat.com>
2015-04-13 08:56:30 +00:00
Aravinda VK
3ed7d6de25 tools/glusterfind: Check existence of Session
Use glusterfind list command to check a session is valid or not

    glusterfind list --session <SESSION> --volume <VOLUME>

Session exists if return code($?) is zero else invalid session

Change-Id: I37faa09e4e1bf2f10725dd8e5b0cc36a36a3cad4
Signed-off-by: Aravinda VK <avishwan@redhat.com>
BUG: 1205057
Reviewed-on: http://review.gluster.org/10154
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
2015-04-09 09:52:33 +00:00
Aravinda VK
0d36d4f6ba tools/glusterfind: Validate Volume name before session Create
Do not create Session directories if Volume does not exists.

Change-Id: I60c9107135c2bed2a7198582267b2328dec7e868
Signed-off-by: Aravinda VK <avishwan@redhat.com>
BUG: 1206065
Reviewed-on: http://review.gluster.org/10072
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Tested-by: Venky Shankar <vshankar@redhat.com>
2015-04-01 10:35:20 -07:00
Aravinda VK
ad0690d3bf tools/glusterfind: Convert Output file path to absolute path
BUG: 1203656
Change-Id: Ie6c8f38894466246c453b2a863538a420acfe9f7
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/10011
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Tested-by: Venky Shankar <vshankar@redhat.com>
2015-04-01 10:33:27 -07:00
Aravinda VK
e3bd2387a5 tools/glusterfind: Changelog_init before changelog_register
With the new Changelog RPC based approach, Changelog_init
needs to be called before changelog_register

BUG: 1206127
Change-Id: I817b84016d3f9ffdf2ea7a4e177953fe14ccc323
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/10009
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
2015-03-30 05:50:25 -07:00
Aravinda VK
6a3024fe59 tools/glusterfind: Ignore .trashcan dir during Brick Crawl
BUG: 1203650
Change-Id: Icf81726ad34bd5cfec6d8e06763b02a159e84c2e
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/10010
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
2015-03-30 05:35:32 -07:00
Aravinda VK
7e98a0e0b1 feature/glusterfind: A tool to find incremental changes
Documentation is available in patch:
http://review.gluster.org/#/c/9800/

A tool which helps to get list of modified files or list of all files in
GlusterFS Volume using Changelog or find command.

Usage
=====
glusterfind --help

Create:
-------
glusterfind create --help

The tool creates status file $GLUSTERD_WORKDIR/SESSION/VOLUME/status
and records current timestamp to initiate the session. This timestamp
will be used as start time for next runs.

As part of create also generates ssh key and distributes to all peers.
and enables build.pgfid and changelog using volume set command.

Pre:
----
glusterfind pre --help

This command is used to generate the list of files modified after session
creation time or after last run. To get list of all files/dirs in Volume,
run pre command with `--full` argument.

The tool gets all nodes details using gluster volume info and runs node
agent for each brick in respective nodes via ssh command. Once these node
agents generate the output file, tool copies to local using scp. Merges all
the output files to generate the final output file.

Post:
-----
glusterfind post --help

After consuming the list, this sub command is called to update the session
time based on pre command status file.

List:
-----
glusterfind list --help

To view all the sessions

Delete:
-------
glusterfind delete --help

Delete session.

Known Issues
------------
1. Deleted files will not get listed, since we can't convert GFID to
   Path if file/dir is deleted.
2. Only new name will get listed if Renamed.
3. All hardlinks will get listed.

Change-Id: I82991feb0aea85cb6ec035fddbf80a2b276e86b0
BUG: 1193893
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/9682
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
2015-03-18 02:54:25 -07:00