Got only one huge file after solr index merge with IndexMergeTool - solr

Thanks for notice.
I'm a new one for solr, and I'm using the IndexMergeTool to merge solr index on 2 servers.
Indexes on server_1 has size 1.4GB with lots of files, and on server_2, the size is 380MB.
After my running:
java -cp /migration/lucene-2.9.1/lucene-core-2.9.1.jar:/migration/lucene-2.9.1/contrib/misc/lucene-misc-2.9.1.jar org.apache.lucene.misc.IndexMergeTool /tmp/newIndex /server_1/data/solr/data/index /server_2/data/solr/data/index
I got only one huge file:
$ ls -lah /tmp/newIndex/
total 1.5G
drwxr-xr-x 2 root root 4.0K Sep 24 08:08 .
drwxrwxrwt 9 root root 4.0K Sep 24 08:06 ..
-rw-r--r-- 1 root root 1.5G Sep 24 08:08 _3.cfs
-rw-r--r-- 1 root root 284 Sep 24 08:08 segments_2
-rw-r--r-- 1 root root 20 Sep 24 08:08 segments.gen
Is it correct? If it will harm the performance when there is only one huge file like that?
Or is there any options for the tool that I can set it to generate multiple small files?
Thanks again for notice.

Yes, it's expected behavior.
No, it will not affect your
performance, your index will be the same, just with more docs (but
you do that for your needs, right?), so it's same as you will add more docs by yourself.
No, there is no any options
for this tool to generate small files.

Related

what is the .stt file used for in TDengine database?

what is the .stt file used for in TDengine database ?
under the dataDir some of the file is called .stt like this :
ssn#TDengine:/var/lib/taos/vnode/vnode14$ cd ..
ssn#TDengine:/var/lib/taos/vnode$ ls -ltR | grep -i stt
-rwxrwxrwx 1 root root 4096 Jan 11 10:19 v18f1736ver22.stt
-rwxrwxrwx 1 root root 4096 Jan 11 10:19 v19f1736ver16.stt
-rwxrwxrwx 1 root root 4096 Jan 10 20:00 v16f1736ver18.stt
-rwxrwxrwx 1 root root 4096 Jan 10 20:01 v17f1736ver27.stt
may I know what is it for ?
a specific description for this file ,what is it used ,does it impact the database performance,etc.
the .sst file is equivalent to the .last file in the TDengine database 2.0
it is used to store the data fragment that smaller than minrows configuration .

How to implement runas in Linux?

I’ve tried fork, setresgid, setresuid and execvpe — doesn’t work.
Specifically, no errors are returned anywhere, the child process starts OK, I have confirmed all 6 magic numbers returned by getresuid and getresgid match, yet the child process doesn’t have the required permissions.
The files in question are in the /dev/input/ folder. Here’s the permissions:
drwxr-xr-x 3 root root 180 Sep 23 19:47 .
drwxr-xr-x 17 root root 4120 Sep 23 19:47 ..
drwxr-xr-x 2 root root 160 Sep 23 19:47 by-path
crw-rw---- 1 root input 13, 64 Sep 23 19:47 event0
crw-rw---- 1 root input 13, 65 Sep 23 19:47 event1
crw-rw---- 1 root input 13, 66 Sep 23 19:47 event2
crw-rw---- 1 root input 13, 67 Sep 23 19:47 event3
crw-rw---- 1 root input 13, 68 Sep 23 19:47 event4
crw-rw---- 1 root input 13, 69 Sep 23 19:47 event5
The user is a member of the input group, that’s why under normal circumstances it can access these files. However, the process launched with fork/setresgid/setresuid/execvpe can’t access these files.
Here’s relevant lines from the strace log, the log was made with -ff option i.e. only includes a single process:
setresgid(10000, 10000, 10000) = 0
setresuid(10000, 10000, 10000) = 0
execve("/usr/local/bin/dotnet", ["/usr/local/bin/dotnet", "/home/user/launcher/Debug/Desktop.dll"], 0xffffea463f50 /* 1 var */) = 0
.. much later
openat(AT_FDCWD, "/dev/input/event0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 EACCES (Permission denied)
The problem was indeed extra groups. Contrary to my expectation, setresuid does not refresh the list of additional groups.
To set them up, the launcher process first needs to call getgrouplist to query the list of additional groups for the target user, then after the fork call setgroups to apply these values.
The complete sequence of kernel calls to implement a Linux equivalent of CreateProcessAsUser is following:
getgrouplist, fork, chdir, setresgid, setgroups, setresuid, execvpe
Note how setresuid needs to be the very last step before the execvpe. The reason for that, after that call the forked process no longer has permissions to modify these security-related things.
P.S. I wonder how many security bugs in Linux software were caused by the developers forgetting to update that list of extra groups when switching user accounts in their programs?

How to destine the configure generated files to special path?

How to destine the configure output files path?
in some case I only can configure like this:
/root/src/my_software-1.8.2/configure
in my case, I can not cd into /root/src/my_software-1.8.2/.
I will generate these files to /:
2 00:06 ..
-rwxr-xr-x. 1 root root 0 Sep 27 07:32 .dockerenv
-rw-r--r--. 1 root root 24790 Oct 2 00:06 Makefile
drwxr-xr-x. 2 root root 53 Sep 28 11:24 att
lrwxrwxrwx. 1 root root 7 May 11 2019 bin -> usr/bin
-rw-r--r--. 1 root root 3934 Oct 2 00:06 config.h
-rw-r--r--. 1 root root 31584 Oct 2 00:06 config.log
-rwxr-xr-x. 1 root root 33874 Oct 2 00:06 config.status
I want destine these generated files to special output path(/root/src/my_software-1.8.2/)? how?
EDIT-01
In fact, I use the command to execute script:
docker exec -it centos-03 "my command"
but I can not execute the compound command like this:
docker exec -it centos-03 "cd /root/src/my_software-1.8.2/;./configure"
so I want use one command to do that.

Choose BSD or sys5 style when creating file in linux

When I create file in linux default group owner becomes gid of process which creates file. If I add SGID to parent directory file will inherit parent directory owner group. Also I can change fs mount options to behave either like sys5 or like BSD.
What if I want to choose this option regardless directory permissions and fs mount options? Is there c function option or syscall parameter which allows you to choose group owner?
$ find . -ls
262 4 drwxrwxr-x 4 devops devops 4096 Apr 24 18:01 .
999 4 drwxrwxr-x 2 devops root 4096 Apr 24 18:03 ./dir1
6093 4 drwxrwsr-x 2 devops root 4096 Apr 24 18:03 ./dir2
$ touch dir1/file dir2/file
$ find . -ls
262 4 drwxrwxr-x 4 devops devops 4096 Apr 24 18:01 .
999 4 drwxrwxr-x 2 devops root 4096 Apr 24 18:04 ./dir1
5576 0 -rw-rw-r-- 1 devops devops 0 Apr 24 18:04 ./dir1/file
6093 4 drwxrwsr-x 2 devops root 4096 Apr 24 18:04 ./dir2
6094 0 -rw-rw-r-- 1 devops root 0 Apr 24 18:04 ./dir2/file
$
And I wish to have something like that:
$ mytouch -s BSD dir1/file1
$ mytouch -s sys5 dir1/file2
$ find dir1 -ls
999 4 drwxrwxr-x 2 devops root 4096 Apr 24 18:10 dir1
6213 0 -rw-rw-r-- 1 devops root 0 Apr 24 18:10 dir1/file1
6214 0 -rw-rw-r-- 1 devops devops 0 Apr 24 18:10 dir1/file2
$
Chances are, you can't.
The implementation of sticky bits exists entirely within the kernel, and there are no options to open() or creat() which control how it operates.
Your program could conceivably call chown() to manually reset the group of the file after creating it. However, this would only work reliably if your process is running as root, or as a member of the group that owns the parent directory.

Abort BGSAVE Already in process

I started saving redis-db snapshot by calling BGSAVE command in redis-cli.
It has started running but I keep getting these errors in the logs
[30853] 27 Jan 07:18:41.129 # Background saving error
[30853] 27 Jan 07:18:47.043 * 1 changes in 900 seconds. Saving...
[30853] 27 Jan 07:18:47.058 * Background saving started by pid 13204
[13204] 27 Jan 07:18:47.058 # Failed opening .rdb for saving: Permission denied
[30853] 27 Jan 07:18:47.158 # Background saving error
[30853] 27 Jan 07:18:53.070 * 1 changes in 900 seconds. Saving...
[30853] 27 Jan 07:18:53.085 * Background saving started by pid 13207
[13207] 27 Jan 07:18:53.085 # Failed opening .rdb for saving: Permission denied
[30853] 27 Jan 07:18:53.186 # Background saving error
[30853] 27 Jan 07:18:59.098 * 1 changes in 900 seconds. Saving...
[30853] 27 Jan 07:18:59.113 * Background saving started by pid 13210
[13210] 27 Jan 07:18:59.114 # Failed opening .rdb for saving: Permission denied
[30853] 27 Jan 07:18:59.213 # Background saving error
looks like the redis BGSAVE command is running indefinitely. How to stop this.
Also I tried checking for process pid by ps -aux| grep redis command.
13196 pts/11 S+ 0:00 grep --color=auto redis
30853 ? Ssl 1292:57 /usr/bin/redis-server *:6379
There is no process to kill.
EDIT: These are the permissions to redis folder and dump.rdb file
f: /var/lib/redis
drwxr-xr-x root root /
drwxr-xr-x root root var
drwxr-xr-x root root lib
drwxr-xr-x redis redis redis
f: /var/lib/redis/dump.rdb
drwxr-xr-x root root /
drwxr-xr-x root root var
drwxr-xr-x root root lib
drwxr-xr-x redis redis redis
-rw-rw-rw- redis redis dump.rdb
EDIT2: Got the answer. The problem was somehow the config parameters are changed. The dbfilename and dir values are changed.
Set these values to original through CONFIG SET command and now its working fine. Adding in-case somebody has same problem.
But the question is how did they change. Did this happen to anybody else?
Help me
Thanks
You can either try and fix the file permissions error (does the default save location exist and does redis have permission to write to it?) or you can disable saving with:
config set save ""

Resources