I’m trying to extract the statical data for multiple subjects using the aparcstats2table program in freesurfer.
I used this command to extract the statical data for one subject.
aparcstats2table --subjects /fefs1/fcit/ezahamalyoubi/freesurfer/subjects/MCI_Female_Subjects/I4688/ --hemi lh --meas volume --parc=aparc --tablefile=I4688_lh_volume.txt
how can I edit this command to work for many subjects and store the statical data in one txt file?
Any clarification will help
-Esraa
Just add the list of subjects to the '--subjects' option:
export SUBJECTS_DIR=/fefs1/fcit/ezahamalyoubi/freesurfer/subjects/MCI_Female_Subjects
SUBJ=$(ls ${SUBJECTS_DIR} | grep -v fsaverage)
echo "Extracting stats for ${SUBJ}"
aparcstats2table --subjects ${SUBJ} --hemi lh --meas volume --parc=aparc --tablefile=lh_volume.txt
Related
I own a small minecraft server, and I would like to create a google spreadsheet for calculating user playtime data. I wan't this data because it would help let me know if my advertising campaigns are working or not. You can try to eyeball this stuff, but a solid data set would be alot more effective than guessing if the advertising is effective. The problem lies in the fact that manually searching for data from the server logs is really hard. I would appreciate anyone who could help me build a simple script or something that reads a .txt file and extracts the data I need. The script needs to be able to:
Detect lines with "User Authenticator" and "Disconnected" then print the entire line.
Format the text in some way? Possibly alphabetize the lines so that were not all over the place looking for specific users logins and logouts, defeating the purpose of the script. Not sure if this is possible.
Exclude lines with certain text (usernames), we want normal player data, not admin data.
I am sorry if did anything wrong, this is my first time on the site.
UPDATE: The admin data would be stored in a file called "admins.txt". By "alphabetizing" i meant it, example: Player A joins at 06:00, Player B joins at 06:30, then, Player A leaves at 06:45, Player B leaves at 07:00. If the data was flat, it would end up reading something like: A: 6:00, B: 6:30, A:6:45, B:7:00. But I would rather it be: A: 6:00, A: 6:45, B: 6:30, B: 7:00. That would make it easier to chart it out and make a calculation. Sorry for the long text.
Also typical server logging looks like this:
[15:46:30] [User Authenticator #1/INFO]: UUID of player DraconicPiggy is (UUID)
[15:46:31] [Server thread/INFO]: DraconicPiggy[/(Ip address)] logged in with entity id 157 at ([world]342.17291451961574, 88.0, -32.04791955684438)
The following awk script will report only on the two line types that you mentioned.
/User Authenticator|Disconnected/ {
print
}
I'm guessing "alpabetize" means sort. If so then you can pass the awk output to sort via a pipe.
awk -f script.awk | sort
I'm assuming the file is a log that's already in date-time sequence, with the timestamps at the start of the line. In this case you'll need to tell sort what to sort on.sort /? will tell you how to do this.
Multiple input files
To process all log files in the current directory use:
awk -f script.awk *.log
Redirect output to file
The simplest way is by adding > filtered.log to the command, like this:
awk -f script.awk *.log > filtered.log
That will filter all the input files into a single output file. If you need to write one filtered log for each input file then a minor script change is needed:
/User Authenticator|Disconnected/ {
print >> FILENAME ".filtered.log"
}
Filtering admins and redirecting to several files
The admins file should be similar to this:
Admin
DarkAdmin
PinkAdmin
The admin names must not contain spaces. i.e DarkAdmin is OK, but Dark Admin woud not work. Similarly the user names in your log files must not contain spaces for this script to work.
Execute the following script with this command:
awk -f script.awk admins.txt *.log
Probably best to make sure the log files and the filtered output are in separate directories.
NR == FNR {
admins[ $1 ] = NR
next
}
/User Authenticator|Disconnected/ {
if ( $8 in admins ) next
print >> FILENAME ".filtered.log"
}
The above script will:
Ignore all lines that mention an admin.
Creat a filtered version of every log file. i.e. if there are 5 log files then 5 filtered logs will be created.
Sorting the output
You have two sort keys in the file, the user and the time. This is beyone the capabilities of the standard Windows sort program, which seens very primitive. Yo should be able to do it with Gnu Sort:
sort --stable --key=8 test.log > sorted_test.log
Where:
--key=8 tells it to sort on field 8 (user)
--stable keeps the files in date order within each user
Example of sorting a log file and displaying the result:
terry#Envy:~$ sort --stable --key=8 test.log
[15:23:30] [User Authenticator #1/INFO]: UUID of player Doris is (UUID)
[16:36:30] [User Disconnected #1/INFO]: UUID of player Doris is (UUID)
[15:46:30] [User Authenticator #1/INFO]: UUID of player DraconicPiggy is (UUID)
[16:36:30] [User Disconnected #1/INFO]: UUID of player DraconicPiggy is (UUID)
[10:24:30] [User Authenticator #1/INFO]: UUID of player Joe is (UUID)
terry#Envy:~$
I have a text file that contain large amount of data. Below shows some part of the data. I am required to create a separate European subset file. How do I filter them out using awk?
File columns are as follows:
User ID, Latitude, Longitude, Venue category name, Country code(2-letter)
Text file containing:
3fd66200f964a52008e61ee3 40.726589 -73.995649 Deli / Bodega US
4eaef4f4722e4efd614ddb80 51.515470 -0.148605 Burger Joint GB
4eaef8325c5c7964424125c8 50.739561 4.253660 Vineyard BE
4e210f60d22d0a3f59f4cbfb 5.367963 103.097516 Racetrack MY
52373a6511d2d4fcba683886 41.434926 2.220326 Medical Center ES
476f8da1f964a520044d1fe3 40.695163 -73.995448 Thai Restaurant US
New text file should look like this:
4eaef4f4722e4efd614ddb80 51.515470 -0.148605 Burger Joint GB
4eaef8325c5c7964424125c8 50.739561 4.253660 Vineyard BE
52373a6511d2d4fcba683886 41.434926 2.220326 Medical Center ES
Note: I can either user latitude longitude bounding box or country code to extract the subset into a new file.
First you need the country codes for the required countries (or all the latitudes and longitudes and corresponding country codes :) in a separate file to check against:
$ cat countries.txt
GB
BE
ES
In awk:
$ awk 'NR==FNR{a[$0];next} $NF in a' countries.txt file.txt
4eaef4f4722e4efd614ddb80 51.515470 -0.148605 Burger Joint GB
4eaef8325c5c7964424125c8 50.739561 4.253660 Vineyard BE
52373a6511d2d4fcba683886 41.434926 2.220326 Medical Center ES
Explained:
NR==FNR { # this block {} is only processed for the first file (take it for granted)
a[$0] # this initializes an array element in a, for example a["GB"]
next # since we only initialize an element for each country code in the first file
# no need to process code beyond this point, just skip to NEXT country code
} # after this point we check whether country code exists in array a
$NF in a # if element in array a[] for value $NF in last field NF (for example a["GB"])
# of second file was initialized, it is required row and is printed.
# this could've been written: { if($NF in a) print $0 }
Using grep:
grep -wFf countries.txt file.txt
Explanation of options:
-F fixed string search (no regex)
-f specifies a file of patterns
-w matches whole words only
I have a CSV file containing the data that I want to convert into a graph database using Neo4j. The Columns in the file are in the following format :
Person1 | Person2 | Points
Now the ids in Person1 and Person2 are redundant , so I am using a Merge statement instead. But I am not getting the correct results.
For a sample dataset , the output seems to be correct , but when I import my dataset consisting of 2M rows, it somehow doesn't create the relationships.
I am putting the cypher code that I am using currently.
USING PERIODIC COMMIT 1000
LOAD CSV WITH HEADERS FROM "file:C:/Users/yogi/Documents/Neo4j/default.graphdb/sample.csv" AS csvline
MERGE (p1:Person {id:toInt(csvline.id1)})
MERGE (p2:Person {id:toInt(csvline.id2)})
CREATE (p1)-[:points{count:toInt(csvline.c)}]->(p2)
Some things you should check:
are you using an index: CREATE INDEX ON :Person(id) should be run before the import
depending on the Neo4j version you're using, the statement might be subject to "eager-pipe" which basically prevents the periodic commit. For more on eager pipe, see http://www.markhneedham.com/blog/2014/10/23/neo4j-cypher-avoiding-the-eager/
I want to import several time-series variables for a list of countries via an API function. I want to store the data in a separate array for each country. The name of the data array should be the 3 letter country code. For one country my code looks like this:
data = Quandl(c(var1,var2,var3), collapse="annual")
I want to loop through a list of countries that also contains the identifiers to download var1-3 from Quandl.com. My problem is that I cannot figure out a way to rename the data array (data) for each country with the respective country code (e.g. USA). In the end I want to have a separate data arrays for each country on my list in the workspace (POL, DEU, USA, CHN ...).
Basically, I'm looking for a functional way to do this:
list[i,1] = data
unfortunately, this isn't working but why isn't there a simple way to rename a data array within a loop?
you could create a list of of the data using lapply and then rename each element of the list with the country codes.
countries<-c("a","b","c")
dat<-lapply(countries,function(x) rnorm(4))
names(dat)<-countries
> dat[["a"]]
[1] -0.5157099 -1.0372721 0.9696698 -0.9280852
You won't have a variable named POL or USA, but you'll have a list from which you can extract the relevant dataframe. I think this might be a cleaner solution than creating variables for each country.
You can probably do away with your loop by looking into one of the apply functions (apply, mapply, sapply, lapply etc). Consider the following:
The outer lapply performs the function of the overall higher country-level "loop", while the inner mapply performs a given set of operations for various arguments for that single country
countries<-c("a","b","c")
arg1<-1:3
arg2<-10*1:3
dat<-mapply(data.frame(c1=x,c2=rnorm(1)*vars),vars,x,SIMPLIFY=F))
names(dat)<-countries
dat<-lapply(countries, function(x)
do.call(rbind,
mapply(function(x,y,z) data.frame(country=x,c1=rnorm(2)*y,c2=rnorm(2)*z),
x=x,y=arg1,z=arg2,SIMPLIFY=F)))
If you still want to stick with your for loop then create a list and add into it and rename as done above
result<-list()
for(i in 1:50) result[[i]]<-YOURFUNCTIONHERE
names(result)<-countryNames
I read in "hadoop design pattern" book, "HBase supports batch queries, so it would be ideal to buffer all the queries we want to execute up to some predetermined size. This constant depends on how many records you can comfortably store in memory before querying HBase."
I tried to search some examples online but couldn't find any, can someone show me the example using java map reduce?
Thanks.
Dan
Is this what you want? You can save HBase Get object in a list and submit the list at the same time. It's a little better than invoke table.get(get) multiple times.
Configuration conf = HBaseConfiguration.create();
pool = new HTablePool(conf, 5);
HTableInterface table = pool.getTable('table');
List<Get> gets = new ArrayList<Get>();
table.get(gets);