I am performing a scatter-gather operation on NextFlow.
It looks like the following:
reads = PATH+"test_1.fq"
outdir = "results"
split_read_ch = channel.fromFilePairs(reads, checkIfExists: true, flat:true ).splitFastq( by: 10, file:"test_split" )
process Scatter_fastP {
tag 'Scatter_fastP'
publishDir outdir
input:
tuple val(name), path(reads) from split_read_ch
output:
file "${reads}.trimmed.fastq" into gather_fatsp_ch
script:
"""
fastp -i ${reads} -o ${reads}.trimmed.fastq
"""
}
gather_fatsp_ch.collectFile().view().println{ it.text }
I run this code with all the benchmarks options proposed by Nextflow (https://www.nextflow.io/docs/latest/tracing.html):
nextflow run main.nf -with-report nextflow_report -with-trace nextflow_trace -with-timeline nextflow_timeline -with-dag nextflow_dag.html
In these tracing files, I can find the resources and speed of the 10 Scatter_fastP processes.
But I would like to also measure the resources and speed of the creation of the split_read_ch and the gather_fastp_ch channels.
I have tried to include the channels' creation in processes but I cannot find a solution to make it work.
Is there a way to include the channel creation into the tracing files? Or is there a way I have not found to create these channels into processes?
Thank you in advance for your help.
Although Nextflow can parse FASTQ files and split them into smaller files etc, generally it's better to pass off these operations to another process or set of processes, especially if your input FASTQ files are large. This is beneficial in two ways: (1) your main nextflow process doesn't need to work as hard, and (2) you get granular task process stats in your nextflow reports.
The following example uses GNU split to split the input FASTQ files, and gathers the outputs using the groupTuple() operator and the groupKey() built-in to stream the collected values as soon as possible. You'll need to adapt for your non-gzipped inputs:
nextflow.enable.dsl=2
params.num_lines = 40000
params.suffix_length = 5
process split_fastq {
input:
tuple val(name), path(fastq)
output:
tuple val(name), path("${name}-${/[0-9]/*params.suffix_length}.fastq.gz")
shell:
'''
zcat "!{fastq}" | split \\
-a "!{params.suffix_length}" \\
-d \\
-l "!{params.num_lines}" \\
--filter='gzip > ${FILE}.fastq.gz' \\
- \\
"!{name}-"
'''
}
process fastp {
input:
tuple val(name), path(fastq)
output:
tuple val(name), path("${fastq.getBaseName(2)}.trimmed.fastq.gz")
"""
fastp -i "${fastq}" -o "${fastq.getBaseName(2)}.trimmed.fastq.gz"
"""
}
workflow {
Channel.fromFilePairs( './data/*.fastq.gz', size: 1 ) \
| split_fastq \
| map { name, fastq -> tuple( groupKey(name, fastq.size()), fastq ) } \
| transpose() \
| fastp \
| groupTuple() \
| map { key, fastqs -> tuple( key.toString(), fastqs ) } \
| view()
}
Using VBS on Windows 2012 R2 I am trying to pass a command line parameter isActive="false" but I cannot get the equal sign to appear in the command line.
Create a dummy batch file like test.bat
echo %1
Pause
Then in the created VBScript
Set oShell = WScript.CreateObject("WSCript.Shell")
oShell.CurrentDirectory = "C:\test"
'strEqual = Chr(61)
strCommand = "test.bat" & " " & "isActive=" &"""false"""
return = oShell.Run(strCommand, 1, True)
Set oShell = Nothing
I get isActive "false" but no equal sign.
I have tried separating out as a unique value
like & Chr(61) & and have tried escaping with / and \ and // and \\ before and after the equal sign. I have tried to use as a variable, strEqual = Chr(61).
I am at a loss as to how to get the = to be part of the string when passed to the command shell. I can write it to a text file and the equal sign is written, but not in the shell.
You observe this behavior because CMD uses not only spaces and tabs, but also commas, semicolons, and the = character as parameter delimiters. Meaning that isActive="false" is parsed as 2 distinct arguments: isActive and "false". If you want it to be parsed as a single argument you need to put the whole key/value pair in quotes: "isActive=false".
Note that the double quotes inside VBScript string literals must be escaped by doubling them. If you require double quotes around the value part of the argument simply add another set of escaped double quotes.
Set oShell = CreateObject("WScript.Shell")
strCommand = "test.bat ""isActive=""false"""""
return = oShell.Run(strCommand, 1, True)
There is also no need to concatenate string literals (except for readability reasons when you want to wrap a long string). Just define your command as a single string.
You may pass a value from VBS to BAT/CMD using process environment variable.
Save the below code as test.vbs:
strCurDir = CreateObject("Scripting.FileSystemObject").GetParentFolderName(WScript.ScriptFullName) & "\"
CreateObject("WScript.Shell").Environment("process").Item("myvar") = "isActive=""false"""
CreateObject("WSCript.Shell").Run strCurDir & "test.bat"
And this code save as test.bat in the same folder:
echo %myvar%
pause
Run test.vbs and console output will be
C:\Windows\system32>echo isActive="false"isActive="false"
I want to include files from a folder in my script (I tried to Google but can't seem to find any way to do this with AutoIt). Example of what I want to achieve:
LoadFiles()
Func LoadFiles()
$FL = _FileListToArray(#ScriptDir&"\Test\", "*")
$X=1
Do
#include $FL[$X] <== How ?
$X=$X+1
Until $X=$FL[0]
EndFunc
Can anyone point me in the right direction?
In order to include a file(s) in your compiled script, you need FileInstall.
FileInstall ( "source", "dest" [, flag = 0] )
source The source path of the file to compile. This must be a literal string; it cannot be a variable or the result of a function call. It can be a relative path (using .\ or ..\ in the path) to the source file (.au3).
dest The destination path of the file with trailing backslash if only the directory is used. This can be a variable.
flag [optional] this flag determines whether to overwrite files if they already exist:
$FC_NOOVERWRITE (0) = (default) do not overwrite existing files
$FC_OVERWRITE (1) = overwrite existing files
Another way is adding the files as a resource
Good afternoon! There isn't currently a good way to do what you're asking. I've been working on building a UDF to do what you'd like, but I've been running into a couple of issues with it. I have a working prototype but there are some bugs in it. First things first, download this script and call it _includeDir.au3.
_includeDir.au3
#CS
Name: _includeDir.au3
Developer: Timothy Bomer
Copyright: Amarok Studios LLC 2016
Version: 1.0
Description:
The purpose of this UDF is to dynamically include all files inside of a folder.
It works for the most part, but I am still working on a couple of bugs.
#CE
#Include <File.au3>
Global $mainUDF = "IncludeDirUDF"
Global $includeLib = $mainUDF & "\" & "loadIncludes.au3"
Global $tempLib = $mainUDF & "\" & "lib.txt"
Global $includeRewrite = $mainUDF & "\rewrite.au3"
Global $iDirHolder = ""
Func _includeDir($iDir, $lineToInc = 1, $restart = True)
If (checkInclude()) = 1 Then
FileDelete($tempLib)
return
EndIf
If NOT (FileExists($iDir)) Then
MsgBox(16,"Directory Doesn't Exists | _includeDir","The directory " & $iDir & " does not exist!")
return 0
EndIf
$iDirHolder = $iDir
initializeCheck()
; MsgBox(0,"Include Directory", "Attempting to include: " & $iDir)
populateLib($iDir)
populateIncLib()
finalize($lineToInc, $restart)
EndFunc
Func checkInclude()
FileOpen(#ScriptName, 0)
For $i = 1 to _FileCountLines(#ScriptName)
$checkLine = FileReadLine(#ScriptName, $i)
If ($checkLine = '#Include "IncludeDirUDF\loadIncludes.au3"') Then
return 1
EndIf
Next
EndFunc
; START Initialize Check
Func initializeCheck()
; MsgBox(0,"Checking. . .", "Is this initialized?")
If (FileExists($mainUDF)) Then
If NOT (FileExists($includeLib)) Then
isError(2)
return
EndIf
; MsgBox(0,"Initialized","The UDF has been initialized")
Else
isError(1)
return
EndIf
EndFunc
; END Initialize Check
; START Library Population
Func populateLib($iDir = $iDirHolder)
; MsgBox(0,"Populating","Attempting to populate the library")
If (FileExists($tempLib)) Then
; MsgBox(0,"Temp File Found","The temporary library file has been found. Attempting to populate.")
$tLibCont = _FileListToArray(#ScriptDir & "\" & $iDir & "\", "*")
$iDirSize = $tLibCont[0]
; MsgBox(0,"Size of Included Directory", $iDir & " contains " & $iDirSize & " files to include!")
$writeLib = FileOpen($tempLib, 1)
While $iDirSize > 0
FileWriteLine($writeLib, '#Include "..\' & $iDir & '\' & $tLibCont[$iDirSize] & '"')
$iDirSize -= 1
WEnd
FileClose($writeLib)
Else
isError(3)
return
EndIf
EndFunc
; END Library Population
; START Include Library Population
Func populateIncLib()
; MsgBox(0,"Rewriting. . .", "Attempting to re-write the include library")
#CS
If (FileExists($includeLib)) Then
FileDelete($includeLib)
_FileCreate($includeLib)
EndIf
#CE
FileOpen($tempLib, 0)
For $i = 1 to _FileCountLines($tempLib)
$line = FileReadLine($tempLib, $i)
$reWriteLib = FileOpen($includeLib, 9)
FileWriteLine($reWriteLib, $line)
FileClose($reWriteLib)
Next
FileClose($tempLib)
EndFunc
; END Include Library Population
; START Finalize
Func finalize($lineToInc, $restart)
_FileWriteToLine(#ScriptName, $lineToInc, '#Include "IncludeDirUDF\loadIncludes.au3"', False)
If ($restart = True) Then
runFile(#ScriptName)
EndIf
exit
return
EndFunc
Func runFile($rFile)
$file_loc = $rFile
If #Compiled = 1 Then
$file_exe = FileGetShortName(#AutoItExe & ' /AutoIt3ExecuteScript "' & $file_loc & '"')
Run($file_exe)
Else
$file_au3 = FileGetShortName($file_loc)
Run(#AutoItExe & " " & $file_au3, "", #SW_HIDE)
EndIf
EndFunc
; START Error Reporting
Func isError($eFlag = "", $eMessage = "There was an error!")
If ($eFlag = "") Then
; MsgBox(16,"ERROR", $eMessage)
Exit
EndIf
If ($eFlag = 1) Then
; MsgBox(16,"Not Initialized","This UDF has not been initialized")
DirCreate($mainUDF)
Sleep(250)
initializeCheck()
return
ElseIf ($eFlag = 2) Then
; MsgBox(16,"Missing File","Missing the include library!")
_FileCreate($includeLib)
initializeCheck()
return
ElseIf ($eFlag = 3) Then
; MsgBox(16,"Missing File", "Missing the temporary library! Creating it now!",3)
_FileCreate($tempLib)
populateLib()
return
EndIf
EndFunc
; END Error Reporting
To use this UDF, include the file with:
Include "_includeDir.au3"
Next, call the function by following the below format.
_includeDir("Directory to Include", $lineToIncludeOn, $restart)
The directory to include would be the name of the directory with all of the files you're trying to include.
The $lineToIncludeOn specifies what line of the script the #Include will be written on. This is an optional parameter, and will default to line 1.
Lastly, $restart specifies if the script needs to be restarted or not. Sadly, the biggest bug is that the script needs to be restarted in order for the UDF to include all of the files. Which probably takes away the useful functionality of the entire script. This is an optional parameter, by default, it will be set to True and automatically restart the script.
Here's an example.
INSIDE OF WORKING DIRECTORY
Includes Folder
Example.au3
_includeDir.au3
INSIDE OF Includes Folder
One.au3
Two.au3
Three.au3
Four.au3
Five.au3
One.au3
$oneVar = "First variable"
Two.au3
$twoVar = "Second variable"
Three.au3
$threeVar = "Third variable"
Four.au3
$fourVar = "Fourth variable"
Five.au3
$fiveVar = "Fifth variable"
So, we are going to try to include One.au3, Two.au3, Three.au3, Four.au3, and Five.au3 into Example.au3.
Example.au3
; Exclude the numbers before the code. It's there just to show you the line the code is written on.
(1) #Include "_includeDir.au3"
(2)
(3) _includeDir("Includes Folder")
(4) MsgBox(0,"Included Variables","Variable One: " & $oneVar & #CRLF & "Variable Two: " & $twoVar & #CRLF & "Variable Three: " & $threeVar & #CRLF & "Variable Four: " & $fourVar & #CRLF & "Variable Five: " & $fiveVar)
This will add the line:
#Include "IncludeDirUDF\loadIncludes.au3"
to line one of Examples.au3, then restart Example.au3 to display the variables from the included files. If you changed the files inside of the Included Files directory, you will need to remove the #Include line for the loadIncludes.au3, and delete the folder that was generated. (IncludeDirUDF).
Let's say you don't want the #Include to be written to line one of Example.au3... To specify what line you want it to be written to, simply add the next parameter to the function call. For example, we want to write it to line 5 of Example.au3, we would use this:
_includeDir("Includes Folder", 5)
The last parameter is the restart parameter. This specifies if Example.au3 should be restarted after the directory is included. It is set to True by default. If you want Example.au3 to exit and stay terminated, simply add False to the end of the function call.
_includeDir("Includes Folder", 5, False)
In order to do what you're trying to do, the best way to use this would be to put it at the top of your Example.au3 (or whatever your script is) right underneath the includes. The reason for this is because it will auto-restart your script if when it generates the library and it could cause an error if it's not at the top. I hope this poorly written UDF helps you out! Let me know if there's something it's not doing that you need it to do. If not, let me know and I will fix it! Happy programming my friend! If this is too hard to follow, see my more detailed demonstration on the official AutoIt forum HERE
Thanks,
Tim
This is how i fixed it if anyone have the same issue ..
Only thing IncludeList.au3 have to exsist in the directory before you run the script or you will get include error
#include <WinAPIFiles.au3>
#include <File.au3>
; Delete Old IncludeList.au3
If FileExists("IncludeList.au3") Then
FileDelete("IncludeList.au3")
EndIf
; Get Files From Dir
$IL = _FileListToArray(#ScriptDir&"\Functions\", "*")
; Create New IncludeList.au3
$FH = FileOpen("IncludeList.au3", $FO_APPEND)
; Check For Errors
If #error <> 1 or #error <> 4 Then
; Loop True Files In Dir
For $FC = 1 To UBound($IL)-1 Step +1
; Write New #Include '.\Function\FilesToInclude.au3'
FileWrite($FH, "#Include '.\Functions\"&$IL[$FC]&"'"& #CRLF)
Next
EndIf
; Close File Handler
FileClose($FH)
; Include All The Files In Directory True IncludeList We Created
#include "IncludeList.au3"
; And Now You Can Call Any Functions From The Scripts From That Directory
Trying to read from a txt file and have the results be displayed in the message box. I plan on copying and pasting lines of 1000 and deleting them from the array, later in my code. For now I'd like to be able to see that the file can be read into the array and be displayed:
Local $List
FileReadToArray( "C:/Users/Desktop/recent_list.txt", $List [, $iFlags = $FRTA_COUNT [, $sDelimiter = ""] ])
MsgBox( 0, "Listing", $List )
I get an error:
>"C:\Program Files (x86)\AutoIt3\SciTE\..\autoit3.exe" /ErrorStdOut "C:\Users\Documents\Test.au3"
"FileReadToArray" has no other parameters than the file to read! You have used the function call from "_FileReadToArray".
The square brackets in the function line means: This parameters are optional! If you want to use them with the default values, its not required to write them in the function call.
And "FileReadToArray" reads the content of a file into an array. Thats why your call should look like so:
Local $arList = FileReadToArray("C:/Users/Desktop/recent_list.txt")
; to show every line in a MsgBox you must iterate
; through the result array
For $i = 0 To UBound($arList) -1
; MsgBox is not sensefull with hundred of lines in file!
; MsgBox(0, 'Line ' & $i+1, $arList[$i])
; better way - console output
ConsoleWrite('['& $i+1 & '] ' & $arList[$i] & #CRLF)
Next
I am using the following code to create tables in Stata:
sysuse auto, clear
table rep78, contents(mean mpg mean weight)
--------------------------------------
Repair |
Record |
1978 | mean(mpg) mean(weight)
----------+---------------------------
1 | 21 3,100
2 | 19.125 3,353.8
3 | 19.4333 3,299
4 | 21.6667 2,870
5 | 27.3636 2,322.7
--------------------------------------
How can I directly export such tables in LaTeX markup?
The community-contributed command tabout provides an out-of-the-box solution:
. tabout rep78 using table.tex, style(tex) content(mean mpg mean weight) sum replace
Table output written to: table.tex
\begin{center}
\footnotesize
\newcolumntype{Y}{>{\raggedleft\arraybackslash}X}
\begin{tabularx} {14} {#{} l Y Y #{}}
\toprule
& mpg & weight \\
\midrule
Repair Record 1978 \\
1 & 21.0 & 3,100.0 \\
2 & 19.1 & 3,353.8 \\
3 & 19.4 & 3,299.0 \\
4 & 21.7 & 2,870.0 \\
5 & 27.4 & 2,322.7 \\
Total & 21.3 & 3,032.0 \\
\bottomrule
\end{tabularx}
\normalsize
\end{center}
Below is a very simple working example of using the community-contributed command texdoc:
/* Create sample data */
clear *
set obs 10
gen date = _n
expand 10
set seed 123
gen i = runiform()
cd "/path/to/my/output"
/* Initialize and create LaTeX document */
texdoc init TexTest, replace
tex \documentclass{article}
tex \usepackage{stata}
tex \begin{document}
tex \section{Table 1}
texdoc stlog TexLog
table date, contents(mean i)
texdoc stlog close
tex \end{document}
The above code snippet creates TexTest.tex at /path/to/my/output/.
I use texdoc stlog before submitting the table command to capture the output of table in the log file. If you open the resulting tex file you will notice there is a line that has \input{TexLog.log.tex}. The compiler will insert the contents of the log file at that location in the LaTeX document.
You can compile the tex file using your preferred method. I use pdflatex in a Linux environment but had issues after initially installing it and had to resolve some dependencies for stata.sty.
After compiling, the resulting pdf file will contain the following table: